Compare commits

...
Author SHA1 Message Date
jakman85 37eea51f24 merge: the substitution check now catches a collapsed doubled dollar
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
The existing check audited only the inverse — a ${...} undefined or
unescaped — so it passed the manifest that broke prod: Postgres
dollar-quoting under a postBuild.substituteFrom path, which envsubst halves,
delivering 'DO $' and a syntax error on every reconciled run (fixed 16b7fcd).

Measured rather than assumed: Flux 2.4.0 embeds fluxcd/pkg/envsubst (a fork
of drone/envsubst — the brief credited the wrong module), and --strict output
is BYTE-IDENTICAL to non-strict here, so strict mode is blind to this class
entirely. The decisive contrast is two commands on the same stream: kubectl
kustomize emits valid 'DO $$' — which is what the kubectl-apply
verification saw — while the same stream through flux envsubst emits 'DO $'.

357 assertions, 0 unjustified. Legitimate uses genuinely exist (18 ${} escapes
plus two reaching shell comments), so the allowlist is a reviewed opt-in
rather than a bypass: a stale entry fails, a wildcard is matched verbatim and
so fails as stale, a short context or reason is rejected, and a genuine entry
covers only the span it matches — a second occurrence in the same file still
fails. Strongest proof: restoring the real pre-16b7fcd manifest verbatim makes
the guard reject the exact commit that broke production.
2026-09-13 16:27:51 -05:00
jakman85 2f579f103b merge: Longhorn and cert-manager metrics exist now — 13 alerts were inert
Neither family was scraped at all, so every Longhorn alert (degraded,
faulted, capacity) and every cert-manager alert (expiry, issuance) was
structurally incapable of firing: storage health unmonitored on a cluster
whose node just had a failing drive, TLS expiry unmonitored with the live
wildcard cert 89 days out. Now proven on prod: longhorn_* 8717 series with
longhorn-backend up=3, certmanager_* 104 series across 3 targets.

Adds an L0 guard (118 assertions) that crosses from each alert rule to the
scrape config and re-proves each enabling declaration against the working
tree, including the negatives — cert-manager's monitor mutual-exclusion, the
Longhorn metrics NetworkPolicy, and kube_node_labels needing
metricLabelsAllowlist. Proven able to fail five ways.

Hubble is deliberately NOT enabled: it opens 9965 on every cilium-agent and
rolls the CNI DaemonSet across all three nodes, and platform/cilium carries an
explicit parity contract with the Ansible template that a HelmRelease-only
change would silently break. Registered as a declared gap instead.

Two findings this surfaced, neither fixed here and neither absorbed silently:
twelve apps/* <app>-down SLI rules discover zero targets because their
ServiceMonitors name port 'metrics' and no app Service defines that port; and
kube-state-metrics is OOMKilled roughly every ten minutes, making thirteen
kube_* rules intermittently inert.
2026-09-13 16:26:12 -05:00
jakman85andClaude Opus 5 830983b791 test(l0): prove the dollar-collapse guard fires — and name the real envsubst
The implementation landed in the preceding WIP commit had never been executed.
This finishes it: the arithmetic is re-measured, the guard and its allowlist are
attacked, and the header's one factual error is corrected.

WHAT ENVSUBST ACTUALLY IS. The header credited `github.com/drone/envsubst`. The
pinned flux 2.4.0 binary in the dev image embeds
`github.com/fluxcd/pkg/[email protected]` — Flux's own fork of drone/envsubst
("This package is a fork of drone/envsubst", its README) — read straight out of
the binary with `grep -ao`. Same behaviour, different module; the header now says
so and says how to check, because a claim about a dependency that nobody can
re-derive is how the last wrong assumption survived.

RE-MEASURED, NOT ASSERTED. Every row of the DOLLAR-RUN ARITHMETIC table was
re-run through `flux envsubst` (flux 2.4.0, DEFINED=VALUE) and every row holds:
`$$`->`$`, `$$$`->`$$`, `$${D}`->`${D}`, `$$${D}`->`$VALUE` (odd run: the brace
is LIVE), `$$$${D}`->`$${D}`, `DO $$ x $$;`->`DO $ x $;`, and `$$${ARR[0]}` dies
with "missing closing brace" while `$${ARR[0]}` is fine. `--strict` output was
byte-identical to non-strict, exit 0 both ways (`cmp`) — strict mode is blind to
this whole class, which is why a source assertion is the only guard available.
The header now carries the 30-second reproduction, flagged to be re-measured on
any FLUX_VERSION bump.

END TO END ON THE REAL PATH. With the pre-16b7fcd manifest restored,
`kubectl kustomize apps/wave-0` emits `DO $$` — valid SQL, and exactly what the
`kubectl apply` verification saw — while the same stream through
`flux envsubst --strict` emits `DO $` and `$;` and exits 0. That two-command
contrast IS the incident.

THE GUARD IS NOT TRIVIALLY SATISFIABLE — 16 adversarial cases, all as intended,
run against a throwaway copy of the tree so nothing could be left behind:
  * an unmarked `$$` under a substituted path FAILS (DOLLAR-COLLAPSE); so do an
    odd `$$${...}` (DOLLAR-HALF-ESCAPED) and an even `$$$${...}` (OVER-ESCAPED);
  * a stale entry FAILS three ways — context no longer in the file, context that
    only covers an already-accepted `$${` escape and so excuses nothing, and a
    path that is not a file in the repo;
  * the allowlist cannot blanket-exempt a path: a 2-char context is rejected
    (<24), a `.*` wildcard is matched VERBATIM and therefore rejected as stale, a
    24-char context carrying no dollar run is rejected, a 4-char reason is
    rejected (<40) — and, the central case, a genuine entry covers only the span
    its context matches, so a SECOND `$$` elsewhere in the same file still fails.
    Two genuine entries for two real occurrences do pass, so the mechanism is
    usable rather than merely unsatisfiable;
  * the CLASS, not the instance: the pre-fix `DO $$ ... $$;` reconstructed in a
    NEW ConfigMap under apps/wave-2 fails at both dollar runs, and restoring the
    real pre-16b7fcd apps/wave-0/nextcloud-db.yaml verbatim fails the check at
    lines 384 and 391. The guard rejects the exact commit that broke production.
  The scratch fixtures lived only in the throwaway copy; `git status` on this
  branch stayed clean throughout, which is the other half of the last attempt's
  unfinished business.

LEGITIMATE USES EXIST AND STAY GREEN. 18 `$${...}` escapes (unifi cloud-init,
the gitea and nextcloud OIDC bootstrap scripts, two longhorn comments) pass
unremarked, and the 2 non-braced `$$` occurrences are the two allowlist entries.
Those two are shell comments inside block scalars, and `kubectl kustomize`
confirms the comment text really does survive into the stream envsubst reads —
so they are genuine, reaching content whose collapse is cosmetic, not a
hypothetical the allowlist was invented to excuse.

STILL NOT PROVEN, AND STILL NOT PROXIED: that a change was verified through a
substituting path. Which command a human ran is a fact about a shell session;
git stores the manifest, never the invocation, so it is not statically decidable
here and no honest formulation exists. Grepping commit prose for `kubectl apply`
would be satisfiable by rewording and would fire on the break-glass runbooks
whose whole content is that command. The answer is to move the proof off the
verification path and onto the source, which is what assertion (7) does.

No ci/l0.sh stage: the `conformance.home.k3s/scope: "whole-repo"` annotation is
what wires it, and whole-repo-invariants-wired fails the gate on a hand-written
stage.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 16:24:42 -05:00
jakman85 6d13822941 merge: the gate no longer prints PASS when a stage did not run
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
ci/gate.sh set BLOCK=1 on a FAILED stage but only warned on a SKIPPED one, so
it printed the words 'not green' and then 'MERGE GATE: PASS', exit 0 — fail
open in the one component whose job is to fail closed. Observed live with
prom-rules, and independently with l0 when the pinned kyverno CLI was absent.

New contract: PASS/0 means every stage that was supposed to run ran and was
green; BLOCKED/1 a failure or a must-run stage that did not; UNDETERMINED/3
nothing failed but greenness was not established; NOT APPLICABLE/0 unchanged.
No knob inside gate.sh can turn UNDETERMINED into 0 — tolerating a non-answer
is the caller's decision to make and to document.

The distinction that makes it usable: a capability the environment DECLARES
absent may still pass (no cluster, named in the headline); a stage that COULD
NOT RUN may not. CI declares GATE_STRICT_SKIPS in the workflow rather than
sniffing for CI, so a missing pinned tool there is exit 1.

Carries over 47068ee's unusable-subset blocking from the unmerged
fix/touched-requirements-fail-closed, now with tests. Measured its companion
claim honestly: 159 of 613 paths (26%) are unmapped today, not 255 of 476
(54%) — 56 prose, 103 non-prose — which remains the larger fail-open and is
recommended as its own change rather than smuggled in here.

47 assertions, 32 of which fail against the old gate.sh.
2026-09-13 16:23:37 -05:00
jakman85andClaude Opus 5 a92fa11701 feat(monitoring): scrape Longhorn + cert-manager — 8 alerts were structurally inert
An alert rule whose metric is never scraped is not a broken alert, it is an ABSENT
alert that reads as a present one. `longhorn_volume_robustness == 2` over a TSDB
with no longhorn_* series returns no vector: permanently quiet, permanently
"healthy", indistinguishable from a correct rule. It survives review (the PromQL is
right), it survives `promtool test rules` (the unit test supplies the series
itself), and it appears on no dashboard.

VERIFIED ABSENT ON PROD before this change (re-verified 2026-09-13 21:0xZ, ~15h
after the first audit, because a stale reading is not evidence):

  count(longhorn_volume_robustness)                      => (empty)   4 rules inert
  count({__name__=~"certmanager.*"})                     => (empty)   3 rules inert
  count(hubble_drop_total) / {__name__=~"cilium.*"}      => (empty)   1 rule inert
  count by (job) (up)                                    => 20 jobs, none of them
                                                            Longhorn, cert-manager,
                                                            Cilium or Hubble

Consequence: storage health was unmonitored on the cluster whose prod-1 had just
had a failing SSD, and TLS expiry was unmonitored with the live `wildcard-home`
wildcard 89 days from expiry on an ACME DNS-01 renewal that can fail silently.

WHAT THIS CHANGES

  * platform/longhorn: `metrics.serviceMonitor.enabled: true` (the supported chart
    way — rendering longhorn 1.12.1 emits exactly 1 ServiceMonitor with the flag and
    0 without), plus networkpolicy-metrics.yaml. The NetworkPolicy is LOAD-BEARING,
    not belt-and-braces: measured with the monitor applied and no allowance, all 3
    targets were DISCOVERED and every scrape failed `context deadline exceeded`
    after 10.0s — a drop, not a refusal — while curling the endpoint from inside
    the manager pod returned 200 in 0.031s. Longhorn 1.12.1 ships 6 NetworkPolicies
    of its own, guarded on `networkPolicies.restrictInternalTraffic` (default TRUE)
    and NOT on `networkPolicies.enabled`, so they are effectively unconditional and
    the manager policy admits only same-namespace Longhorn components. The one-line
    alternative (restrictInternalTraffic: false) was rejected: it deletes the
    internal-traffic restriction across all six components to open one port to one
    namespace, trading an R-SEC-02 control for an observability one when the two do
    not conflict. This policy is Ingress-only, ns monitoring -> app=longhorn-manager,
    TCP 9500 only; 9502/9503 (webhook, recovery backend) stay closed.

  * platform/cert-manager: `prometheus.servicemonitor.enabled: true`. Chart default
    already serves 9402; only the monitor was missing. No NetworkPolicy needed —
    neither ns cert-manager nor ns monitoring carries one. NOT paired with
    prometheus.podmonitor.enabled: the chart calls `fail` if both are set.

  * platform/kubevirt: KubeVirtVMLaunchFailing queried
    `kubevirt_vmi_phase_count{phase="Failed"}`, which DOES NOT EXIST on the deployed
    KubeVirt — a different flavour of the same defect, with the scrape target up and
    healthy. Prod has only `kubevirt_vmi_info` and `kubevirt_vmi_number_of_outdated`;
    the only phase-bearing names are the phase_transition_time histograms. Corrected
    to `kubevirt_vmi_info{phase="failed"}` — LOWERCASE, because KubeVirt lowercases
    the value (observed `phase="running"` for the unifi VMI) and a capital-F matcher
    would select nothing and be exactly as inert.

PROVEN, NOT ASSERTED. The rendered ServiceMonitors were applied to prod by hand and
the metrics observed to land, then REMOVED again so the merge's Helm upgrade creates
and owns them rather than colliding with an unowned object:

  count(up{job="longhorn-backend"})        => 3        (all three managers, up==1)
  count({__name__=~"longhorn.*"})          => 8717     (was empty)
  count(longhorn_volume_robustness)        => 44
  up{namespace="cert-manager"}             => 3 targets, all 1
  count({__name__=~"certmanager.*"})       => 104      (was empty)
  wildcard-home days to expiry             => 89.2, ready_status{condition="True"}=1

All 58 alert rules in the repo were then evaluated against prod one by one: every
one returns a result without error, and exactly ONE fires.

TRIAGE OF THE ONE NEWLY-FIRING ALERT — A TRUE POSITIVE, NOT TUNED AWAY.
LonghornVolumeCapacityHigh on the Prometheus TSDB volume:
`longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes` = 0.971
(48.6 GiB allocated of 50 GiB) while the kubelet's view of the same filesystem is
only 20.4% used. Longhorn is not reclaiming freed blocks, so ~48.6 GiB x 3 replicas
is committed for ~10 GiB of real data. On a cluster whose failure mode is disk
pressure and whose etcd WAL shares a spindle with Longhorn, that is worth knowing
and was invisible until now. The threshold is left alone; the follow-up is
filesystem trim, not a bigger number.

THE GUARD (conformance/tests/l0-static/alert-metrics-have-scrape-targets)

Whole-repo L0 invariant, 118 assertions. It crosses from each alert to the scrape
config that makes its metric exist: every metric name is claimed by exactly one
declared owner, and each owner's enabling declaration is re-proven against the
working tree on every gate run, including the negative conditions that defeat an
enablement without deleting it. `up{job="X"}` is audited on its own axis, because
`up` always exists: where this repo authors a Service and a ServiceMonitor in the
same file, the monitor's endpoint port must be a port that Service defines.

It DECLARES conformance.home.k3s/scope: "whole-repo" and adds NO ci/l0.sh stage —
the whole-repo-invariants-wired meta-test fails the gate if one reappears.

PROVEN ABLE TO FAIL, five ways, each reverted:
  1. an alert on a fabricated metric          -> FAIL "claimed by NO entry"
  2. longhorn metrics.serviceMonitor removed  -> FAIL "no longer matches"
  3. networkpolicy-metrics.yaml unwired       -> FAIL "the monitor alone is NOT enough"
  4. cert-manager podmonitor also enabled     -> FAIL "FORBIDDEN pattern"
  5. a register entry made stale / removed    -> FAIL in both directions

TWO SHRINK-ONLY REGISTERS, because a blanket fail would make the gate red until
work that is not this change lands, and a disabled gate protects nothing:

  * KNOWN_UNWIRED_METRICS (1): hubble_drop_total. See below.
  * KNOWN_UNWIRED_JOBS (12): every apps/* `<app>-down` SLI rule is ALSO inert, and
    this was found by the re-audit, not previously known. The rendered
    ServiceMonitors name `port: metrics` and NOT ONE of the twelve app Services
    defines a port of that name (zero occurrences of `name: metrics` across all
    twelve manifests); prod confirms none of the twelve jobs appears in `up`. This
    is the nastiest variant of the class: `up` exists in abundance so nothing looks
    wrong, and PrometheusTargetsMissing cannot see it either, because a target that
    was never discovered has no up series to be 0. Closing it needs an exporter per
    app (Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr speak no
    Prometheus) — except keycloak, where the live jobs are keycloak-service /
    keycloak-discovery and the rule is a rename away from being live.

NOT DONE HERE, DELIBERATELY: Hubble. hubble.metrics.enabled RECONFIGURES THE CNI —
it opens 9965 on every cilium-agent and rolls the DaemonSet across all three nodes,
serialised by this repo's own maxUnavailable: 1 (set after the 2026-09-01
k8sServiceHost outage, where a CNI roll deadlocked 2 of 3 agents). It also needs a
matching edit to roles/cilium/templates/cilium-values.yaml.j2 in the same commit,
because platform/cilium/helmrelease.yaml carries a [parity] contract against it and
a rebuilt node would otherwise bootstrap a CNI config that disagrees with the
reconciled one. Recommended as its own change on the cilium Kustomization, with the
Ansible parity edit, a narrow labelsContext, and a one-node canary — not smuggled in
here.

ALSO FOUND, NOT FIXED HERE: kube-state-metrics is OOMKilled every ~10 minutes on
prod (90 restarts in 16h against a 128Mi limit) while its logs show the
custom-resource-state factory reloading every 3 seconds. Thirteen alert rules read
kube_* metrics, so those rules are INTERMITTENTLY inert — the same defect class,
arriving through a crashloop rather than a missing monitor. Needs its own change.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 16:21:44 -05:00
jakman85andClaude Opus 5 cd64b4b672 fix(ci): SKIPPED is not green — the gate can no longer PASS on a stage that did not run
Completes the WIP preserved in the previous commit. `MERGE GATE: PASS` + exit 0 now means
what it says: every stage that was SUPPOSED to run ran, and was green.

THE DEFECT. In gate.sh's decision section a FAILED stage set BLOCK=1 but a SKIPPED stage only
warned, so the gate printed `l0: skipped (unwired seam / missing tool) — not green` and then
`MERGE GATE: PASS`, exit 0. Seen twice for real: `prom-rules` skipped because promtool was
absent, and `l0` skipped because the pinned kyverno CLI was. Exit 0 is what CI and every
`||`-chained script read as merge-approved, so the one component whose entire job is to fail
closed was failing OPEN — in a repo whose stated principle is that SKIPPED IS NOT GREEN, and
whose history already contains the 2026-08-31 "local false green" that UNDECIDED was added to
prevent. The offline stages simply never used that vocabulary.

A SKIP IS NOT ONE THING — the per-stage rule, with the justification in the file header:

  * lint / l0 / render-golden / l1 — COULD NOT RUN (a pinned tool absent, a seam unwired).
    BLOCKED under strict skips (a CI runner has the toolchain installed for it by
    install-tools.sh, so a skip there is a defect); UNDETERMINED (exit 3) otherwise — a
    workstation missing a tool is a real local condition, still non-zero, still never a pass.
  * spec-immutability — the spec repo is not checked out, so R-TEST-06 COULD NOT BE VERIFIED:
    UNDETERMINED. Deliberately NOT upgraded to a block by strictness, because the live runner
    does not check the spec repo out at all and blocking would leave the only enforced status
    check permanently red for a reason no PR can clear. It is the ONE skip a caller may
    knowingly tolerate; REQUIRE_SPEC_VERIFICATION=true makes it hard once wired.
  * conformance L2/L3 — CLUSTER_AVAILABLE!=true is the environment DECLARING the capability
    absent, not a stage failing to run where it could have. A declared-inapplicable stage was
    never "supposed to run", so it does not deny a PASS — but it is now NAMED in the PASS
    headline and the step summary, so PASS never implies more than was verified. A skip
    DESPITE CLUSTER_AVAILABLE=true still blocks.

    declared-not-applicable may pass; could-not-run may not.

STRICTNESS IS DECLARED, NOT SNIFFED. .gitea/workflows/gate.yml sets GATE_STRICT_SKIPS=true on
the gate step — the workflow is the thing that knows it is CI — with GITHUB_ACTIONS/CI (set by
the runner itself) only as the `auto` fallback. No hostname/path/tool-availability heuristic.
A wrong answer in either direction cannot fail open: both branches are non-zero and neither is
a PASS. That step now calls ./ci/gate.sh directly and tolerates exit 3 in one visible line,
because under strict skips exit 3 can ONLY mean the unverifiable spec repo — a missing tool is
exit 1 and untolerated. Tolerating a non-answer is a reviewed choice in the CALLER; there is
deliberately no knob inside gate.sh that turns UNDETERMINED into exit 0.

ALSO, from the unlanded fix/touched-requirements-fail-closed (47068ee), whose reasoning is
carried over rather than its tree: gate.sh special-cased only exit 3 from
touched-requirements.sh, so ANY other non-zero exit left the touched set empty and printed PASS
having applied no coverage gate at all. An unusable subset is not an empty subset: it BLOCKS.
Latent today (the script exits only 0 or 3) and wired now so the hole is already closed when a
fail-closed path map makes exit 1 reachable — the measured state of that map is in the PM report.

TESTED — ci/tests/gate-decision.test.sh, 47 assertions over the whole decision table: the real
gate.sh and lib.sh run in a throwaway CI_DIR against STUB stage scripts whose exit codes the
suite dictates, so it asserts how the gate REASONS, with no tools, no cluster, no network. All
stages green -> PASS/0; each stage FAILED -> BLOCKED/1; each stage SKIPPED -> never PASS and
non-zero; strict skips -> BLOCKED/1; the spec exception; the declared-gap PASS; the
undetermined-touched-subset paths unchanged (on-base -> NOT APPLICABLE/0, no-base and
diff-failed -> BLOCKED/1); UNDETERMINED outranking NOT APPLICABLE; an unusable subset ->
BLOCKED/1; truthy()'s misspelling direction. 32 of the 47 FAIL against the pre-change gate,
including "l0 SKIPPED: printed 'MERGE GATE: PASS'" — the suite is falsifiable, not decorative.

lib.sh: truthy() is now the one spelling of "is this switch on" and cluster_available() uses it
(the comment claimed that; now it is true), and load_touched_requirements' rc!=0 branch says
what actually happened instead of "proceeding with the full layer".

Refs R-TEST-02, R-TEST-03, R-TEST-06.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 16:20:41 -05:00
jakman85andClaude Opus 5 377fa205ea WIP: catch envsubst collapsing a doubled dollar in substituted paths
INCOMPLETE — preserved by the PM after the authoring agent hit the session
rate limit mid-task. Do not merge without finishing and gating it.

Closes the inverse of what this check already covered. It asserted that every
${...} in a substituted path is defined or escaped; it did NOT catch content
containing a literal doubled dollar that the author needs delivered verbatim
and that envsubst silently halves. That shipped a broken CronJob (fixed in
16b7fcd): Postgres dollar-quoting became a single dollar and every
Flux-reconciled run died with a syntax error, while passing review because
verification used kubectl apply, which runs no envsubst.

Introduces a dollar-collapse.allow allowlist so each legitimate use is a
reviewed decision rather than an accident.

WHAT REMAINS: the agent's own next step was proving it catches the CLASS and
not just this instance, via a scratch manifest under a substituted path. Its
scratch fixture (apps/wave-2/zz-scratch-class.yaml and the wave-2
kustomization edit) was deliberately temporary and has been reverted here, so
the fail-proof is NOT yet demonstrated. Also unanswered: brief point 3, on
whether the check can say anything about a manifest verified only through a
non-substituting path.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 16:09:37 -05:00
jakman85andClaude Opus 5 63bc890261 WIP: the gate must not print PASS when a stage was skipped
INCOMPLETE — preserved by the PM after the authoring agent hit the session
rate limit mid-task. Do not merge without finishing and gating it.

Problem: in ci/gate.sh a FAILED stage sets BLOCK=1 but a SKIPPED stage only
warns, so the gate prints the words 'not green' and then concludes
'MERGE GATE: PASS' and exits 0. Observed live tonight with prom-rules, and
independently with l0 when the pinned kyverno CLI was absent. That is
fail-open in the one component whose whole job is to fail closed.

The agent was mid-way through adding truthy/strict_skips helpers to ci/lib.sh.

WHAT REMAINS:
  - Per-stage justification for which skips must BLOCK and which may degrade
    to a non-zero UNDECIDED (spec-immutability and L2/L3 skip for legitimate
    environmental reasons and must stay usable locally).
  - Whether CI and local can be distinguished reliably here.
  - New ci/tests/ suite for the decision logic, and a before/after proof.
  - Reconciling with the unmerged branch fix/touched-requirements-fail-closed,
    whose 47068ee already makes an unusable touched subset block, and whose
    56c242d claims 255 of 476 paths were unmapped in the path->requirement
    map — a far larger fail-open that still needs measuring on current main.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 16:08:11 -05:00
jakman85andClaude Opus 5 f4f6e5f7a3 docs(pm): handoff — five WIP branches preserved after a session rate limit
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 08:55:05 -05:00
jakman85andClaude Opus 5 a79693d3bd feat(monitoring): scrape the Flux controllers — five gotk_* alerts were inert
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
kube-prometheus-stack does not scrape flux-system: the controllers are
bootstrapped, expose no Service, and no chart value covers them. So every
gotk_* alert rule in platform/monitoring/alerts.yaml could never fire —
verified on prod, count({__name__=~"gotk.*"}) returned nothing. A repo whose
entire delivery model is Flux had no alerting on reconciliation failing, a
Kustomization erroring, or a resource left suspended.

Proven live before landing: with this PodMonitor applied, gotk_suspend_status
returns 39 series and gotk_reconcile_condition 117, and FluxResourceSuspended
actually fired on a deliberate test suspension — evidence the rule is live,
not merely present.

Extracted from the wider inert-alert audit on
fix/scrape-targets-for-inert-alerts, which stalled unfinished. This is the
self-contained, verified half; Longhorn and cert-manager metrics are still
ABSENT and their alerts remain inert. It also adopts the object into git: it
existed on prod only as kubectl-client-side-apply, so a rebuild would have
lost it.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 08:53:21 -05:00
jakman85andClaude Opus 5 16b7fcd677 fix(cnpg): the grant reconciler died on every Flux-managed run
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
The SQL asserted convergence in a dollar-quoted PL/pgSQL DO block. That
cannot survive this delivery path: the ConfigMap is rendered through Flux's
postBuild substitution, and envsubst collapses a doubled dollar into one, so
Postgres received a half-quoted DO and failed with a syntax error. Three
scheduled runs had left nine Error pods in nextcloud-db.

It passed verification because it was hand-applied with kubectl, which runs
no envsubst — the defect only existed once Flux owned the object. The GRANT
on the first line still succeeded, so the privilege held and the blind spot
stayed closed; what was broken was the assertion meant to prove it, which is
the part that makes a silent regression visible.

Rewritten in plain SQL with no dollar sign anywhere in the file, comments
included. Escaping by doubling again would also work and would be the next
person's trap. has_database_privilege()::int is 1 or 0, so the division
raises division_by_zero and ON_ERROR_STOP=1 fails the Job; the divisor is
not constant so it cannot be folded away at plan time.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 00:19:49 -05:00
jakman85 5fa893095b merge: the placeholder guard test was hiding a RED, not a pass
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
It had been permanently SKIPPED because it expected bootstrap/standup.sh,
which never landed on main — planned, never merged, still living only on the
unmerged prod-standup branches along with the 2026-09-05 incident it came
from. Dropping the real script in and running the old check produced FAIL
2/34: it would have false-blocked the very merge that landed its own input.

Now split into the half that needs no guard (no overlay ships a
placeholder-shaped effective cluster-vars value; an empty value must be
STATED by the overlay, not inherited in silence; a loud base sentinel must
be explicitly overridden) and the half that self-activates. The guard is
discovered by CONTENT — any tracked *.sh defining <name>_is_placeholder() —
so landing one anywhere activates the second half with no edit here, and two
definitions is a failure. The check never exits 3 again.

Unscoped L0 goes 13 PASS / 1 SKIP -> 14 PASS / 0 SKIP.
2026-09-13 00:14:17 -05:00
jakman85 e49b86eb41 merge: 35 runbooks, served, with a guard that they resolve
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
All 46 runbook_url annotations pointed at https://<apex>/runbooks/<name>
and every one returned 404 — there was no runbooks/ directory and no route.
Every alert that reached a human handed them a dead link at the moment they
needed it.

Runbooks are now served from their own non-gating tier at the apex prefix,
PUBLIC on purpose: gating them behind Keycloak would make keycloak-down
unreadable exactly when Keycloak is down, which is a loop with no exit. The
new L0 guard checks both directions — a runbook_url with no file, and a file
not wired into the generator — and it caught a live one on contact with
main: CNPGPrimariesCoLocated had shipped hours earlier pointing at a runbook
that did not exist.
2026-09-13 00:11:03 -05:00
jakman85 d616fd4025 merge: promtool is in the toolchain, so the alert unit tests actually run
324 lines of mutation-checked promtool tests had landed with nothing to
execute them: promtool was not in the pinned toolchain, so the coverage was
present and enforced by nothing — the failure mode this repo keeps hitting.

promtool 3.14.0 is now pinned in both versions.env files (matching prod's
Prometheus, verified against the running image) and baked into the
container. The two competing harnesses are consolidated into one, and the
two competing wirings into one: the bespoke ci/prom-rules.sh stage is
deleted in favour of a scope-declaring conformance test, because a
hand-written stage running one hardcoded script is exactly the pattern
b9f8807 just removed. Missing promtool is exit 3 — UNWIRED, never green.

51 of 58 alert rules still have no unit test; that gap is a shrink-only
ratchet that fails if it grows, rather than a blanket FAIL that would get
disabled or a report nobody reads.
2026-09-13 00:11:03 -05:00
jakman85andClaude Opus 5 87c3d83dff fix(conformance): placeholder-guard-coupling actually RUNS — it was skipping forever
`conformance/tests/l0-static/placeholder-guard-coupling` (R-ENV-02, R-STATE-03) exited 3
SKIPPED on every gate run since it landed, because its check keyed its whole existence on
`bootstrap/standup.sh`. Measured, not assumed: that file has NEVER been on `main` — there is
no deletion commit for it anywhere; it landed only on unmerged prod-standup branches
(`work/prod-standup-campaign`, `prod/*`, `feat/prod-hardening`). It was planned and never
merged, not removed. SKIPPED is not green, and a permanently-skipped incident-derived test is
the quiet version of the failure it was written to prevent.

The skip was also hiding a RED. Run against the real `standup.sh` from
`work/prod-standup-campaign`, the old check FAILED 2 of 34 assertions — false positives from
treating the empty string as placeholder-shaped for knobs cluster-vars authors empty on
purpose (`BACKUP_CREDENTIALS_SUFFIX`, `VELERO_BACKUP_PREFIX`, both `""` on dev and both
documented there as load-bearing). It would have false-blocked the merge landing its own input.

NOT deleted — the hazard is not gone. Split into two halves, one of which needs no guard:

  HALF A (always runs): no environment overlay ships a placeholder-shaped EFFECTIVE
  cluster-vars value; an empty effective value is STATED by the overlay rather than inherited
  in silence; a LOUD base sentinel is explicitly overridden by every overlay. This is the
  2026-09-05 outcome asserted at the source instead of at the guard — if prod's
  S3_BACKUP_ENDPOINT is never a placeholder, velero cannot be pointed at a hostname that does
  not exist. Input surface is `clusters/`, all of it on `main`.

  HALF B (self-activating): the real predicate extracted and executed against those same
  values, the consumer trace, and the mid-string anchoring property.

The check NEVER exits 3 again, and the guard is DISCOVERED by content rather than hardcoded:
any tracked `*.sh` defining `<name>_is_placeholder()` is found, so a guard landing at
`bootstrap/standup.sh` — or moved into a sourceable library, the remedy the check already
recommends — activates HALF B with no edit to the test. Two files defining it is a FAIL. When
no guard exists the gap is printed loudly every run instead of swallowed as a skip.

LOUD vs EMPTY is the correction that makes it honest in both modes: a loud placeholder is
always a defect in an overlay, while `""` is held to the explicitness rule instead — and still
counts as a placeholder for a knob a guard covers, because a `""` case-arm is the guard
author's declaration that empty is not a usable target.

Proof, all reverted: 84 assertions and exit 0 on `main`; prod S3_BACKUP_ENDPOINT set to the
literal `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` FAILS (A1); deleting dev's
explicit VELERO_BACKUP_PREFIX op so it inherits `""` FAILS (A2); a LOUD base BULK_NFS_SERVER
with prod's override deleted FAILS 2/86 (A1+A3); and with the real `standup.sh` present,
reverting `s3_is_placeholder` to the `TBD-*` PREFIX glob FAILS 12/110 (B3 anchoring). With the
pristine `standup.sh` the check PASSES at 110 assertions where the old one failed 2 of 34.

L0 layer: 14 checks, 14 PASS, 0 SKIP, 0 FAIL (was 13 PASS / 1 SKIP). Coverage unchanged —
R-ENV-02 and R-STATE-03 both still list this test, 49/51 with the same 2 gaps.

Also:
  * `ci/touched-requirements.sh` — `bootstrap/*` -> R-STATE-03 R-GITOPS-01 is KEPT, its stale
    comment corrected. It claimed standup.sh "carries the placeholder guards", describing a
    file that is not here; R-STATE-03 there is ANTICIPATORY. Deleting the entry would leave the
    change that lands a placeholder guard selected by nothing.
  * `ci/tests/touched-requirements.test.sh` — new path-map section (12 assertions, was 8)
    locking `bootstrap/*` and `clusters/*`, so removing either entry is a test failure rather
    than a silent loss of selection. All three ci/tests pass (12 + 31 + 15).
  * `docs/BULK-NFS-STORAGE.md` §7 — its "nothing in the gate currently refuses a
    placeholder-shaped overlay value" is no longer true; corrected.
  * Scope re-audited: still deliberately requirement-scoped, not whole-repo. Every path that
    can break it attributes a requirement it declares.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 00:10:37 -05:00
jakman85andClaude Opus 5 bce8778096 docs(runbooks): forward-auth IS enforced — correct the posture note
The serving route's comment claimed that because no Gateway ext-authz
attachment exists, every forward-auth route is 'in practice reachable
unauthenticated today'. That is wrong, and wrong in a dangerous direction
for a document an operator reads during an incident.

Enforcement here is PER-POD, not gateway-level: each of the five services
declaring auth: forward-auth (pihole, jackett, qbittorrent, radarr, sonarr)
publishes both its app port and 4180 on its Service, and its HTTPRoute
targets :4180 — the oauth2-proxy sidecar. Verified on prod. The app port
remains reachable in-cluster on purpose, which is how sonarr/radarr drive
qBittorrent's API, and Cilium default-deny governs that path.

The public-by-design decision for this route is unchanged and still correct;
what changes is that it is genuinely the exception rather than one open route
among many. Headlamp's exclusion is likewise deliberate (native OIDC with
per-user id_token forwarding and no ServiceAccount cluster rights).

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 00:10:24 -05:00
jakman85andClaude Opus 5 78d0f1b0f6 feat(ci): promtool in the pinned toolchain, one alert-rule harness, and the gaps declared
The CNPG alert unit tests (324 lines, mutation-checked) were run by NOTHING. promtool was
not in the baked toolchain, no gate stage evaluated any *.promtool-test.yaml, and the only
runner `curl | tar -xz`'d a promtool release from GitHub at TEST-RUN time if one was
missing. Hours later a second, independent harness landed for the Velero rules
(platform/monitoring/tests/) with its own fallback — a `podman run` of
quay.io/prometheus/prometheus. So the repo had two harnesses, two acquisition paths, and
coverage that looked present and was enforced nowhere: the exact failure mode that had just
produced a permanently-firing CNPG alert and a structurally inert Velero alert.

PROMTOOL IS NOW PINNED LIKE EVERY OTHER TOOL. PROMTOOL_VERSION=3.14.0 in
infra/devcontainer/versions.env and ci/versions.env (the two stay in lockstep, per the
repo's rule), baked by the Containerfile from the prometheus release tarball (promtool ships
inside it), recorded in /etc/dev-toolchain-versions, passed by dev.sh, and installed for the
l0 stage by .gitea/workflows/scripts/install-tools.sh so the CI runner is not permanently
SKIPPED. The pin is not free-floating: promtool IS the rule engine, so it tracks the
Prometheus prod actually runs — verified against the live cluster, not assumed:

    kubectl --context k3s-prod get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'
      -> v3.14.0-distroless   (image quay.io/prometheus/prometheus:v3.14.0-distroless)

ONE HARNESS. platform/cnpg/run-alert-tests.sh is folded into
platform/monitoring/tests/run.sh, which now covers every PrometheusRule in the repo that has
tests, via two conventions it discovers automatically: `<manifest>.promtool-test.yaml`
beside its manifest (cnpg), and `<component>/tests/*.test.yaml` over that component's
manifests (monitoring). It writes the flattened rules under whatever name each test file's
own `rule_files:` declares, so no test file needs to know the harness exists, and it
`promtool check rules` first — a rules file that does not parse would make every unit test
pass vacuously by finding no rules at all. The cnpg script's reasoning is preserved: why a
CR must be flattened (`.spec.groups`, never re-typed), and why generated files go to a temp
dir. Its `${BASE_DOMAIN}` substitution is dropped in favour of extract_rules.py's verbatim
placeholders, with the trade-off recorded there — the hazard it guarded against is addressed
by the temp dir, and verbatim leaves one transformation fewer between the shipped rule and
the rule under test.

NO ENGINE IS ACQUIRED AT RUN TIME any more — not the GitHub release, not the container
image. Not because a pinned download is unusual here (the Containerfile downloads every tool
it bakes) but because those two decided the version, the network and the machine SILENTLY,
at test time, on whatever host ran the tests; and each quietly turned "the toolchain has no
promtool" into a green run, which is how the toolchain stayed without one. Missing promtool
is exit 3 (UNWIRED, never green); a PROMTOOL= that does not resolve is exit 2.

ONE WIRING: conformance/tests/l0-static/alert-rules-unit-tested, NOT a gate stage. The
`prom-rules` stage (ci/prom-rules.sh + its Makefile target + ci/gate.sh entry) is deleted:
it is the bespoke-per-check pattern b9f8807 had just removed from ci/l0.sh, and the
declarative replacement gives what a hand-written stage cannot — selection by
`conformance.home.k3s/scope: "whole-repo"` (asserted: the test is selected even under a
requirement filter matching nothing), requirement-attributed JUnit/coverage/ledger evidence,
and run_l1_offline.py's rule that a SKIP of a scope-declaring check is the runner's rc 3.
The scope declaration is load-bearing here: touched-requirements.sh keys apps/* to nothing
and platform/monitoring/* to R-OBS-01/02, never R-OBS-04, so a new untested PrometheusRule
would otherwise be gated by nothing.

The check asserts the pins and their lockstep, that the image bakes and records promtool,
that the promtool IN USE reports exactly the pin (a FAILURE, not a skip: the wrong engine
still prints SUCCESS), that exactly ONE harness exists and none fetches an engine, that
every rule-unit test file passes, and — on every run — that the harness CAN fail, by putting
a synthetic rule through it twice with one expectation mutated.

MISSING TESTS ARE A RATCHET, deliberately, and the reasoning is in the check's header. 51 of
58 alert rules have no unit test. Failing on all of them makes the gate red until ~51 tests
exist, so it gets disabled and protects nothing; merely reporting is exactly how the CNPG
tests came to be enforced by nothing. So the gap is DECLARED in the check as a shrink-only
register, printed with counts every run (13 alert files nothing tests at all; 35 named
alerts inside covered files that nothing asserts on), and it cannot grow: a new
PrometheusRule, or a new alert in an already-covered file, FAILS without a test, and a stale
entry FAILS too. A missing test is never an exit-3 skip — SKIPPED is reserved for the one
genuine environment gap, promtool being absent.

PROVEN, in the rebuilt image (promtool 3.14.0):
  * both suites pass through the single harness: cnpg 6 rules, monitoring 36 rules, SUCCESS.
  * the check passes: 95 assertions, 7/58 alert rules unit-tested, 13 declared gap files.
  * NOT VACUOUS, twice over. Shortening CNPGPrimariesCoLocated's `for: 30m` to 5m turns it
    red (pending expected at 25m/29m, got firing), reverted. Adding an untested
    PrometheusRule (platform/kubevirt/probe-alerts.yaml) turns it red on the ratchet,
    removed. The harness's own mutation self-test runs on every invocation.
  * the scope mechanism selects it: `run.sh --layer l0 --requirements r-nosuchreq-99
    --include-scope whole-repo --print-selection` lists alert-rules-unit-tested.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 00:02:27 -05:00
jakman85 55b90849d1 merge: the arrs had no download client at all, and now agree on paths
Investigating the path disagreement found the larger defect: sonarr and
radarr each returned [] from /api/v3/downloadclient — Completed Download
Handling had never been wired up, not merely broken. Both now have
qBittorrent configured plus a remote path mapping /downloads/ ->
/media/downloads/, proven with a real completed torrent and a negative
control: same torrent, same client, mapping absent gave an unresolvable
/downloads/... and mapping present resolved the real file at inode 36144.

This state lives in each arr's database, NOT in git — a rebuild from git
does not restore it. Recorded in DECISIONS-LOG with the recommendation to
close it properly via path agreement instead.
2026-09-13 00:01:55 -05:00
jakman85andClaude Opus 5 b7341b329b fix(runbooks): declare whole-repo scope, drop the bespoke l0 stage, +cnpg-primary-spread
Three corrections after rebasing onto main.

1. THE GUARD CAUGHT A REAL ONE ON ITS FIRST CONTACT WITH main. main had gained
   `CNPGPrimariesCoLocated` (platform/cnpg/alerts.yaml) pointing at
   /runbooks/cnpg-primary-spread, with no runbook. The check failed the rebase exactly as
   designed — 46 annotations, 35 names, 34 present — naming both the missing file AND the
   missing configMapGenerator entry. runbooks/cnpg-primary-spread.md now exists: what
   co-located primaries actually cost here (blast radius, not an outage, because
   instances: 3 + required anti-affinity), why the drift is uncorrectable-by-declaration
   (CNPG has no primary-placement field), the readiness gate that keeps the rule quiet
   during a real node outage, and the ~9s-per-database switchover rebalance, one cluster
   at a time. The 2026-09-07 outage in which all three primaries sat on one node is the
   reason the alert exists at all.

2. WHOLE-REPO SCOPE, NOT A HAND-WRITTEN STAGE. main's refactor replaced ci/l0.sh's
   per-check "UNCONDITIONAL" stages with a declaration:
   `conformance.home.k3s/scope: "whole-repo"`, unioned into every gate run by Stage 3b's
   `--include-scope`. My +38-line Stage 3f is removed (ci/l0.sh is now byte-identical to
   main) and the annotation is declared instead. Assertion (7) of the check was rewritten
   to match: it no longer asserts its own path appears in ci/l0.sh — it asserts the test
   DECLARES the scope and that ci/l0.sh still passes --include-scope, since neither half
   alone makes the check load-bearing. That matters here because
   ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02", not
   R-OBS-04, and maps platform/kubevirt/* and runbooks/* to nothing at all.

3. A FACTUAL CORRECTION IN ALL 35 RUNBOOKS. Every page claimed `dev.sh` defaults its
   current-context to k3s-dev. main's `9b4744d fix(devcontainer): dev.sh defaults to
   prod` made that false, and a runbook that misstates which cluster a command talks to is
   worse than one that says nothing. The pages now say what is durably true: the default
   has already changed once, so naming `--context k3s-prod` is what makes a command mean
   the same thing next month, and the per-invocation banner is there to be read. The
   check's own failure message was corrected the same way.

Re-verified after all three: check green at 374 assertions (46 annotations, 35/35 runbooks
present, 35/35 projected), re-applied to prod, and re-probed in-cluster through the
Gateway — node-down, keycloak-down, keycloak-client-secrets, cnpg-archiving,
longhorn-faulted, vpn-down, node-io and cnpg-primary-spread all HTTP 200 with
text/plain; charset=utf-8 and 8.9-12.1 KB of real body; /runbooks/ index 200; an unknown
name 404s with a pointer rather than a blank page; and homepage still answers the apex `/`,
confirming Gateway API longest-path precedence.

Full merge gate: PASS (spec-immutability, lint, l0, render-golden, l1 all green;
prom-rules SKIPPED for want of promtool in the image; L2/L3 SKIPPED, no cluster).

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 00:00:24 -05:00
jakman85andClaude Opus 5 2f38b8156b fix(arr): wire qBittorrent as a download client + Remote Path Mappings
qBittorrent reports finished releases at /downloads/<name>; sonarr and radarr
see the same inodes at /media/downloads/<name>, so Completed Download Handling
could never resolve what the client handed it. Configured, on prod, via each
app's own HTTP API:

  * download client qBittorrent -> qbittorrent.qbittorrent.svc.cluster.local:8080
    (it did not exist in EITHER arr: both /api/v3/downloadclient returned [])
  * Remote Path Mapping /downloads/ -> /media/downloads/ for that host

Both built from each app's /downloadclient/schema, matched on identity before
writing, and re-running creates nothing. downloadclient/testall reports
isValid=true in both. Proven with a negative control: Sonarr's queue showed
outputPath=/downloads/... with the mapping deleted and /media/downloads/...
with it present, for the same completed torrent.

This state lives in sonarr.db / radarr.db on the config PVCs, NOT in git — a
rebuild from git alone does not restore it. Recorded as a wart, with path
agreement (repointing SavePath to /media/downloads) recommended as the durable
fix and put to the operator rather than taken unilaterally.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 23:57:56 -05:00
jakman85andClaude Opus 5 44983095e1 feat(runbooks): make the 42 alert runbook_url annotations resolve
Every platform-health alert rule carries
`runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations over 34
distinct names, across platform/monitoring/alerts.yaml,
platform/monitoring/pvc-capacity-alerts.yaml, platform/cnpg/alerts.yaml and
platform/kubevirt/alerts.yaml. There was no runbooks/ directory and no HTTPRoute
serving that path: an in-cluster probe of
https://home.keithjackson.dev/runbooks/node-down returned HTTP 404. Every alert that
reached a human handed them a dead link at the exact moment the documentation is read,
and the annotation made it look solved — the same green-surface-over-absent-mechanism
class as the import Job that reported success while storing placeholder secrets.

No alerts.yaml is touched and no runbook_url value changes. The existing URL scheme is
made to resolve instead.

WHAT LANDS
- runbooks/<34>.md — one per name, each grounded in this cluster's real topology and,
  where one exists, its real incident history: the 2026-09-07 k3s-prod-1 etcd I/O
  starvation (measured PSI 11.36 vs 0.75/1.29, the exact journal lines, the four
  mechanisms proven inert), the Longhorn reboot tax and the freeze/thaw of
  concurrent-replica-rebuild-per-node-limit, Cilium's once-resolved egress interface
  cache (Ifindex 3 against a live 1780) and the absence of OSS failover, the CNPG
  38-hour ContinuousArchiving=False with its measured 6.8 MiB/h disk-full clock and the
  2026-09-12 aggregation-scope fix, the 2026-09-06 Keycloak lockout in which all 12
  clients held literal placeholder secrets behind an entirely green cluster, the Velero
  R2 x-amz-tagging NotImplemented incompatibility and its v1.14.3-rc.2 pin, and swap
  re-enabling itself across reboots via systemd-gpt-auto-generator.
  Each page states what the alert means HERE, the first diagnostic commands, likely
  causes ordered by probability with a remediation each, and when to escalate or stop.
- Every page uses FULLY-QUALIFIED resource names and says why: three CRDs register the
  kind `Backup` here, the bare short name resolves to Longhorn's (zero objects), so
  `kubectl get backup -A` prints "No resources found" on a cluster holding 6 Velero and
  27 CNPG backups. Acting on that false negative is how a CLI quirk becomes data loss.
- Every page states that kubectl only works via
  `infra/devcontainer/dev.sh run bash -c '...'` with an explicit `--context k3s-prod`,
  because dev.sh defaults its current-context to k3s-dev.
- Pages also record which alert groups are INERT today (measured 2026-09-13: no
  Longhorn, cert-manager, Flux-controller or Hubble scrape job exists, so 13 rules
  evaluate against nothing) so nobody reads their silence as health.

SERVING — runbooks/serving/ plus clusters/base/runbooks.yaml
A two-replica nginx-unprivileged serving a ConfigMap generated from runbooks/*.md, at
the apex `/runbooks` prefix, in its own non-gating post-platform Flux tier
(dependsOn platform-config, wait:false) — the platform/homepage pattern, file for file.
- The service catalog cannot express this route: ingressRoutes[].host is pattern-bound
  to a single DNS label plus the apex, so the bare apex is unrepresentable, and there is
  no path field.
- Rejected: redirecting to the Gitea web UI. A runbook is read when the platform is
  broken, and Gitea is an app-tier workload whose own dependencies several of these
  runbooks document; Gateway API's RequestRedirect also has no "append .md" modifier,
  so it would need 34 hand-maintained ReplaceFullPath rules.
- configMapGenerator maps each file to an EXTENSIONLESS key (node-down=node-down.md),
  so /runbooks/node-down is a plain static lookup with no rewrite rule. Served as
  text/plain: the namespace has DNS-only egress, so a CDN markdown renderer could not
  load anyway.
- The tier declares NO postBuild.substituteFrom, deliberately: envsubst would run over
  the prose and silently empty every ${...} shell example (it fails open — the 2026-09-01
  emptied Cilium k8sServiceHost). The HTTPRoute omits `hostnames:` and inherits the
  https-apex listener's, so nothing needs substituting.

AUTH — PUBLIC, deliberately
auth-mode=none. Gating these pages behind oauth2-proxy/Keycloak would make
runbooks/keycloak-down unreadable when Keycloak is down and
runbooks/keycloak-client-secrets unreadable during the exact 2026-09-06 failure mode.
An operator who cannot log in is this route's primary reader. Content is operational
procedure only, no secrets. The rationale is recorded in an annotation on the HTTPRoute
so the route is left OUT of the Gateway ext-authz attachment when forward-auth
enforcement lands.

PROVEN — in-cluster probe through the Gateway on prod
  /runbooks/node-down        HTTP 200   8613 bytes  text/plain; charset=utf-8
  /runbooks/keycloak-down    HTTP 200   9378 bytes  text/plain; charset=utf-8
  /runbooks/cnpg-archiving   HTTP 200  11729 bytes  text/plain; charset=utf-8
  /runbooks/longhorn-faulted HTTP 200   9135 bytes  text/plain; charset=utf-8
  /runbooks/vpn-down         HTTP 200  11092 bytes  text/plain; charset=utf-8
  /runbooks/backup-failure   HTTP 200   9604 bytes  text/plain; charset=utf-8
HTTPRoute Accepted by io.cilium/gateway-controller on sectionName https-apex; the
`/runbooks` prefix wins over homepage's apex `/` by Gateway API longest-path precedence.

GUARD — conformance/tests/l0-static/alert-runbooks-resolvable (R-OBS-04)
363 offline assertions. It checks BOTH halves, because the regression is silent in both
directions: an alert with a new runbook_url and no runbook links to a 404, AND a runbook
committed without its configMapGenerator entry is in git, reviewed, merged, and still
404s in the cluster. Also asserts a quality floor so a stub cannot make the link look
fixed, and that route/listener/nginx prefix agree with the annotations.
Proven to fail, then reverted: a bogus runbook_url (2 FAIL), a runbook removed from the
generator (1 FAIL), and a 31-byte stub (7 FAIL).

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 23:49:58 -05:00
jakman85 4865734c5f merge: alert when a backup schedule has NEVER run, and when one is paused
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
VeleroNoRecentBackup could not see the failure that actually happened:
velero-daily sat paused having never produced a successful backup, so
velero_backup_last_successful_timestamp{schedule=...} did not exist and
time() - <absent series> yields no series. Prod had no volume backups and
nothing alerted, further masked by a resources-only schedule succeeding
daily. No velero_* metric describes a Schedule OBJECT, proven by scraping a
throwaway paused Schedule and finding zero lines, so this exposes the CRs
via kube-state-metrics customresource_state and keys on
defaultVolumesToFsBackup — the property, not a name list. Adds the repo's
first prometheus rule-unit-test harness, mutation-checked both ways.
2026-09-12 23:49:26 -05:00
jakman85 b9f8807aa6 merge: whole-repo L0 invariants are DECLARED, not hand-wired per test
The gate had been accreting one bespoke UNCONDITIONAL stage per whole-repo
L0 test (3c/3d/3e), each repeating the same ~12-line rationale and each one
something the next author had to remember to write. Forget it and the test
runs only when some unrelated requirement happens to be touched.

Tests now declare conformance.home.k3s/scope: whole-repo and run.sh unions
them in via --include-scope; the three stages are deleted. 10 of 13 tests
are whole-repo, 3 stay path-scoped with the path map widened instead
(infra/* -> R-ENV-01, ci/* -> R-TEST-06). A new meta-test makes the bespoke
pattern fail the gate if it creeps back, and a change under
conformance/tests/<test>/ now attributes that test's own requirements, so
weakening a test is gated by what it certifies.

rc-3 (SKIPPED is UNWIRED, not green) is preserved and demonstrated; the
incident narratives moved into the tests they describe rather than being
deleted. Gate cost for a scoped change rises 20.7s -> 41.8s, a ceiling any
empty-touched-set change already paid.
2026-09-12 23:49:26 -05:00
jakman85andClaude Opus 5 7057bbb711 fix(monitoring): alert when a Velero schedule has NEVER backed up, or is paused
VeleroNoRecentBackup could not fire for the failure mode that actually happened.
`(time() - velero_backup_last_successful_timestamp{schedule!=""}) > 24*3600` is the
right SHAPE — per-schedule, un-aggregated — but Velero only creates that series once
a backup for the schedule SUCCEEDS, and `time() - <absent series>` yields no series.
So it detects "backed up once, then stopped" and is structurally INERT on "never
backed up". Prod ran with `velero-daily` (the schedule carrying VOLUME DATA) paused
and never-successful: no volume backups at all, nothing alerted, and it was masked
because `velero-daily-resources-only` completed cleanly every 03:00, so every other
backup signal read green while only Kubernetes objects were protected.

A PromQL-only fix is impossible, verified rather than assumed: velero v1.18.1 exports
35 velero_* metric names and every one carrying a `schedule` label is created by a
backup ATTEMPT, not by the Schedule object. A throwaway `paused: true` Schedule on
prod produced ZERO velero_* series, and nothing exposes `spec.paused`. The fact is
not in the TSDB, so the Schedule CRs are exported via kube-state-metrics'
custom-resource-state collector — the path this repo already uses for
kube_node_labels — as kube_velero_schedule_{created,paused,protects_volume_data,info}.
Proven by running KSM v2.20.0 with this exact config against prod's API: the CRD
list/watch rule the chart adds is MANDATORY, not cosmetic — without it KSM emits the
CRS metrics as an empty set, silently.

Three rules, all keyed on PROPERTIES rather than schedule names:
  * VeleroScheduleNeverBackedUp (critical) — a Schedule older than 36h with no
    successful backup ever. The 36h grace comes from creationTimestamp so a newly
    added schedule is not paged before its first run is even due.
  * VeleroSchedulePaused (warning) — the direct root-cause signal; a paused backup
    schedule must never be silently indefinite.
  * VeleroNoVolumeDataBackup (critical) — THE ANTI-MASKING RULE. Fires unless some
    schedule with spec.template.defaultVolumesToFsBackup=true succeeded in 36h, so a
    healthy resources-only schedule can never make the platform look protected.
    Reading the property that DEFINES volume protection (not a name list) matches
    platform/velero/volume-policies.yaml and survives renames. `or vector(0)` is
    load-bearing: `count(<empty>) == 0` returns no series and would be inert —
    absence of the exporter is deliberately the alarm.

Adds the repo's first Prometheus rule-unit-test harness (platform/monitoring/tests/,
wired as the `prom-rules` gate stage). It flattens the COMMITTED PrometheusRule CRs —
never a copy — and pins every rule in the group both ways: quiet on today's healthy
prod state, firing on the reconstructed bad state, with `for:` durations asserted.
Two mutation checks confirm the tests are not vacuous: dropping `or vector(0)` and
breaking the `unless on (schedule)` join each turn the suite red. This is the tool
that makes both of this repo's recurring alert bugs — permanently-firing and
structurally inert — provable offline, which neither yamllint, Kyverno, nor a healthy
cluster can do.

Proof of the fix is by construction; the live metrics cannot show it any more, because
`velero-daily` was unpaused on 2026-09-12 and has since completed 2492/2492 items.
The throwaway probe Schedule and the probe KSM deployment were deleted and the
deletion confirmed; `velero-daily` remains unpaused.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 23:45:15 -05:00
jakman85andClaude Opus 5 5fa278f61c fix(ci): declare whole-repo scope instead of hand-writing a gate stage per invariant
R-TEST-03 runs the conformance subset a change TOUCHES. That has one blind spot, and it
is the worst-shaped kind: an invariant that quantifies over the whole tracked tree — or
over subtrees ci/touched-requirements.sh's coarse path map keys to NOTHING (apps/, infra/,
most of platform/) — cannot be selected BY the requirements a change touches, because the
change most likely to break it attributes no requirement at all. Requirement-scoped
selection then filters the check out of exactly that change.

The repo's answer had been a hand-written "UNCONDITIONAL" stage per such check in
ci/l0.sh: 3c flux-postbuild-substitution, 3d keycloak-client-secret-reconciler, 3e
bulk-tier-distinct-from-replicated. Three near-identical 12-line rationales, three
hardcoded check.sh paths, three copies of the same rc 0/3/* case block — and worst, a
mechanism that failed SILENTLY, because whoever added the NEXT whole-repo invariant had
to REMEMBER to write a stage, and if they forgot the check ran only when some unrelated
requirement happened to be touched.

A test now DECLARES the scope of its invariant and the runner enforces it:

    metadata:
      annotations:
        conformance.home.k3s/scope: "whole-repo"

* conformance/lib/common.sh  conf_dirs_for_scope(s) resolves the annotation. Carries the
                             general rationale, once, instead of per stage.
* conformance/run.sh         --include-scope SCOPE unions declaring tests into select_dirs,
                             still bounded by --layer. New --print-selection resolves the
                             selection and exits, running nothing.
* lib/run_l1_offline.py      --unwired-scope: a SKIP of a declaring test exits 3, not 0.
                             --summary-out: one step-summary line per executed check.
* ci/l0.sh                   Stage 3b gains --include-scope whole-repo and an rc-3 arm.
                             Stages 3c/3d/3e DELETED (-150 lines).
* ci/l1.sh, ci/conformance.sh  pass --include-scope too, so the next whole-repo invariant
                             at ANY layer needs only the annotation.

rc 3 SEMANTICS PRESERVED EXACTLY. A whole-repo check reporting "nothing to audit" is
reporting that the tree it audits has VANISHED. The deleted stages called that "not a
pass" and set the gate's rc=3; the runner now returns 3 and ci/l0.sh maps it to rc=3.
A missing check file still degrades to not-green, never to silence.

THE REASONING WAS RELOCATED, NOT DELETED. Each stage's incident narrative moved into the
header of the test it describes, beside its scope annotation: the Cilium k8sServiceHost
empty-substitution CNI outage (2026-09-01), 12 unusable Keycloak clients behind a green
cluster (2026-09-06), and a multi-TB bulk volume landing on the 3-replica tier.

AUDITED ALL 13 EXISTING l0-static TESTS, individually. 10 declare whole-repo scope; 3 are
DELIBERATELY left requirement-scoped and say so in their own headers:
  * placeholder-guard-coupling — inputs are bootstrap/ + clusters/, both already mapped to
    a requirement it declares. Scoping cannot hide a breaking change.
  * dev-nodes-provisioned-as-code — inputs are infra/ and nothing else.
  * spec-immutability-enforced — inputs are ci/ and the read-only spec repo.
The latter two were nonetheless under-gated because the PATH MAP was incomplete, which is
the right place to fix a path-scoped invariant: infra/* -> R-ENV-01 added, and R-TEST-06
added to .gitea/*|ci/* (it is no longer an uncovered ID — spec-immutability-enforced
carries the label, so attributing it no longer false-blocks the coverage gate).

NEW conformance/tests/l0-static/whole-repo-invariants-wired (R-TEST-03) keeps the
declarative mechanism honest for ALL declaring tests at once, replacing the three tests'
individual "ci/l0.sh greps my own check.sh path" self-assertions. It proves the union
EMPIRICALLY: with a requirement filter matching no test the L0 selection is empty, and
with --include-scope it is every declaring test; a scope-declared synthetic SKIP exits 3
while an undeclared one exits 0; and a ci/*.sh that hardcodes a check.sh path must belong
to a test that also declares the scope, so the bespoke pattern cannot creep back.

TASK 2 — a changed test attributes the requirements it CERTIFIES.
ci/touched-requirements.sh gains detection rule 3: a change anywhere under
conformance/tests/<test>/ attributes that test's own conformance.home.k3s/R-* labels, read
from its chainsaw-test.yaml. The content scan could not close this — it reads only
added/removed lines, so it sees a change to the LABEL BLOCK but not to the test BODY,
which is exactly the case that matters. Labels are read from HEAD or, failing that, from
the BASE, so DELETING a test still attributes what it used to certify; otherwise removing
the only verifier of a requirement is the cheapest way to make the gate stop checking it.
Whole-line `#` comments are dropped first and *.md is untouched, honouring the existing
"reference, not declaration" rules.

Both changes are fail-closed: every one of them can only ADD tests or requirements to a
run, never remove one.

TESTS: new ci/tests/touched-requirements-test-attribution.test.sh (15 assertions) covers
rule 3 — body-only edit, nested-fixture walk-up, test deletion, header-comment R-IDs not
attributed, non-test paths unaffected, and both new path-map entries. The two existing
suites still pass (39 assertions). `make -C ci test` runs all three.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 23:45:06 -05:00
jakman85 78d76fa089 merge: qBittorrent can finally write where the arrs can see it
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
qBittorrent's SavePath was /downloads and no such directory existed in the
container, so the download client had no working destination and nothing
downstream could see its output. It now mounts the same NFS export the arrs
use, narrowed to the downloads subtree via subPath — one mount, so the
EXDEV hardlink trap the bulk-tier test guards stays closed. PUID/PGID are
load-bearing: the export is owned 1000:1000 and the image's default 911 is
denied.
2026-09-12 23:36:48 -05:00
jakman85 9e7d919d12 merge: CNPG primaries spread one per node, and the condition is now reported
All three primaries had re-concentrated onto prod-1 (the node with known
etcd/Longhorn I/O contention), so losing it meant three simultaneous
failovers. Two switchovers rebalanced them; semaphore-db deliberately stays
on prod-1 as the lightest writer, saving a third write interruption.

CNPG offers no role-aware scheduling — targetPrimary exists only under
status — so the placement is CURRENT, NOT DURABLE: the next failover or
rolling update can re-concentrate it. What is durable is CNPGPrimariesCoLocated,
which reports the condition instead of leaving it to be found by hand.
2026-09-12 23:36:48 -05:00
jakman85andClaude Opus 5 9b4744d450 fix(devcontainer): dev.sh defaults to prod — dev is decommissioned
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
Every unprefixed './dev.sh run kubectl ...' pointed at the dev cluster,
which was decommissioned on 2026-09-12. The banner cheerfully named dev and
the command then hung or errored against a context that no longer answers,
with nothing saying the cluster is gone. I hit this repeatedly tonight.

CLUSTER=dev is still accepted rather than removed, because the dev VM disks
were deliberately kept so the environment is reversible — but it now warns.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 23:33:12 -05:00
jakman85andClaude Opus 5 78bc53afc7 feat(qbittorrent): attach the downloads area of the bulk export — the chain was broken end to end
qBittorrent was the one member of the download chain with NO media volume.
Its live config said

    Session\DefaultSavePath=/downloads/
    Session\TempPath=/downloads/incomplete/
    Downloads\SavePath=/downloads/

while `ls /downloads` in the container returned "No such file or directory".
The catalog entry declared exactly one volume — config at /config on longhorn —
and carried a comment deferring the downloads volume "until OQ-2 is resolved".
OQ-2 WAS resolved (schema 1.5.0 static bulk NFS PVs); plex, sonarr, radarr and
nextcloud all took their mounts and qBittorrent's was simply never added. So the
download client was writing, or failing to write, a path that did not exist —
and anything it did create would have been invisible to sonarr/radarr, lost on
every pod recreation, and consuming the node's ephemeral disk until then.

WHAT THIS ADDS to catalog/qbittorrent.yaml (rendered into apps/wave-0):

  * storage[downloads]: class bulk, export /mnt/Storage/plex, subPath downloads,
    mountPath /downloads, 9Ti advisory. ONE mount of the SAME export sonarr,
    radarr (rw) and plex (ro) already consume, narrowed to the handoff subtree.
  * env PUID=1000 / PGID=3000. LOAD-BEARING, not cosmetic: the container ran as
    uid 911 and /mnt/Storage/plex/downloads is 0755 owned 1000:1000, so the
    export would have mounted cleanly and then failed every write with EACCES.
    Measured both ways on prod: a write as uid 1000 succeeded, the same write as
    uid 911 was denied.
  * podSecurityContext.fsGroup 3000, for the LONGHORN config PVC only, exactly as
    sonarr/radarr set it — fsGroup is a no-op on an in-tree NFS volume.

WHY subPath AND NOT THE EXPORT ROOT AT /media (sonarr/radarr's shape): a download
client has no business writing the library. The arrs need the root mount because
THEY hardlink downloads/ -> tv/|movies/ and link() is a same-MOUNT operation;
qBittorrent performs no such link, so narrowing to downloads/ removes movies/,
tv/ and photos/ from its mount namespace without removing any operation it
performs. It is still ONE mount of the export, so the L0 assertion that no
workload mounts one export twice (the EXDEV trap) holds — and the arrs' own
link() stays intra-mount. Proven live: nlink 1 -> 2, same inode, no EXDEV, every
step executed as uid 1000.

NETWORK: nothing to grant and nothing to steer. An NFS PersistentVolume is
mounted by the NODE KERNEL in the host netns — the established socket to
192.168.2.102:2049 is on the node's own LAN address, the pod netns has no socket
to 2049 at all, and no CiliumNetworkPolicy on this cluster names that host or
port. `ip route get 192.168.2.102` picks eno1 even on the tunnel-owning node, and
Cilium's own egress map lists 192.168.0.0/16 as an Excluded CIDR for every
vpn-class pod while 0.0.0.0/0 goes to the gateway. Public egress still exits the
PIA address; the VPN guarantee is untouched.

NOT CLOSED BY THIS COMMIT, deliberately: absolute paths still differ between this
pod (/downloads/x) and the arrs (/media/downloads/x), so Completed Download
Handling needs one Remote Path Mapping per arr. That lives in the arrs' own
databases, not in git.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 23:32:03 -05:00
jakman85andClaude Opus 5 505083fded feat(cnpg): alert when >1 CNPG primary shares a node — the drift CNPG cannot prevent
`instances: 3` + `podAntiAffinityType: required` fixed the INSTANCE spread but moved
the PRIMARY concentration from k3s-prod-2 to k3s-prod-1: all three primaries
(keycloak-db-2, nextcloud-db-2, semaphore-db-2) ended up on one node, so losing it
meant three simultaneous failovers.

There is NO declarative fix, established from the CRD installed on prod (CNPG 1.30.0,
clusters.postgresql.cnpg.io/v1) rather than from memory. Every spec.affinity field and
spec.topologySpreadConstraints applies to "all the cluster's pods" and is not role-aware;
spec.replica.primary names a Cluster object in a distributed topology, not a node;
primaryUpdateMethod/switchoverDelay/primaryLease govern how a promotion happens, never
where. `targetPrimary` exists only under STATUS, never spec — so `cnpg promote` is an
imperative status patch by construction and there is nothing to put in git. The obvious
workaround is worse than nothing: pod anti-affinity is only ever
...IgnoredDuringExecution while CNPG promotes by relabelling a RUNNING pod, and with
3 instances on 3 nodes it would leave the third instance unschedulable.

So this commits the part that IS durable — visibility — and rebalances live.

CNPGPrimariesCoLocated joins CNPG's own answer to "who is primary" (the <cluster>-rw
Service selects only the primary) against the node: kube_endpointslice_endpoints carries
endpoint_nodename + targetref_name, which two label_replaces lift into node/pod. Neither
family works alone — cnpg_* metrics have no node label, and kube_pod_labels is EMPTY here
(no kube-state-metrics label allowlist), so cnpg.io/instanceRole is not in Prometheus at
all. `* on (namespace, pod) group_left () cnpg_collector_up` keeps an unrelated `*-rw`
Service from triggering it, and uses the one metric present for EVERY instance rather than
the cnpg_pg_stat_* family that is missing for nextcloud-db.

A Ready-node gate stops this becoming another permanently-firing alert: with three
clusters on three nodes a node loss NECESSARILY co-locates two primaries, so the rule
only speaks when the spread is actually achievable, and re-arms when the node returns.

Proven both ways. Live: the expr returned {node="k3s-prod-1"} => 3 before, empty after,
with the inner join still resolving all three primaries (quiet because false, not blind);
loaded into prod's Prometheus as a throwaway `for: 1m` twin it went pending at +70s and
firing at +130s. alerts.promtool-test.yaml pins four cases under promtool 3.14.0 (the
exact version prod runs) and is mutation-checked — dropping the join, dropping the gate
or shortening `for:` each makes a test FAIL.

Rebalanced live, one cluster at a time: keycloak-db -> k3s-prod-2, nextcloud-db ->
k3s-prod-3, semaphore-db deliberately LEFT on k3s-prod-1 (lightest writer, and that node
has the known etcd/Longhorn I/O contention) — two switchovers, not three. Write outage
measured with a `select txid_current()` probe, not estimated: 9.4s and 9.43s, one
contiguous burst each. All three back to readyInstances: 3 with
synchronous_standby_names = ANY 1 (...) and both standbys streaming/quorum.

The placement is CURRENT, NOT DURABLE, and the alert is the point: the next failover or
rolling update can re-concentrate the primaries, and now that gets reported instead of
discovered by hand.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 23:32:03 -05:00
jakman85 df77affc23 merge: the nextcloud database has never been monitored
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
cnpg_metrics_exporter lacked CONNECT on the nextcloud database: an
out-of-band REVOKE materialized its datacl and dropped PUBLIC's implicit
CONNECT, which keycloak and semaphore still inherit. CNPG's API cannot
express a database-level grant, so this follows the repo's existing
level-triggered CronJob reconciler pattern rather than a one-shot Job that
a rebuild would silently skip. CONNECT only; pg_monitor already carries the
read access.
2026-09-12 23:28:23 -05:00
jakman85 501d587541 merge: CNPG DR alerts scoped to the cluster, and the hint that lied
CNPGNoRecoverabilityPoint fired permanently on all six CNPG replicas because
only the primary owns the backup catalogue; CNPGNoRecentBaseBackup had the
same latent defect, held back only by a > 0 guard. Both now aggregate pod
away and key on the cluster. The remediation text also told the on-call to
run 'kubectl get backup', which reports no resources here.
2026-09-12 23:28:23 -05:00
jakman85andClaude Opus 5 bacb4c61ee fix(cnpg): the alert told the on-call to run the command that lies
CNPGNoRecentBaseBackup's own remediation hint said `kubectl get backup`.
On this cluster longhorn.io, postgresql.cnpg.io and velero.io each register
kind Backup, so the bare short name resolves to Longhorn's empty set and
prints 'No resources found' while CNPG backups exist. An on-call following
the alert that fires BECAUSE backups look stale would have been told, with
authority, that no backups exist at all — and might re-bootstrap a database
over good ones. Fully qualify it.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 23:28:12 -05:00
jakman85andClaude Opus 5 7880d84bce fix(cnpg): grant cnpg_metrics_exporter CONNECT on the nextcloud database
The CNPG metrics exporter could not CONNECT to the `nextcloud` application
database, so every user-defined collector query against that target failed on
all three instances:

  FATAL: permission denied for database "nextcloud" (SQLSTATE 42501)
  DETAIL: User does not have CONNECT privilege.

The whole cnpg_pg_stat_* / pg_settings / pg_postmaster family was therefore
MISSING for nextcloud-db while the scrape target still reported up=1 — the blind
spot platform/cnpg/alerts.yaml's CNPGMetricsCollectorErroring was written to
catch. It had been firing on all three pods; measured before this change,
cnpg_pg_settings_setting had 765 series for keycloak, 765 for semaphore-db and
ZERO for nextcloud-db. Nextcloud's database had never been monitored.

Root cause is the live ACL, not the spec — all three Cluster specs bootstrap
identically:

  nextcloud : datacl = {nextcloud=CTc/nextcloud}   <- explicit, no PUBLIC entry
  keycloak  : datacl = NULL
  semaphore : datacl = NULL

A NULL datacl means PostgreSQL's defaults, which include CONNECT for PUBLIC, so
keycloak and semaphore admit the exporter only by inheriting PUBLIC's implicit
CONNECT. On `nextcloud` something ran the equivalent of REVOKE ALL ON DATABASE
nextcloud FROM PUBLIC, which materialises datacl and drops that implicit entry.
That REVOKE is in neither this repo nor DECISIONS-LOG.md — it was applied out of
band, which is why the remedy has to be declarative rather than another hand-run
psql.

CNPG 1.30.0 cannot express a database-level GRANT (verified against the live
CRDs): `Database` has no ACL field (`allowConnections` is the datallowconn flag,
not a per-role grant), and managed.roles/DatabaseRole cover role attributes plus
`inRoles` membership only — and membership would hand the exporter the owner's
CREATE/TEMP and all table access to buy one CONNECT. postInitApplicationSQL runs
once at bootstrap, so it cannot repair the already-bootstrapped instances that
are actually broken. So this follows the pattern the repo already uses for state
Flux cannot reach: the level-triggered CronJob reconciler alongside
platform/keycloak/client-secret-reconciler.yaml. A one-shot Job is deliberately
avoided — a completed Job never re-runs, so a rebuild or PITR restore would
silently reopen the blind spot.

Exactly one privilege, and nothing else:

  GRANT CONNECT ON DATABASE nextcloud TO cnpg_metrics_exporter;

giving `cnpg_metrics_exporter=c/nextcloud` — CONNECT only, no CREATE, no
TEMPORARY, no ownership, no superuser, no table or schema grants. Nothing more is
needed because CNPG already creates the role as a member of the built-in
pg_monitor role (verified on all three clusters), which carries the read access
the collector's pg_catalog queries want. It runs as the database OWNER, not a
superuser: enableSuperuserAccess stays off, and a database owner holds the grant
option on its own database.

Verified live on prod:
  * the grant issued under SET ROLE nextcloud, proving no superuser is needed;
  * permission-denied and "Error collecting user query" both went to 0 on
    nextcloud-db-1/-2/-3, i.e. every collector query now succeeds;
  * cnpg_pg_settings_setting now 765 series for nextcloud-db, matching the other
    two, and cnpg_pg_postmaster_start_time present on all three instances;
  * cnpg_last_error = 0 on all NINE instances, so keycloak-db and semaphore-db
    are unaffected and their datacl is still NULL;
  * convergence proven by revoking the grant again and letting the committed
    CronJob restore it, then re-running it against a converged database to show
    idempotence;
  * CNPGMetricsCollectorErroring, firing on all three pods beforehand, has
    cleared (`for: 6h` delays firing, not resolution).

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 23:22:27 -05:00
jakman85andClaude Opus 5 5e7ab65c65 docs: kubectl get backup lies on this cluster — ambiguous Backup short name
Three API groups register kind Backup (longhorn, cnpg, velero). The bare
short name resolves to Longhorn's empty set, so kubectl reports absence
while 27 CNPG backups and live Velero backups exist. A false negative on
'do I have backups' is the worst possible time to be wrong, so record the
rule: fully-qualified resource names only.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 23:15:06 -05:00
jakman85andClaude Opus 5 21491f901b fix(cnpg): scope DR alerts to the CLUSTER, not the pod — stop a permanent false positive
CNPGNoRecoverabilityPoint has been firing continuously on prod against six
pods — keycloak-db-1/-3, nextcloud-db-1/-3, semaphore-db-1/-3 — i.e. exactly
the REPLICAS of all three Clusters, while the databases themselves are fine:
every Cluster reports firstRecoverabilityPoint=2026-09-07T07:0x, a
lastSuccessfulBackup within the last few hours, ContinuousArchiving=True and
LastBackupSucceeded=True.

Root cause is aggregation scope, not archiving. `cnpg_collector_first_recoverability_point`
reads the backup CATALOGUE, which only the primary owns; a replica reports a
hard 0 for as long as it is a replica. The metric carries no cnpg.io/cluster
label (its label set on prod is container/endpoint/instance/job/namespace/pod),
so `max by (namespace, pod)` evaluated each replica as its own subject and an
`== 0` comparison then held forever. The rule went from correct to permanently
wrong the moment the Clusters went to `instances: 3`; nothing about the
expression changed.

WHY THIS IS WORSE THAN HAVING NO ALERT. An alert that is always firing has
zero information content: its value is identical in the healthy state and in
the failure state, so it cannot distinguish them. It is worse than silence in
two specific ways. First, it trains the operator to delete the notification on
sight, so the day a base backup genuinely stops the seventh identical email is
discarded with the other six — the alert actively MASKS the failure it was
written to catch, which for a DR alarm means discovering the loss at restore
time. Second, it is indistinguishable from a broken alert, so nobody can tell
whether the absence of new alerts means health or means the rule rotted. This
file exists because a real 38-hour archiving failure told nobody; a rule that
cries wolf about six healthy replicas rebuilds exactly the blindness it was
supposed to remove. This repo has been here before — see the permanently-firing
critical VPNTunnelDown post-mortem in platform/monitoring/alerts.yaml.

Fix: group by (namespace, job) and aggregate `pod` away. The CNPG operator
creates one PodMonitor per Cluster, named after the Cluster, in the Cluster's
namespace, so prometheus-operator derives job="<namespace>/<cluster>" —
keycloak/keycloak-db, nextcloud-db/nextcloud-db, semaphore-db/semaphore-db —
which identifies the CNPG Cluster uniquely. `job` rather than bare `namespace`
deliberately: namespace alone works today (one Cluster per namespace, verified
live) but would let two Clusters in one namespace mask each other, since the
healthy one's non-zero value would satisfy `max` for both. The alert now
states what its summary always claimed — this CLUSTER has no recoverability
point — and fires only when no instance of that Cluster reports one.

Proven live against prod Prometheus, read-only (promtool query instant):
  OLD  max by (namespace, pod) (...) == 0  -> 6 series, and 6 ALERTS firing
  NEW  max by (namespace, job) (...) == 0  -> empty
The new expression is not vacuous: forcing the metric to 0 for one cluster's
primary only yields exactly one series, {job="keycloak/keycloak-db"}, so a
genuine single-cluster loss of DR still alerts, per cluster. `promtool check
rules` passes on all 5 rules.

CNPGNoRecentBaseBackup carried the same defect and is fixed the same way.
`cnpg_collector_last_available_backup_timestamp` is also primary-only: on prod
the primaries read 3.1h/2.1h/0.1h old while all six replicas read 0, i.e.
time()-0 = ~56.7 YEARS. It was not firing only because the `> 0` guard was
load-bearing — a single clause was all that stood between this rule and six
more permanent false positives, too thin a margin for a DR alarm. Aggregating
`pod` away removes the dependence: `max` takes the primary's timestamp as the
cluster's answer, and the guard now means "this cluster has had at least one
backup". Verdict unchanged live (old empty, new empty).

The other three rules were audited and deliberately left on `by (namespace, pod)`:
  * CNPGWALArchiveBacklog — every instance reports archive_status and the shape
    is `> 0`, so a replica at 0 never fires and no pod can mask another; `pod`
    names the instance whose archiver is stuck.
  * CNPGWALVolumeNotRecycling — per-pod is REQUIRED, not merely safe:
    aggregating it away would compare one instance's WAL count against
    another's max_wal_size.
  * CNPGMetricsCollectorErroring — unchanged in meaning AND grouping. Its
    grouping is already correct: cnpg_last_error is genuinely per-instance, and
    prod proves it discriminates rather than tracking role — all three
    nextcloud-db pods report 1 while all six keycloak/semaphore pods report 0.
    It is a true positive (the nextcloud 42501 CONNECT grant) owned by separate
    work; nothing here silences or rescopes it.

The general rule, recorded in the file: `== 0`/staleness shapes must aggregate
`pod` away or every structurally-silent pod fires forever; `> 0` shapes are
safe per-pod and keep the instance name.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 23:09:53 -05:00
jakman85andClaude Opus 5 ebce6dfabb docs: the Flux->Gitea allowance is rendered and Flux-adopted — debt closed
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 22:58:39 -05:00
jakman85 eda4a9f674 merge: the Flux->Gitea allowance is rendered, not hand-applied
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
2026-09-12 22:51:35 -05:00
jakman85andClaude Opus 5 d617983e83 feat(gitea): render gitea-allow-flux-ingress — prod's GitOps source, in git at last
The CNP that lets flux-system's source-controller clone from the in-cluster Gitea
Service was APPLIED LIVE on prod on 2026-09-13 and existed nowhere in this repo. It
carried a provenance annotation saying so. Until now it could not be authored: the
gitea CNPs are GENERATED from the catalog entry, and the contract had no field for
"also admit this in-cluster client" (fixed in the previous commit).

Why it matters more than a missing allow normally would. prod's GitRepository points
at the Service, not a public name (clusters/prod/kustomization.yaml:
`url: http://gitea.gitea.svc.cluster.local:3000/Home-Network/home-network-k3s-impl.git`),
deliberately, so prod's GitOps depends on neither public DNS, Cloudflare, prod's own
Gateway nor ${BASE_DOMAIN}. But gitea-allow-gateway-ingress admits only
`fromEntities: [ingress]`, so source-controller's clone was dropped — and the
HTTPRoute that would have served the post-apex-flip public name instead lives in a
DOWNSTREAM tier that cannot apply while the source is down. Source down -> routes
never rename -> source stays down. Every `kubectl delete cnp` and every cluster
rebuild re-armed that deadlock, and it would have been extremely hard to diagnose
cold.

The rendered object is EQUIVALENT to the live one — same apiVersion/kind, same
name/namespace/labels, and a byte-identical spec (endpointSelector
app.kubernetes.io/name=gitea; fromEndpoints matchLabels
k8s:io.kubernetes.pod.namespace=flux-system; toPorts 3000/TCP), verified against
`kubectl --context k3s-prod -n gitea get cnp gitea-allow-flux-ingress -o json`. The
name matches, so Flux ADOPTS the live object in place: no spec change, so no policy
recompute and no flap on prod's own source. The only difference is that the rendered
object carries no `homelab.keithjackson.dev/provenance` annotation — the reasoning now
lives where the object is authored, in catalog/gitea.yaml, next to the field that
generates it.

Golden re-rendered with the canonical command; the whole apps/ diff is the one new
document.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 22:46:57 -05:00
jakman85andClaude Opus 5 1f20506829 feat(renderer): inClusterClients — authorable direct east-west ingress (schema 1.5.0)
A platform controller that dials a service's ClusterIP directly had NO authorable
form in the ServiceDefinition contract:

  * ingressRoutes renders the north-south allow only — gateway-ingress admits
    `fromEntities: [ingress]`, the Gateway's Envoy identity, and nothing else;
  * dependencies compiles edges only BETWEEN CATALOG SERVICES, from the source
    entry's own declaration.

Under the default-deny baseline (R-NET-01) such a client is therefore DROPPED, and
the only way to unblock it was a live `kubectl apply` of a hand-written CNP — a
cluster object outside git, one `kubectl delete` or one rebuild from coming back.

`inClusterClients: [{namespace, port, protocol?, name?}]` closes that. Each entry
renders ONE least-privilege ingress CiliumNetworkPolicy: this service's
endpointSelector, `fromEndpoints` matching ONLY `k8s:io.kubernetes.pod.namespace`,
`toPorts` carrying ONLY the declared port/protocol — never `fromEntities: all`.
Additive: the namespace default-deny keeps denying everything undeclared (R-NET-02),
and the entry is NOT a dependency edge, so it creates no wave ordering.

Shape notes:
  * selected by NAMESPACE, not pod labels: an upstream controller's labels belong to
    its chart, not to this platform's contract. The cost is explicit — every pod in
    that namespace is admitted on that port, so the field is for platform-controller
    namespaces (flux-system, monitoring), never a shared one. The schema description
    says so.
  * optional `name` (default: the namespace) only names the rendered policy,
    `<service>-allow-<name>-ingress`. That name is operator-facing (kubectl get cnp,
    a Hubble drop), and pinning it lets an allowance that already had to be applied
    live be authored under its EXISTING name, so Flux adopts the live object in place
    instead of creating a near-duplicate and leaving the original unmanaged.
  * one field, not a policy DSL: no arbitrary selectors, entities or CIDRs.

Covered by render.py --selftest (4c): name/namespace/selector/port shape, no
fromEntities widening, default-deny still `ingress: []`, the gateway allow not
displaced, `name` defaulting to the namespace, and nothing rendered when the field is
absent. No golden changes here — no catalog entry uses the field yet.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 22:46:57 -05:00
jakman85andClaude Opus 5 a863542bf7 docs: record the apex migration, the GitOps deadlock, and the live-only policy
Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 22:37:58 -05:00
jakman85andClaude Opus 5 f252ecec25 fix(gitops): prod's source is the in-cluster Service, not a public name
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
The apex flip deadlocked prod's GitOps and this is the durable fix.

The GitRepository pointed at a public hostname. The flip renamed prod's HTTPRoute
hostnames, so that name stopped serving — but the HTTPRoute that would serve the new
name lives in a DOWNSTREAM tier which cannot reconcile while the source is broken.
Source down -> routes never rename -> source stays down. Observed live: srcReady=False,
"unable to list remote for https://gitea.home.keithjackson.dev".

Pointing at gitea.gitea.svc.cluster.local:3000 removes the cycle permanently: prod's
GitOps no longer depends on public DNS, Cloudflare, its own Gateway, or ${BASE_DOMAIN}.
No future domain change can sever prod from its own source.

Plain http is deliberate: cluster-internal traffic to a ClusterIP, TLS terminates at
the Gateway for external clients, the GitRepository has no secretRef (anonymous clone),
and all secrets are SOPS-encrypted at rest regardless of transport.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 22:14:27 -05:00
jakman85andClaude Opus 5 ae37e27f14 fix(gitops): move prod's own Git source with the apex flip
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
The GitRepository url is a hardcoded literal, not templated on ${BASE_DOMAIN}, so
the apex flip did not carry it. But the flip renames prod's HTTPRoute hostnames, so
external-dns (policy=sync) deletes gitea.prod.home.keithjackson.dev and CoreDNS's
split-horizon zone moves from prod.home. to home. — which would have severed prod
from its OWN GitOps source moments after the flip reconciled.

Landed in the same push as the flip deliberately: prod has not yet fetched 9161cab
(still on b9c0970, all hostnames still prod.home.), so both commits arrive in one
fetch and the url changes in the same apply that renames the routes. A live patch
would not survive — the root Kustomization renders this very file and reverts it.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 22:10:57 -05:00
jakman85andClaude Opus 5 9161cab9cd feat(dns): prod takes the apex — home.keithjackson.dev
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
Dev is decommissioned and has released the name. Prerequisites completed and
verified BEFORE this flip:

  1. dev's 18 A records released by its own external-dns (policy=sync; sources
     deleted after suspending dev's Flux so they could not be recreated).
  2. dev's 18 LEGACY UNPREFIXED TXT registry entries deleted from Cloudflare by
     hand. external-dns v0.21.0 deleted only the `a-` prefixed form and then logged
     "All records are already up to date" — it would NEVER have cleared the older
     unprefixed format it no longer writes but still READS for compatibility. They
     carried `owner=default`; prod runs `owner=prod`, so prod could not have
     claimed these names while they existed. This was the real blocker, and it was
     invisible until measured.
  3. dev's `*.home.keithjackson.dev` + apex Certificate deleted so it stops
     renewing. Issued ~18d ago, so prod's request is #1 this week against LE's
     duplicate limit (5 per identical name set per week).

Verified at the AUTHORITY (betty.ns.cloudflare.com), not via a resolver: 1.1.1.1
served stale TXT for minutes after deletion and would have shown a false failure.

UNCHANGED, deliberately: EXTERNALDNS_TXT_OWNER_ID stays `prod` — giving prod
`default` would let it adopt and sync-delete records it does not own, which is the
2026-09-04 incident. EXTERNALDNS_DOMAIN_FILTER stays the ZONE `keithjackson.dev`:
finding F-DNS1 proved a strict subdomain filter drops the hosted zone and external-dns
then publishes NOTHING.

The zone has wildcards `*.keithjackson.dev` and `*.home.keithjackson.dev` pointing at
the public IP (legacy non-k3s prod). Dev's specific records were shadowing the latter;
prod's new specific records shadow it again. The legacy stack lives on
`*.keithjackson.dev` and is untouched.

k3s_base_domain moves with it to hold the [parity] contract, but that value renders
the apiserver's install-time oidc-issuer-url: it does NOT take effect until the
k3s_server role re-renders config.yaml and k3s is restarted serial: 1. Until then the
apiserver trusts the OLD issuer while Keycloak serves the NEW one, so kubectl OIDC
fails in that window. Cert-based kubeconfigs are unaffected.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 22:07:20 -05:00
jakman85 b9c0970232 merge: Keycloak client URLs reconcile, not just secrets
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
2026-09-12 21:35:15 -05:00
jakman85andClaude Opus 5 f7d4648851 docs(keycloak): record the URL-reconciliation decisions
Records K1-K9: extend-vs-sibling and why, the generated+gated desired state, verbatim
token pass-through, per-pass fail-closed, order-insensitivity, unmanaged-means-untouched,
public clients' URLs managed, one alert pair not two, and the mutation harness.

Also records a process note: my first pass reported three false findings (reconciler
absent from main, import not templated, BASE_DOMAIN dead) because I diffed against
origin/main -- origin is dev's Gitea, stale at 4809199, while the live remote is
prod-gitea at fa4deac. All three were wrong. The rule for this repo is written down so
the next agent does not repeat it: origin is not the source of truth.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 21:31:26 -05:00
jakman85andClaude Opus 5 6202d92c27 test(conformance): gate the URL desired state, and prove the selftest can fail
L0 (212 assertions, was 175):
  * runs ci/render-keycloak-client-urls.py --check, so client-urls.yaml must be exactly
    what the realm import renders. A hand edit that makes the two disagree is
    unmergeable -- this is what keeps the realm import the single source of truth for
    hostnames rather than just the place they were first typed.
  * asserts client-urls.yaml and the realm import are rendered by the SAME Flux
    Kustomization. postBuild.substituteFrom is per-Kustomization, so a split would let
    one ${BASE_DOMAIN} resolve to two different values and the realm would converge on
    a hostname the HTTPRoutes do not serve.
  * asserts the desired state uses no substitution token the realm definition lacks,
    names only clients the import declares, and declares no field outside the
    five-field write surface; that the CronJob mounts it readOnly at the path the
    script defaults to; and that the script, renderer and check agree on which fields
    are writable.
  * NEW mutations.py: injects 12 plausible regressions into the EXTRACTED script and
    requires --selftest to catch every one. A passing suite proves nothing if it cannot
    fail, and "reported success while the realm was broken" is precisely what this
    component exists to end -- so the same standard applies to its own test. ~20s.

L2: the audit Job now mounts keycloak-client-urls and asserts
KEYCLOAK_CLIENT_URLS_AUDIT_OK. Without that it would pass on a realm still holding
pre-migration redirect URIs -- the exact state prod was measured in.

Alerts: the existing Failing/Stale pair is EXTENDED to name redirect-URI drift, not
duplicated. One CronJob gets one pair; a second pair on the same job_name regex would
fire twice for one cause. No new critical, no new permanently-firing rule.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 21:26:12 -05:00
jakman85andClaude Opus 5 23ea72d01e feat(keycloak): reconcile client redirect URIs, not just client secrets
The realm import is a one-shot that never re-runs, so a redirect URI changed in git
reaches the realm never. Prod's stored URIs were measured still on the pre-migration
hostname while git declared the new one -- after a BASE_DOMAIN flip every OIDC login
fails with a redirect_uri mismatch, with the CR still reading Done=True and nothing
alerting. Same failure class as the literal-placeholder secrets, same cure.

EXTENDS the existing reconciler rather than adding a sibling. Decisive reason: a URL
write needs NO new endpoint. It is the same minimal client PUT, and the current URLs
already come back in the allow-listed client lookup -- so the allow-list is widened by
nothing and the users/groups/roles/authentication guarantee is preserved unchanged
rather than re-declared in a second copy. The ConfigMap also carries exactly one
script key by L0 contract, so a sibling would have had to fork ~300 lines of
safety-critical code that the header explicitly forbids forking.

Reconciles redirectUris, webOrigins, rootUrl, baseUrl, adminUrl -- only the fields a
client actually declares, and on a write only the fields that actually drifted. A
field the desired state omits is UNMANAGED and left untouched; absence never means
force-to-empty.

THE TWO PASSES ARE INDEPENDENT. Each reads its own desired state and fails closed for
its own fields only. A malformed URL ConfigMap cannot stop client-SECRET repair, which
would otherwise make credential healing depend on a hostname file.

List fields compare order-insensitively (sorted, not set -- duplicates are still
drift), because Keycloak does not preserve the authored order of redirectUris and an
order-sensitive compare would rewrite every client forever while reporting success.

--selftest grows to 84 assertions covering the domain flip, a client gaining a URI, a
client losing a stale one, a public client's URLs being managed while its secret is
not, per-pass fail-closed, order-insensitivity, and a server that accepts a PUT
without applying it.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 21:25:34 -05:00
jakman85andClaude Opus 5 51d8676f90 feat(keycloak): render the client-URL desired state from the realm import
The realm import is a one-shot: once Done=True it never re-runs, so a redirect URI
changed in git never reaches the realm. Healing that needs a DESIRED STATE the
reconciler can read -- and the reconciler pod has no Kubernetes API access at all
(automountServiceAccountToken: false), so it cannot read the KeycloakRealmImport CR.
Its desired state must be mounted, hence a ConfigMap.

A hand-authored ConfigMap would be a second source of truth for every public
hostname in the realm, free to disagree with the import -- the same shape as the bug
being closed. So it is GENERATED from the import and the render is gated, the same
render==golden idiom catalog/render.py already uses for apps/.

URL strings are copied VERBATIM: a ${BASE_DOMAIN} token in the source appears
unchanged here, and both files are rendered by the SAME Flux Kustomization
(./platform-config), so both are substituted in one pass with one value. The
renderer never parses a hostname and never reads cluster-vars, so there is no
second place the domain can be decided.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 21:24:52 -05:00
jakman85andClaude Opus 5 fa4deac877 fix(conformance): the dirty-tree assertion must judge HONOURED evidence
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
One historical `-dirty` row reddened this gate permanently, even after a clean
re-run of the same test. status.py already resolves duplicates latest-wins
(latest_by_test), so a superseded row is not evidence of anything — but this check
iterated every row, so the poisoned row from the git_sha bug (fixed in the parent
commit) would have failed the audit forever.

The dirty-tree assertion is now scoped to rows status.py would actually honour, by
importing latest_by_test rather than reimplementing the rule, so the two cannot
disagree about which rows are live claims. Structural assertions — provenance keys,
sentinel, test-still-exists — still apply to EVERY row.

Nothing is erased: the row stays in the ledger and runs/20260913T014437Z-19fc8bc/
run.log stays committed. It simply stops being read as a live claim. Verified the
guard is intact afterwards — the check still REFUSES both forgeries it is built to
catch (a self-skip presented as a pass, and a proven row from the root commit).

A dirty-then-clean sequence is also the normal shape of noticing a dirty tree,
fixing it, and re-running; that should not permanently fail an audit.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 21:12:55 -05:00
jakman85 a317493bca merge: R-NET-06 re-proven on prod (recovery in <=3s) 2026-09-12 21:10:57 -05:00
jakman85andClaude Opus 5 c66a7e29e9 fix(conformance): the harness contradicted itself in two places
1. auditability-proofs-producible rejected any proven ledger row without a
   sentinel, with no layer exemption — while record_run.py DELIBERATELY records
   L0/L1 rows sentinel-less via its own SENTINEL_EXEMPT_LAYERS, because
   run_l1_offline.py renders skips as JUnit <skipped/> and those layers have no
   pass token to attribute. So ANY `--record` run touching L0 or L1 wrote a row
   this gate then rejected. Consequence: a requirement whose verifier set includes
   an L1 check could never be both PROVEN and L0-green — R-NET-06 is exactly such
   a requirement, and hit it the moment it was re-proven on prod. The check now
   honours the same constant instead of restating the policy, so the two halves
   cannot drift apart again.

2. git_sha() marked the tree dirty from tracked files, but conformance/runs/
   ledger.jsonl IS tracked — so the FIRST `--record` run of a session dirtied the
   tree and every LATER run was stamped `-dirty`, which check.sh then refuses to
   honour. Recording one test poisoned the next, and that cost a second live
   fault-injection run against prod to work around. Its docstring already stated
   the intent for the untracked runs/<id>/ half of the same artifact; the ledger is
   the tracked half. Now excluded via pathspec, mirroring status.py's
   IMPL_EXCLUDE_PREFIXES which already lists this path for the same reason.

Neither change weakens an assertion: (1) applies an exemption that already existed
and was already documented, (2) stops the recorder invalidating its own output.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 21:10:38 -05:00
jakman85andClaude Opus 5 155df8ee3f test(conformance): R-NET-06 PROVEN on prod — vpn-killswitch re-run from a clean tree
Supersedes the -dirty row from the previous commit by the ledger's own
append-only / latest-wins convention (lib/status.py latest_by_test): nothing
is rewritten, a newer row simply outranks the older one.

  runs/20260913T014923Z-9d0814a  vpn-killswitch (L3)  proven  env=prod
  git_sha 9d0814ae... (CLEAN — resolvable, so the freshness check passes)

Fault injection against LIVE prod, gateway k3s-prod-3. Timeline (UTC):

  01:49:27  tunnel-stop Job applied
  01:49:31  Job succeeded — host vpn-egress-tunnel STOPPED, tun0 gone
  01:49:31..01:51:01  vpn-class blackholed for 8 consecutive polls, NO response
                      of any kind; internet-class unaffected on the WAN
  01:51:02  self-healing finally: tunnel-restart Job
  01:51:05  kubectl wait confirms the host brought the unit back
  01:51:04  egress-uplink label AND the BPF egress map both move 134 -> 139
            (measured independently, out of band, during the run)
  01:51:08  vpn-class egress back on PIA 158.173.241.11

RECOVERY WAS UNAIDED and took ~3s from tunnel-up to a re-steered datapath —
far inside the ~2-minute convergence timer, so the EVENT-DRIVEN up-script
path did the work, not the timer backstop. This is the half that was
previously unprovable: before the egress-uplink label, Cilium held the stale
ifindex and vpn-class egress never came back.

WHAT THE NO-LEAK ASSERTION DOES AND DOES NOT PROVE: it proves vpn-class
fails CLOSED (zero packets out, never the WAN IP). It does NOT credit the
host nftables backstop, which provably cannot fire for pod traffic —
bpf_redirect() skips netfilter. Stopping the tunnel removes tun0 and Cilium
drops on its own because the cached egress ifindex is unusable. Fail-closed
is real; the mechanism is Cilium's drop, as the test header now says.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 20:52:47 -05:00
jakman85andClaude Opus 5 9d0814ae61 test(conformance): R-NET-06 run against prod — L1 proven, L3 pass recorded -dirty
Two recorded runs against PROD (CONF_ENV=prod, context k3s-prod):

  20260913T014321Z-19fc8bc  egress-class-render (L1)  proven
  20260913T014437Z-19fc8bc  vpn-killswitch      (L3)  proven, but git_sha -dirty

The L3 run PASSED for real on prod — every declared sentinel present
(NS_PRIVILEGED_OK, TUNNEL_HEALTHY_OK, NO_LEAK_OK, INTERNET_UNAFFECTED_OK,
TUNNEL_SELF_HEAL_OK, TUNNEL_RECOVERED_OK), tunnel stopped 01:44:48Z and
recovered unaided by 01:46:25Z.

Its row is nevertheless NOT usable as proof: git_sha carries `-dirty`, so
status.py's ancestry check cannot resolve it and the row degrades to
stale-proof. Cause is a harness flaw, not the cluster:
lib/record_run.py's git_sha() ignores UNTRACKED files (so a run's own
runs/<id>/ does not mark it dirty) but conformance/runs/ledger.jsonl is
TRACKED — so the FIRST --record run of a session appends a row, dirties the
tree, and every SUBSEQUENT --record run in that same session is stamped
-dirty. Recording one test poisons the proof of the next.

Committed as-is rather than rewritten: the ledger is append-only and this is
honest evidence of a real run. A clean re-run follows and supersedes it by
latest-wins (lib/status.py latest_by_test).

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 20:48:55 -05:00
jakman85 19fc8bcb27 merge: VPN egress gateway on prod, plus three datapath fixes
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
2026-09-12 18:15:02 -05:00
jakman85andClaude Opus 5 53e09a2610 docs(vpn): dev recovered the minute prod left the shared PIA server
Closes the loop on the collision entry with the recovery evidence rather than
leaving it as an open question for the operator: dev's last Inactivity timeout is
23:05:06 UTC — the minute prod switched to .135 — and at 23:12:03 that same
session was still the live one, seven minutes unbroken against a two-minute
cycle, 4/4 ping replies at 146 ms, 0% loss, and vpn-class pod egress back to
158.173.241.36 on three consecutive probes.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 18:12:46 -05:00
jakman85 9b97e073e2 merge: main (velero R2 + arr config PVC) into feat/vpn-egress-prod 2026-09-12 18:11:18 -05:00
jakman85andClaude Opus 5 38e4bf1459 docs(vpn): record the PIA server collision, and the accidental proof it handed us
The stand-up broke dev, so the "one gateway per PIA server" rule and the
measurement behind it are now written down where the next person will look:
platform/vpn-egress/README.md and the session's DECISIONS-LOG entry. Both carry
the verbatim journal lines — same server, same pushed `ifconfig 10.72.192.7` on
both gateways — and the flap timestamps, because "same account, inside the
connection limit" is true and is exactly what makes this non-obvious.

Also corrected: the dev/prod table now shows the two environments on different
servers with different exit IPs (dev 158.173.241.36, prod 158.173.241.11), and the
open item about them sharing one public IP is replaced with the real remaining
decision — whether prod gets its own PIA account, which would remove the
hand-maintained per-environment server value entirely.

The unplanned bonus, recorded in both places: moving prod to .135 gave it a
genuinely NEW tunnel address on a real reconnect (10.72.192.7 ifindex 128 ->
10.150.192.86 ifindex 129) — the precise case Cilium does not follow and the one
defect 1 was about. The egress-uplink label moved on its own, Cilium re-resolved,
and all four vpn-class workloads kept egressing with no manual step. Better
evidence than the synthetic address swap used to characterise it, and it cost
nothing to capture.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 18:09:31 -05:00
jakman85andClaude Opus 5 37a6c94168 fix(vpn): two gateways on one pinned PIA server evict each other every 2 minutes
Standing up prod broke dev, and the mechanism is worth spelling out because
"same PIA account, well inside its connection limit" is true and still not enough.

The vaulted .ovpn pins ONE server. With both gateways using it, dev and prod
landed on the SAME server AND were handed the SAME tunnel address — both journals
read `[Server-10779-2a] Peer Connection Initiated with [AF_INET]158.173.241.112:1198`
and `ifconfig 10.72.192.7 255.255.192.0`. Each reconnect steals that address on
that server; the loser then receives nothing, trips `--ping-restart` after 60s,
reconnects and steals it back. Measured cycle: dev restarting at 22:57:05,
22:59:03, 23:01:05 UTC with prod interleaved about a minute off, each ~2 minutes
apart, indefinitely. Neither environment had reliable vpn-class egress — prod's
probes looked green only because they happened to land inside an up-window.

Fix: `vpn_egress_gateway_pia_remote` ("<host> <port>", empty = honour the .ovpn)
rewrites just the `remote` line of the deployed .ovpn, and prod's group_vars sets
it to 158.173.241.135. That is another member of the same PIA Stockholm LB pool —
`sweden.privacy.network` resolved from k3s-prod-3 to
158.173.241.{34,55,112,135,239} — and every server in the pool takes the same
credentials and the same inline CA, so only the address differs. Dev keeps the
.ovpn's .112 untouched, which is why the override is per-group rather than in the
role or a second vaulted .ovpn.

Two details that matter for whoever touches this next:

* The regex is anchored to a line STARTING with `remote` plus whitespace, so
  `remote-cert-tls` and `remote-random` are not collateral damage. The task keeps
  `no_log: true` — the transformation never widens what is logged.
* A PIA server address is NOT a credential, so it lives in
  group_vars/vpn_egress_gateway_prod/vars.yml in the clear, next to the vault
  rather than inside it. Keeping it out of the vault is what makes it reviewable.

If prod ever gets its own PIA account, this override can be deleted again.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 18:05:49 -05:00
jakman85andClaude Opus 5 04606d48b2 docs(vpn): stop crediting the nftables kill-switch for a drop it never makes
Four places in the tree told the reader that the host nftables table is the
authoritative fail-closed backstop for egress-class=vpn pods: platform/vpn-egress
README, egress-steering.yaml, the role README, and the vpn-killswitch test's own
GIVEN/THEN prose. It is not, and cannot be — Cilium's eBPF egress-gateway
datapath bpf_redirect()s those skbs to the egress device without invoking
netfilter POSTROUTING. Corrected with the measurement in each place rather than
softened, because a guard the repo trusts and that does not fire is worse than no
guard.

What replaces the claim, everywhere: vpn-class fails closed because CILIUM drops
the traffic when it cannot use its cached egress interface, verified by tcpdump on
both the WAN and tun0 during a tunnel stop and a tunnel restart (0 packets out
either). The nft table keeps a narrower, now-proven job: node-originated traffic
sourced from a tunnel address (killswitch_drop moved 0 -> 3 packets on
k3s-prod-3 when such a packet was routed at the WAN).

Also recorded:

* egress-steering.yaml — `interface: tun0` is resolved ONCE and cached, and the
  gateway node is what tells Cilium it changed (the egress-uplink node label). A
  future reader's instinct will be to have the node write to that object; the
  comment says not to, because it is Flux-managed, and points at the label and
  `cilium-dbg bpf egress list` as the debugging entry points instead.
* egress-steering.yaml — why the new NFS media mounts are not affected by
  pod-scoped steering: an NFS PV is mounted by the node kernel in the host netns,
  so podSelector never matches it. The 192.168.0.0/16 exclusion covers the server
  too, but that is the second reason, not the first.
* vpn-killswitch test — its recovery step depends on the re-steer path, and could
  not have passed before it existed. Its no-leak assertion was always right; only
  the explanation was wrong. Also states the prerequisite plainly: the test cannot
  run at all without a labelled gateway node.
* platform README — prod is live on k3s-prod-3 with the before/after egress IPs,
  and the note that prod's vpn-class pods had NO internet beforehand (CEGP with no
  matching node -> `Gateway IP: Not Found` -> dropped), so arming the gateway
  restored egress rather than moving it off the WAN.
* role README — the standing "add the shared fwmark to the interface contract"
  escalation is closed as not implementable on Cilium OSS, rather than left open
  for someone to attempt.

DECISIONS-LOG carries the full session entry, including the drafted ADR-0015 /
docs-03 amendment for the architect (specs are immutable to agents) and the five
things left for the operator — chief among them that a silent gateway outage still
has no alert, which is what let the dev gateway sit dead for 80 minutes.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 17:57:52 -05:00
jakman85andClaude Opus 5 60e3f09c67 feat(vpn): prod egress-gateway group — k3s-prod-3, singleton, opt-in only
`[vpn_egress_gateway_prod]` is a SEPARATE inventory group with its own vaulted
credentials, not a second member of `[vpn_egress_gateway]`. That group is the
default target of playbooks/vpn.egress.gateway.yml, so adding a prod host to it
would mean a flagless `ansible-playbook playbooks/vpn.egress.gateway.yml`
silently reconfigures a prod node's kill-switch datapath. Prod is reached only by
asking for it, with `--list-hosts` first:

  -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3

Node 3 because node 1's SSD was replaced yesterday (least operational history)
and node 2 carries the §0b memattr history. Exactly one node, forever: Cilium OSS
picks "the first node in lexical ordering" among nodeSelector matches, so a
labelled k3s-prod-1 would silently steal the gateway role — and the public egress
IP — from k3s-prod-3, with no event and no alert.

vault.yml is a verbatim copy of the dev group's CIPHERTEXT: same PIA account per
the operator, and copying the encrypted blob means the plaintext was never
decrypted to disk, through a shell, or into a transcript. The two files are now
independent — rotating means re-encrypting both, which is the right behaviour for
two environments.

Also corrects a stale inventory note that claimed k3s-prod-3 has wlp1s0 up on
192.168.2.195. Re-verified before choosing it as the gateway, because a second
live uplink in a kill-switch datapath is not a footnote: there is no wl* netdev on
the node at all, no phy80211 device, the /etc/network/interfaces wlp1s0 stanza is
commented out, 192.168.2.195 is assigned nowhere, and the only IPv4 addresses are
lo and eno1 192.168.2.12. Nothing needed bringing down.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 17:38:46 -05:00
jakman85andClaude Opus 5 6f698f59af fix(vpn): Cilium never re-resolves tun0 — the gateway dies on every reconnect
Measured on k3s-dev-3, not reasoned about. Three findings, one of which had the
dev gateway silently dead for ~80 minutes before this change:

1. Cilium caches (egress IP, egress ifindex) when it first resolves the CEGP's
   `interface: tun0` and does NOT re-resolve when tun0 changes. Every tunnel
   restart re-creates tun0 with a NEW ifindex, so vpn-class egress is blackholed
   from then on. Observed: node had tun0 at ifindex 1987 while `cilium-dbg bpf
   egress list` still said 1986; pod curl timed out. Forcing a re-apply moved the
   map to 1987 and egress recovered instantly. `egress-gateway-reconciliation-
   trigger-interval: 1s` does NOT cover netlink link events — dev has that same
   1s setting and still did not self-heal, 20s+ after recovery.

   Fix: vpn-egress-resteer.sh publishes "<tun0 addr>-<ifindex>" as a node label.
   A node update makes the gateway re-process the policy (verified: cached
   ifindex 1986 -> 1987 within 10s). A node label, not an annotation on the
   policy, because the policy is Flux-managed GitOps and the node's labels are
   already this role's territory. Fired by the OpenVPN up-script on every
   connect, plus a 2-minute convergence timer for the boot case where the
   kube-API is not up yet. Proof: `systemctl stop` then `start` of the tunnel now
   recovers on its own (map 1989 -> 1990, PIA IP returns); before this commit the
   identical sequence stayed broken indefinitely.

2. The nft kill-switch CANNOT see the traffic it claims to guard. A counters-only
   diagnostic table on the gateway node recorded, while a vpn-class pod egressed:
   0 packets in forward and 0 in postrouting for both the pod CIDR and the PIA
   range, against 45166 packets on the same postrouting hook for WAN traffic and
   16 packets for a node-originated curl out tun0. Cilium's eBPF datapath
   bpf_redirect()s these skbs to the egress device without invoking netfilter.

   So `ip saddr @set oifname != tun0 drop` can never fire for vpn-class pods.
   What actually fails closed is Cilium's own drop when the cached egress
   interface is unusable — tcpdump during a tunnel stop AND during a restart
   showed 0 packets leaving either the WAN or tun0. Every rule now carries a
   `counter` so the claim stays falsifiable, and every comment that called this
   table the authoritative vpn-class backstop is corrected rather than left to
   mislead the next reader.

   Also removed the `meta mark 0x1c8 oifname != tun0 drop` rule and its variable:
   nothing stamps that mark (the only Cilium mark on the node is 0x200/0xf00, the
   L7-proxy rule to table 2004), the traffic never reaches netfilter anyway, and
   a live rule matching an arbitrary mark is a hazard to whatever picks it next.

3. The up-script flushed the nft source set on every connect, so a packet still
   sourced from the previous tunnel address matched neither the ip-rule nor the
   drop rule. The set is now additive (`add element` is idempotent; the
   down-script already retained it): an element can only ever tighten the guard.
   Finding 2 means that window could not have leaked POD traffic, but
   node-originated tunnel traffic does traverse netfilter, and additive is free.

Kill-switch reloads no longer disarm the guard either: the .nft file uses
`table` + `flush table`, which clears rules but leaves the named set's elements
intact, so an Ansible run cannot empty the set mid-flight.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 17:38:27 -05:00
jakman85 839f7fed4a merge: velero works against R2 — plugin pin, NFS volume policy, metrics
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
2026-09-12 17:20:55 -05:00
jakman85andClaude Opus 5 2d5171e064 feat(velero): scrape the backup controller — three critical alerts could never fire
platform/monitoring/alerts.yaml has shipped VeleroBackupFailure (critical),
VeleroBackupPartialFailure and VeleroNoRecentBackup since the observability
wave. None of them could ever have fired. The velero chart defaults
`metrics.serviceMonitor.enabled: false`, and prod confirms it live:
`kubectl -n velero get servicemonitor` returns no resources, so
velero_backup_failure_total and velero_backup_last_successful_timestamp were
never scraped. A nightly backup could fail — or silently stop running — for
weeks with nothing to say so. That is exactly what R-STATE-03 is for, which
makes the scrape part of the backup path rather than an observability extra.

Both flags verified against chart 12.1.0: `helm template` emits neither object
without the monitoring.coreos.com/v1 capability, and emits both PodMonitor and
ServiceMonitor with it — so the chart gates on capability detection AND
`enabled`, and helm-controller supplies real capabilities from the API server.
No additionalLabels are needed: platform/monitoring already sets
serviceMonitorSelectorNilUsesHelmValues and podMonitorSelectorNilUsesHelmValues
to false, so Prometheus selects every monitor regardless of labels.

Independent of the R2/bulk-tier fix in the previous commit; droppable on its own
if the PM wants to sequence it separately.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 17:18:26 -05:00
jakman85andClaude Opus 5 2e259a808f fix(velero): prod backups could not write metadata to R2, and opt-out FSB would have shipped 8.6 TiB
Velero was resumed on prod (live `flux resume`; suspension was never in git) and
its BackupStorageLocation validated Available against Cloudflare R2 on the first
try — bucket k3s-prod-backup, prefix `velero`, region auto, checksumAlgorithm "".
Backups still failed, and a second, unrelated hazard was armed. Both are fixed
here; both were measured on prod, not reasoned about.

1. velero-plugin-for-aws v1.14.2 cannot write to R2 at all.
   It sets PutObject's `Tagging` field unconditionally, so the SDK emits an empty
   `x-amz-tagging` header on every PUT. R2 does not implement object tagging in
   any form and answers HTTP 501 NotImplemented. Two separate Backups reproduced
   it on the write of velero/backups/<name>/velero-backup.json: kopia uploaded
   the volume data fine (it uses its own S3 client), then the backup went
   phase=Failed with no metadata object — i.e. unrestorable. No config avoids it;
   `config.tagging` with a real value sends a tag set R2 also rejects. Upstream
   PR #309 makes the field conditional and landed on release-1.14 after v1.14.2
   was cut, so v1.14.3-rc.2 is the earliest build with the fix. Pinned to it,
   with the exit condition (v1.14.3 GA) stated at the pin.

2. The opt-out FSB schedule would have uploaded the NFS bulk tier.
   Schedule (1) runs defaultVolumesToFsBackup:true and excludes only monitoring
   and unifi. Since 2026-09-11 prod also carries four static NFS PVs —
   plex-media, sonarr-media and radarr-media all bound to the SAME export
   /mnt/Storage/plex (8.32 TiB measured via zfs list), plus
   nextcloud-legacy-archive at 270 GiB. The 03:00 run would have read that
   library three times over NFS and pushed ~8.6 TiB to R2 nightly, retained
   720h. Namespace exclusion cannot express the fix: those four namespaces also
   hold real Longhorn PVCs (plex/sonarr/radarr config, nextcloud-data) whose
   loss is unrecoverable.

   So the opt-out is now per VOLUME, by SOURCE TYPE: a velero volume policy
   (ConfigMap velero-volume-policies) skips any volume whose source is NFS, and
   schedule (1) references it. That tests the property that makes a volume
   bulk-tier (R-STATE-04: the bulk tier IS the NFS export) instead of a name list
   the next PV would slip past. The per-pod backup-volumes-excludes annotation
   was rejected for failing OPEN — a new NFS mount would be backed up in full
   until someone remembered.

   PROVEN on prod (velero 1.18.1 / chart 12.1.0): a Backup of ns nextcloud with
   defaultVolumesToFsBackup:true plus this policy produced exactly ONE
   PodVolumeBackup — `data` (nextcloud-data, Longhorn, 875,417,102 bytes,
   Completed). The NFS `legacy-archive` volume produced none.

Residual gap, stated rather than hidden: the rule matches NFS only. A future
bulk volume on another out-of-cluster source needs its own condition.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 17:18:26 -05:00
jakman85 3ec67bafdd merge: platform-admins may reach the media app UIs
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
2026-09-12 17:15:14 -05:00
jakman85andClaude Opus 5 5da116b34d fix(oauth2-proxy): platform-admins may reach the media UIs — the immediate 403
The four forward-auth media UIs (jackett, qbittorrent, radarr, sonarr) gate on
`--allowed-group=media-users` alone. On prod the `homelab` realm has exactly one
user — the operator's `jakman85` — and it is a member of `platform-admins` ONLY.
`media-users` has zero members, so those four hosts were reachable by nobody:
oauth2-proxy completed OIDC discovery, redirected to Keycloak, got a token whose
`groups` claim is `["platform-admins"]`, found no allowed group in it, and
answered 403. No login form appeared because the browser already held a Keycloak
SSO session from a platform-admins login — hence "403 immediately".

pihole and crafty were unaffected: they already gate on platform-admins.

`--allowed-group` is repeatable and OR-ed, so listing both groups means "media
consumers OR platform operators". This grants no real privilege — platform-admins
already maps to cluster-admin via k8s-oidc-rbac.yaml (oidc:platform-admins), so a
member can already exec into these pods — and it keeps media-users meaningful for
non-admin household accounts.

Ruled out while diagnosing: client-secret drift (all 7 sidecar secrets hash-match
keycloak-realm-clients), stale apex URLs (every live sidecar carries
keycloak.prod.home.keithjackson.dev), missing group mappers (all 7 clients emit
claim=groups, full.path=false), and the central oauth2-proxy pod restarts — those
were Keycloak returning 503 "no healthy upstream" to OIDC discovery at startup, a
consequence of the Keycloak roll, not a cause.

Goldens re-rendered with the canonical command:
  python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation

R-IDP-06

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 17:12:05 -05:00
jakman85andClaude Opus 5 6b953cc56b fix(arrs): the config PVC was mounted where the app never looks
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
sonarr, radarr, jackett and qbittorrent all mounted their Longhorn config PVC at
the renderer's default /data/config, but every one of these images runs with
`-data=/config`. So the PVC held nothing but lost+found while the app wrote its
entire configuration to the container's EPHEMERAL writable layer — destroyed on
every pod recreation: restart, reschedule, image bump.

Measured in-cluster before fixing, per app, rather than inferred:
  /data/config  dev=2080  [lost+found]                      <- the Longhorn PVC
  /config       dev=2053  [Sentry asp config.xml logs ...]   <- the node's own disk
and a marker file written to /config did not appear in /data/config, so they are
demonstrably different storage. qbittorrent's /config sat on dev=66309
(/dev/nvme0n1p5), i.e. whichever node it happened to land on.

This is the SAME failure catalog/plex.yaml:54 already records having fixed for
Plex; plex was the only one of the five that was right.

The live /config contents were copied onto each PVC BEFORE this landed, so
jackett's API key and qbittorrent's credentials — which other apps reference —
survive the move rather than being regenerated.

catalog/sonarr.yaml:61 documented this as a latent bug and deliberately left it
because it changes live on-disk state. It is being fixed now because all four pods
had just been recreated, so their configs were freshly generated and nothing had
accumulated to lose.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 16:35:18 -05:00
jakman85 f5e62d9169 merge: bulk NFS volumes for plex, sonarr, radarr and nextcloud
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
2026-09-12 16:26:31 -05:00
jakman85andClaude Opus 5 daac05e7e8 feat(bulk): prod mounts the NFS export through pfSense's WAN address
The operator added the pfSense port-forward, so BULK_NFS_SERVER is no longer a
placeholder. Prod gets 192.168.2.102 (pfSense WAN, DNAT tcp/2049 -> 192.168.1.9);
dev stays on the base 192.168.1.9 because it is already on pfSense's LAN.

This value could not be left as a placeholder safely: spec.nfs.server is
unvalidated by Kubernetes, so the PV would be created, the PVC would BIND, and the
failure would only surface as four pods stuck in ContainerCreating.

Verified from k3s-prod-1 before committing, deliberately as uid 1000 rather than
root (the export carries no_root_squash, so a root write proves nothing, and LSIO
images default to PUID 911 which would go Ready and then fail every import EACCES):
mount negotiated nfs4 vers=4.1 proto=tcp, write succeeded, and ln() across
downloads/ -> tv/ gave link count 2 — real hardlinks, so no silent copy-then-delete.

The Nextcloud export was additionally tightened to server-side ro= for prod's three
nodes; a write as root is refused. Its content is config/, config_old/ and
config_backup/ with no data/ directory, so it is an archive, not a live data store.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 16:25:20 -05:00
jakman85andClaude Opus 5 da8a0eac04 test(l0): R-STATE-04 gets a real check — bulk storage provably is not the replicated tier
The gate BLOCKED this branch, correctly, and the block was the interesting part.

Attaching the media library selects R-STATE-04 as a touched requirement ("bulk media storage has
a capacity strategy distinct from the replicated tier"), and R-STATE-04 had NO scaffolded test, so
the coverage REQUIRE stage failed both l0 and l1. There were two ways out. One was to keep
avoiding the token — the existing discipline in catalog/plex.yaml and catalog/qbittorrent.yaml,
which say in as many words that naming the requirement would block the merge. That made sense while
the tier was a stub. It does not make sense in the change that DELIVERS the tier: it would mean
shipping the requirement and claiming nothing. So: the check.

WHAT IT ASSERTS, and why each one is a real failure mode rather than a shape
The `bulk` StorageClass sat as `server: REPLACE-ME` for six roadmap phases, and the only thing
keeping a multi-terabyte library off the 3-replica `longhorn` class was four hand-written comments.
That is the class of failure being closed: a bulk volume on the replicated tier does not fail
loudly — Longhorn accepts the PVC and starts replicating 3x of data three mini-PCs cannot hold.

  (1) DISTINCTNESS — no bulk PV/PVC on any class the repo declares with the Longhorn provisioner
      (derived from platform/longhorn/, so adding a Longhorn class cannot create a blind spot),
      and the class is explicitly EMPTY.
  (2) CAPACITY STRATEGY as arithmetic — a bulk volume at least an order of magnitude above the
      same service's replicated volume. The requirement is about two magnitudes of data; a "bulk"
      volume sized like a config PVC means the tier was named, not applied. The 10x floor is
      CALIBRATED against the real corpus, not guessed: the media library is ~460x its config
      volume and the Nextcloud archive ~13x its 20Gi, so a higher floor would fire on a legitimate
      volume and teach the next reader to weaken the check. My first draft used 100x and did
      exactly that — the corpus caught it.
  (3) PRE-BOUND FROM BOTH ENDS (claimRef + volumeName, agreeing). Guards the most deceptive
      failure available in this tier: a dynamic bulk class hands out a FRESH EMPTY directory, the
      pod mounts, the PVC Binds, everything is green, and the 8 TiB was never attached.
  (4) Retain, always. (5) mountOptions carry `hard` and exactly one nfsvers=4.x, and never `soft`
      — a soft mount hands a server hiccup to the app as a short read, which for a media library
      is silent corruption; and v4 is what keeps the firewall surface at one port.
  (6) ONE EXPORT, AT MOST ONE MOUNT PER WORKLOAD. This is the assertion I most wanted. Splitting
      one export into two subPath mounts looks TIGHTER and breaks link() with EXDEV; Sonarr and
      Radarr then degrade to copy-then-delete, rewriting every import over NFS and destroying the
      seeding copy, surfacing weeks later as "imports are slow and my ratio is gone". No reviewer
      sees that in a diff.
  (7) READ-ONLY COUPLING — nfs.readOnly, the volumeMount flag and ReadOnlyMany must agree, so
      dropping one still leaves the others; and every bulk mode is a `...Many`.
  (8) The server is a BARE cluster-vars token, that knob has a base default, and EVERY overlay
      supplies it — the seam that lets one authored definition serve two clusters that reach this
      host by different addresses.
  (9) LOAD-BEARING and NON-VACUOUS. Wired unconditionally into ci/l0.sh as Stage 3e for the same
      reason as 3c/3d: Stage 3b scopes to touched requirements, and the path map keys NEITHER
      apps/* NOR platform/longhorn/* to this requirement — so the change most likely to break
      this could attribute nothing and be gated by nothing. An EMPTY bulk corpus FAILS rather
      than passing trivially, and a floor on the assertion count keeps a pass from degenerating.

It reads the COMMITTED apps/ goldens rather than calling the renderer. Asking the renderer would
only prove it agrees with itself; the goldens are what Flux reconciles, and render==golden is
already its own gate stage.

PERTURBED, because a check that has never failed is not evidence. Each of these was applied to the
real tree and each fired with the intended message: bulk PVC moved onto `longhorn`; Retain -> Delete;
`hard` -> `soft`; the token replaced by the literal 192.168.1.9; sonarr's single mount split into
two subPath mounts (assertion 6); and the prod overlay's knob deleted.

Also hardens render.py: `size` is structurally required of a bulk entry (a PV with no capacity is
rejected by the API server) but the JSON Schema only enforces `export`, so omitting it raised a bare
KeyError three frames down. It now raises a sentence naming the service, the volume and the remedy.
Found by the same perturbation pass.

Coverage: 48/51 -> 49/51 (96.1%), 2 gaps, 0 safety-critical.
BASE_REF=main make -C ci gate -> MERGE GATE: PASS.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-08 00:03:34 -05:00
jakman85 cac413e87a feat(cnpg): synchronous replication, quorum of 1, dataDurability preferred
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
Three instances replicating ASYNCHRONOUSLY still lose data on failover: COMMIT
returns as soon as the primary has flushed locally, so a primary that dies
before its standbys catch up takes the last few transactions with it and the
promoted standby comes up quietly missing them. On keycloak-db that is a
just-created user, a rotated credential or a consent vanishing at exactly the
moment of a failover.

`method: any` + `number: 1` makes COMMIT wait for ONE of the two standbys to
confirm — the point of having them.

dataDurability: preferred, NEVER required. `required` is a strict quorum:
fewer than `number` healthy standbys and writes BLOCK. Postgres here shares
three nodes with etcd and the CNI, so one degraded standby — a node reboot, a
Longhorn replica rebuild, a Cilium restart — would turn a partial degradation
into a total write outage, and on the SSO database that takes kubectl OIDC,
Grafana, Headlamp, Gitea and forward-auth with it. `preferred` keeps
synchronous commit while a healthy standby is there and relaxes the required
count dynamically when one is not. dataDurability is only honoured while
standbyNamesPre/standbyNamesPost are unset — both are.

NON-DISRUPTIVE: this compiles to `synchronous_standby_names`, a reload-only
PostgreSQL parameter. No pod restart, no switchover, no client disconnect.
It is the last of the three commits because it wants standbys that already
exist and are already on their final nodes.

Field shape verified against the live clusters.postgresql.cnpg.io CRD before
relying on it (CNPG 1.30.0, byte-identical schema on k3s-prod and k3s-dev):
spec.postgresql.synchronous requires `method` (enum any|first) and `number`
(x-kubernetes-validation self > 0); `dataDurability` is enum required|
preferred. None of the three Clusters previously had a `postgresql:` block,
so this adds one rather than merging into existing parameters.

DEV IMPACT: identical and non-disruptive, same reload-only path.

Verified: yamllint clean; both tiers render the block.

Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit f1ce19461415e77eb2f1ca5109ebab49edf0860b)
2026-09-07 23:54:19 -05:00
jakman85andClaude Opus 5 82aff4d053 feat(apps): attach the media library and the legacy Nextcloud dataset over NFS
plex, sonarr, radarr and nextcloud each gain one bulk volume. Authored in the catalog and
re-rendered, never hand-edited into apps/ (render-golden: 10/10). Layout, the export change the
operator must apply, the uid analysis and the validation probes are in docs/BULK-NFS-STORAGE.md;
everything there was measured on the host over read-only SSH, not assumed.

WHAT THE LAYOUT TURNED OUT TO BE, AND WHY IT DECIDED THE DESIGN
/mnt/Storage/plex holds movies/ tv/ photos/ AND downloads/ as SIBLINGS in one dataset — so the
arrs' import source and their libraries are one filesystem on the server, and the download client's
completed folders are already named radarr/ and tv-sonarr/. The k3s arrs are joining a tree an arr
stack has been driving for a long time.

That sibling fact is the whole argument for the arrs mounting the export ROOT rather than a
subPath per directory. Sonarr and Radarr import by hardlinking or atomically renaming, both
same-filesystem operations, and to the kernel "same filesystem" means SAME MOUNT. Two subPath
mounts of one export are two mounts: link() across them returns EXDEV regardless of what the
server knows, and the arrs do not fail loudly — they degrade to copy-then-delete, rewriting every
file over NFS and destroying the seeding copy. The smallest subtree containing both tv/ and
downloads/ IS the export root, so the root mount is the MINIMAL mount that preserves hardlinks.
Least privilege bounded below by a hard requirement. The cost is stated, not hidden: sonarr can
also write /media/movies.

  plex       /mnt/Storage/plex      -> /media                 ro   ReadOnlyMany
  sonarr     /mnt/Storage/plex      -> /media                 rw   ReadWriteMany
  radarr     /mnt/Storage/plex      -> /media                 rw   ReadWriteMany
  nextcloud  /mnt/Storage/nextcloud -> /mnt/legacy-nextcloud  ro   ReadOnlyMany  (subPath: nextcloud)

Plex is read-only because it indexes and streams; every write on that tree belongs to the arrs, and
saying so removes Plex from the set of things that can damage 8.33 T. All three media consumers use
the SAME mountPath deliberately: when qbittorrent is eventually given this export, the arrs and the
download client will already agree on absolute paths, which is the other precondition for hardlinked
imports.

*** /media AND NOT /data — A COLLISION CAUGHT WHILE RENDERING. *** The arrs' Longhorn config PVC
renders at the renderer default /data/config. A media mount at /data is mounted OVER it and MASKS
the config volume: the pod comes up with its app database shadowed by an 8 T media tree. That
discovery also exposed a SEPARATE PRE-EXISTING BUG, reported and deliberately NOT fixed here:
LinuxServer arr images run with -data=/config, so /data/config is very likely the wrong home for
that PVC and the app has been writing its EPHEMERAL layer — the exact failure catalog/plex.yaml's
mountPath comment records already having been bitten by. Plex was fixed; sonarr, radarr, qbittorrent
and jackett were not. Fixing it starts those apps from an empty config volume, so it needs its own
change with the operator present. This change does not make it worse.

UID/GID — THE FAILURE THIS TIER ALWAYS HAS
The export is sec=sys + no_root_squash, so the client's numeric uid passes through verbatim and the
server's ownership is the entire access story. The tree is 0755 owned 1000:3000, so ONLY uid 1000
can write it. LinuxServer images default to 911. Without intervention sonarr and radarr would have
mounted cleanly, gone Ready, and failed EVERY import with EACCES.

And fsGroup cannot fix that — the in-tree NFS plugin does NO ownership management, so
podSecurityContext.fsGroup is a no-op on an NFS volume. Reaching for it is how this gets
misdiagnosed. So PUID=1000/PGID=3000 are added to both arrs (PGID 3000 keeps what they write
group-readable to the live non-k3s consumers of the same export), and fsGroup: 3000 is added for
the LONGHORN config PVC only, so changing PUID does not depend on the image re-chowning /config.
plex already runs as 1000 and reads a 0755 tree; nextcloud reads as www-data. Neither needs work.

NEXTCLOUD IS MOUNTED READ-ONLY, WHICH IS A DEVIATION FROM THE BRIEF — ON EVIDENCE
The task said nextcloud writes. The dataset says otherwise: of 270 G, 267 G is config_backup/, the
rest is config/ (103 M) and config_old/ (3.4 G). There is NO data/ directory. This is the OLD
non-k3s instance's configuration plus a backup blob — an archive — and that instance is STILL LIVE
on it (exported rw to 192.168.1.6 and .5, which the operator has said must stay active). Two
Nextclouds writing one instance's config directory, each with its own database and therefore its own
oc_filecache, is unrecoverable corruption. "Do not grant more than needed" lands on ro, and ro is
the trivially reversible choice. If the intent is in fact a MIGRATION, that is a larger and more
deliberate change than a mount; the one-line flip is documented.

Read-only for nextcloud can additionally be enforced SERVER-side (only nextcloud consumes that
dataset from k3s), and the sharenfs value in the doc uses ro= for the k3s clients. The media tree
cannot: plex and the arrs may land on any node, so the server sees the same addresses for reader
and writers. Plex's ro is therefore a client-side guarantee — stated as a limitation, not left
implied.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 23:54:13 -05:00
jakman85 e8d8601ca6 feat(cnpg): required pod anti-affinity + switchover primary rolls
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
`instances: 3` alone does not keep the three instances apart. All three
Clusters used CNPG's default `podAntiAffinityType: preferred`, which is a
scheduler HINT — two instances may share a node whenever that scores better,
and they WILL on the path that matters: when CNPG recreates an instance after
a node loss, the surviving nodes are the only candidates and one of them
already runs an instance. `preferred` is how a 3-instance cluster decays back
into the single-node failure domain the previous commit exists to remove.

`required` makes it a scheduling constraint. On a 3-node cluster with no
spare, the cost is that an instance whose node is gone stays Pending until a
node returns. That is correct: a replacement squeezed onto an already-occupied
node dies with that node while making the cluster LOOK three-way replicated.
2 of 3 instances still serve and CNPG still promotes.

ALSO SETS primaryUpdateMethod: switchover. This is not cosmetic. Changing the
affinity changes the instance pod spec, so CNPG performs a rolling update, and
the CRD default for primaryUpdateMethod is `restart` (verified against the
live CNPG 1.30.0 CRD on both clusters) — a delete-and-recreate of the PRIMARY
pod, i.e. tens of seconds of no Postgres. On keycloak-db that is an SSO
outage, which also means kubectl OIDC, Grafana, Headlamp, Gitea and
forward-auth. With standbys now present, `switchover` promotes an
already-updated streaming replica instead, reducing the roll to a connection
reset. Known constraint documented in the manifest: under `switchover` the
operator rejects an update that changes imageName and a postgresql parameter
in the same reconcile.

Verified satisfiable before merge (read-only): k3s-prod and k3s-dev each have
3 Ready, schedulable, untainted nodes, and every pgdata PVC already has a
Longhorn replica on all three (replica-soft-anti-affinity=false), so any
instance can attach from any node.

DEV IMPACT: identical, and dev can absorb it. Longhorn dev nodes are 124.9 GiB
each with 152.0 GiB already scheduled under a 200% over-provisioning ceiling
(249.8 GiB); the six new 5Gi volumes add 30 GiB scheduled per node -> 182.0,
still inside the ceiling, and only ~600 MiB per volume is really consumed.
Prod is far looser: 239 -> 269 GiB scheduled against a 829 GiB ceiling.

THIS IS THE DISRUPTIVE COMMIT of the three. Land it on its own reconcile,
after the scale-out has produced two Ready standbys per Cluster, so the blast
radius is one clean switchover per database rather than a scale-out and a roll
at once. Expect a brief SSO interruption for keycloak-db.

Verified: yamllint clean; both tiers render the new fields; field names,
types and enum values checked against the live clusters.postgresql.cnpg.io
CRD (CNPG 1.30.0, identical on prod and dev).

Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit e1d364238d73e6ee99c9a50108fe8b4e09481a7e)
2026-09-07 23:48:32 -05:00
jakman85andClaude Opus 5 2590bc4317 feat(env): BULK_NFS_SERVER — the one knob prod cannot answer yet, and why it is a placeholder
The bulk tier's PersistentVolumes need a server address, and there is no single correct one: the
two clusters sit on OPPOSITE SIDES of pfSense from the storage host, which has exactly one
interface (vmbr0 192.168.1.9/24, on pfSense's LAN).

  dev  192.168.1.40/.41/.42 — the SAME segment. Reaches it directly, and already holds one NFS
       export from it (/mnt/Storage/k8s-volumes, in the host's /etc/exports). Proven path, so
       base's default is the dev value per this file's existing convention, and dev restates it.
  prod 192.168.2.10/.11/.12 — pfSense's WAN. tcp/2049 does not traverse that today; 22 and 8006
       answer only because pfSense port-forwards them via 192.168.2.102. Opening the storage path
       is a PENDING operator decision whose two candidate outcomes give DIFFERENT values here:
       192.168.2.102 (another port-forward) or 192.168.1.9 (a route). So prod holds
       TBD-PROD-BULK-NFS-SERVER, this repo's established loud-placeholder convention.

Two findings recorded in the prod overlay because they change what the operator has to do:

  * THE EXPORT'S CLIENT LIST IS THE SAME UNDER BOTH OUTCOMES. Measured today from this
    workstation (192.168.2.132) through the existing port-forward: the storage host logged the
    connection as coming from 192.168.2.132 — a WAN-side address — not from a pfSense LAN address
    (`ss -tn` peer 192.168.2.132; the host's only route off-LAN is its default via 192.168.1.1).
    pfSense DNATs WITHOUT source NAT, so the NFS server sees prod's REAL node addresses either
    way. The `zfs set sharenfs=` work therefore does not have to wait for the routing decision.
  * ONE PORT, NOT FOUR. The PVs mount nfsvers=4.1, and NFSv4 carries mount and lock in-protocol —
    no rpcbind/111, mountd or statd. Whichever outcome is chosen, it opens tcp/2049 and nothing
    else. NFSv3 would have needed four ports, three of them on ephemeral numbers.

*** AND ONE HONEST WEAKNESS, stated in the overlay where an operator will hit it. ***
Unlike the LAN knobs above it, this placeholder does NOT fail at apply time. A PV's
spec.nfs.server is an unvalidated string: the API server accepts it, the PV is created, and the
PVC BINDS (binding is pure control-plane). The failure appears only when a pod mounts —
MountVolume.SetUp "Failed to resolve server TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr
and nextcloud would sit in ContainerCreating on prod. There is no CRD validation to lean on here,
so the guard has to be procedural: set the value before prod reconciles a revision carrying the
bulk volumes. Dev renders a working value and is where these manifests are meant to be proven.

Related finding, NOT fixed here because the file does not exist on main: the static guard this
convention is supposed to lean on, `bootstrap/standup.sh`'s `s3_is_placeholder` family, is absent
from main (it lives on unmerged prod branches), so
conformance/tests/l0-static/placeholder-guard-coupling exits 3 SKIPPED and nothing in the gate
currently refuses a placeholder-shaped overlay value. When standup.sh lands, its guard needs an
arm for this knob and a consumer that calls it — assertions (2) and (3) of that check then cover
this value automatically, since the detector's SEED vocabulary already matches "TBD".

Evidence: `kubectl kustomize clusters/dev` -> BULK_NFS_SERVER: 192.168.1.9;
`kubectl kustomize clusters/prod` -> BULK_NFS_SERVER: TBD-PROD-BULK-NFS-SERVER. Both exit 0.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 23:47:39 -05:00
jakman85andClaude Opus 5 cec550045b feat(storage): the bulk tier becomes real — static NFS PVs, authored once per environment
The `bulk` class has existed since Phase 0 as a deliberate stub: the StorageClass named an NFS
CSI driver that is not installed and a server/share of REPLACE-ME, so plex, sonarr, radarr and
qbittorrent all carry a comment saying "attach the library once OQ-2 is resolved". This is that
resolution — for the case that actually exists here, which is NOT the case the stub anticipated.

The stub assumed DYNAMIC provisioning: ask a CSI driver for a bulk volume and get a fresh empty
subdirectory. That is the wrong shape for this platform's bulk data. The media library is 8.33 TiB
that already exists, on ZFS datasets on the Proxmox host, with live non-k3s consumers. Nothing
needs provisioning; existing datasets need ATTACHING. So a bulk storage entry now renders a
STATIC, pre-bound NFS PersistentVolume plus its PVC, and needs no CSI driver at all (the in-tree
NFS volume source is not deprecated and has no CSI migration).

Schema 1.4.0 adds, on a storage entry:
  export     absolute NFS export path; REQUIRED when class is bulk, FORBIDDEN otherwise (an `if`
             in the schema, so a bulk entry cannot be authored without saying what it attaches).
  subPath    mount a subtree instead of the volume root — any class.
  readOnly   any class; on bulk it also sets the PV's nfs.readOnly, so the mount really is `ro`.

The SERVER is deliberately NOT a schema field. It is the `${BULK_NFS_SERVER}` cluster-vars knob,
because dev and prod reach the SAME storage host by different addresses and R-ENV-02 says the
service definition is authored once. A ServiceDefinition says WHICH dataset; the overlay says
where the host is.

Three decisions worth the reader's time, argued in full in the r_bulk_pvs docstring:

  * PV/PVC, not an inline `nfs:` pod volume. Inline is fewer objects and would work. It cannot
    express `persistentVolumeReclaimPolicy: Retain`, which is the one property most worth having
    in git for 8.33 TiB of data with consumers outside this cluster; it puts read-only enforcement
    one Deployment edit away from silent removal; and it is invisible to `kubectl get pv,pvc`,
    which is where an operator looks for attached external storage.
  * storageClassName: "" on both ends, NOT the `bulk` StorageClass. Explicitly empty is the
    canonical marker for a statically provisioned volume. It also sidesteps a trap: a
    StorageClass's `provisioner` is IMMUTABLE, so re-pointing the deployed `bulk` stub would make
    Flux fail on that object forever.
  * Access mode is always `…Many` and follows the read/write posture — ReadOnlyMany for a reader,
    ReadWriteMany for a writer. One export is consumed by several namespaces AND several nodes at
    once, which is exactly what ReadWriteOnce cannot express; which `Many` it is then makes the
    intent legible in `kubectl get pv`.

mountOptions are a platform decision held in the renderer rather than a per-service field:
nfsvers=4.1,hard,noatime,timeo=600,retrans=2. NFSv4 carries mount and lock in-protocol, so the
only port that must be reachable is tcp/2049 — no rpcbind/111, mountd or statd. That is a
firewall decision as much as a storage one. `hard` over `soft` because a soft mount converts a
server hiccup into a short read handed to the application.

The contract doc records the trap that will otherwise be discovered in production: fsGroup is a
NO-OP on an NFS volume (the in-tree plugin does no ownership management), so a writable bulk mount
requires the container's own uid/gid to suit the export's ownership. A mount that succeeds and
then cannot write is this tier's characteristic failure.

Every pre-existing golden renders byte-identical: the new keys are emitted only when declared, and
`readOnly` defaults to False. render-golden: 10/10. selftest: PASS.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 23:45:09 -05:00
jakman85 f338e91770 feat(cnpg): instances 3 on all three Clusters — a standby to fail over to
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
All three CNPG Clusters ran `instances: 1`. Longhorn's 3-way replication
saves the DATA when a node dies; it does not save the SERVICE — a single
instance has to be rescheduled, re-attach its volume and replay WAL before
Postgres accepts a connection again.

With nothing in the spec keeping them apart, all three primaries had drifted
onto ONE node (k3s-prod-2 on prod, k3s-dev-3 on dev). On 2026-09-07 that node
took SSO, Nextcloud and Semaphore down together, and with SSO went kubectl
OIDC, Grafana, Headlamp, Gitea and forward-auth.

Three instances give each Cluster a streaming standby CNPG can promote in
seconds, repointing the -rw Service with no volume re-attach and no restore.

UNBLOCKED BY WORKING BACKUPS, NOT A SUBSTITUTE FOR THEM. Standbys replicate
corruption and deletions faithfully; only the Barman archive is a backup, and
adding standbys while archive_mode=on with a failing archive_command would
have multiplied the disk-full clock, not hedged it. All three Clusters on
both k3s-prod and k3s-dev now report ContinuousArchiving=True with a real
firstRecoverabilityPoint (verified read-only, 2026-09-07).

Scaling out is non-disruptive on its own: CNPG creates -2 then -3
sequentially, each seeded by pg_basebackup from the RUNNING primary. The
primary is not restarted and clients are not disconnected. The anti-affinity
change that DOES cause a switchover is deliberately the next commit.

DEV IMPACT: identical — dev is on the same shape (three single-instance DBs,
all on k3s-dev-3) and gets the same scale-out. Dev has three untainted nodes
and Longhorn headroom for it (see the next commit's message for the numbers).

Cost: 2 extra 5Gi PVCs per Cluster; ~30 GiB provisioned cluster-wide against
live pgdata of ~600 MiB each, on Longhorn with 200% over-provisioning.

Verified: yamllint clean; `kubectl kustomize --load-restrictor
LoadRestrictionsNone platform-config` and `kubectl kustomize apps/wave-0`
both render `instances: 3`.

Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit ed6fe6cb019b96f677a92448a0347f979d1a6d02)
2026-09-07 23:41:31 -05:00
jakman85 a624c638ef merge: docs/node-1-drive-replaced
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
2026-09-07 23:05:59 -05:00
jakman85 90b39de527 merge: fix/swap-off-persists-across-reboot 2026-09-07 23:05:59 -05:00
jakman85 0247d796a3 merge: fix/founder-moved-to-node-2 2026-09-07 23:05:59 -05:00
jakman85 c9e9dcf960 merge: fix/r-env-03-overlay-selected-credentials 2026-09-07 23:05:59 -05:00
jakman85 89cf59ffb7 docs(conformance): the GIVEN/WHEN/THEN header states the exception it now allows 2026-09-07 23:04:39 -05:00
jakman85 e319777849 docs(pm): draft the R-ENV-03 amendment, and name the gate hole that hid it
The requirement's literal reading ("no prod-specific artifact in a shared path")
forbids the environment-selected PAIR that R-ENV-02 prescribes for a value that
must differ per environment while the manifests stay identical — and a Secret's
difference cannot live in cluster-vars, because it is ciphertext in a SOPS file.
Drafted wording that keeps the guard's teeth (an artifact only prod APPLIES
still fails) and removes the false positive, for the System Architect: the spec
repo is read-only here and CI-gated by R-TEST-06.

Also recorded, checked rather than assumed: the merge that turned R-ENV-03 red
was gated with a touched-requirement subset that never selected R-ENV-03, so
ci/l0.sh Stage 3b filtered the already-red check out of the run that printed
PASS. Replaying that delivery's twelve paths through
fix/touched-requirements-fail-closed's requirement-paths.map still yields no
R-ENV-03, so landing that branch would NOT have caught this. The two changes
that would (unconditional whole-repo L0 stages; a changed conformance test
attributing its own requirement labels) are ci/ work and are left to that
branch, not smuggled in here.
2026-09-07 23:02:56 -05:00
jakman85 730d12ddc4 fix(conformance): R-ENV-03 shared-path rule tests the harm, not the filename
The blanket regex around RENDERED_DIRS failed any tracked file whose name
carries a delimited `prod` token. That caught the four per-environment backup
credential halves (BACKUP_CREDENTIALS_SUFFIX) even though BOTH halves are
applied to BOTH clusters as base resources and the overlay selects only which
one the shared consumer REFERENCES — i.e. exactly the R-ENV-02 contract the
regex exists to protect. Renaming the files to dodge the regex was rejected as
gaming the guard, so the rule now encodes the property instead.

The fix is not an allowlist of those four paths. A prod-token file in a shared
path is excused only when the tree proves all of:

  (A) it declares nothing but Secrets whose names end in the environment suffix
  (1) its own directory's kustomization applies it, and EVERY environment value
      of the selector resolves to a Secret that some resource of that same
      kustomization declares, in the same namespace
  (2) nothing names the suffixed Secret — or the suffixed file — literally;
      every reference is `<base-name>${VAR}`
  (3) that VAR is defined for the base and for every overlay, and prod's value
      is the one that selects this file

References are read from parsed YAML values, never raw text: the mechanism is
spelled out by name in comments beside the resources, and a comment references
nothing.

No other assertion is weakened; the check still runs 96 assertions, and the
pass output now notes each excused file and why.
2026-09-07 22:55:58 -05:00
jakman85andClaude Opus 5 df1d2ab44d docs(prod): node 1's drive was replaced — and §2's exit condition still is not met
Records the 2026-09-07 I/O-starvation incident and the drive swap: the C400 was
replaced with a Samsung 850 EVO, and under the same load class (an active
18-replica Longhorn rebuild, with node 1 as the write target) I/O pressure fell
from full avg300 11.36 to 3.79 — from ~4x worse than its peers into node 3's band.

Updates the three §0 cells that had become false (disk model, drive health, kernel
— all three nodes now run 7.1.8, not 6.12) and adds §0c rather than rewriting the
'NOT hardware-symmetric' analysis, which is still the correct reasoning about the
state it described.

Two things the rebuild taught, recorded so the next one is cheaper: Longhorn
refuses a replaced disk with DiskFilesystemChanged and needs disable -> remove ->
re-add in that order, and the inventory's [k3s_prod_primary] decides BOTH
--cluster-init and k3s_api_host, so rebuilding the founder without moving that
designation would have founded a second cluster.

The replacement does NOT close §2: etcd and Longhorn still share one SATA spindle.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 22:50:45 -05:00
jakman85andClaude Opus 5 69a25cff82 fix(node_health): say 'active via <unit>', not a bare parenthetical
The unit name in the swap-active detail read as an unlabelled aside. Naming it as
the unit that activated the swap is what makes the line actionable.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 22:42:38 -05:00
jakman85andClaude Opus 5 c1e36d9353 docs(node_baseline): say why the Longhorn claim guard's swap list is now always empty
The claim guard builds node_baseline_lh_protected_devices from ansible_mounts plus
`swapon --show`. With swap.target masked, a baselined node has no active swap, so
that half of the list contributes nothing — which reads like a guard that quietly
stopped guarding. It is not: the claim candidate comes from whatever is mounted at
node_baseline_longhorn_claim_mountpoint, and a swap partition is neither mounted nor
ext4, so it could never be the candidate. Say so where the next reader will ask.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 22:39:39 -05:00
jakman85andClaude Opus 5 9d00a3fcd3 fix(node_health): memory.swap blamed the wrong thing, and could not see the defect
The WARN read "roles/node_baseline runs swapoff and neutralises the fstab entry;
this node has not been baselined yet". That was a MIS-DIAGNOSIS. All three prod
nodes HAD been baselined, had their fstab swap line commented out by the role, and
were still swapping 977 MiB each after a reboot. Anyone reading that WARN would
have re-run the role that had already run, and watched swap come back next boot.

The detail now names the actual mechanism — systemd-gpt-auto-generator re-creating
a .swap unit from the partition TYPE GUID 0657fd6d-a4ab-43c4-84e5-0933c84b4f4f at
every boot, without reading /etc/fstab — says in as many words that this is not
"the node has not been baselined", and points at `node-baseline.yml --tags system`.

The check was also blind in the other direction: SwapTotal == 0 is not evidence
that swap is disabled, only that nothing has been swapped on yet this boot, so a
node freshly swapoff-ed PASSed while still set to swap again on reboot — exactly
the state the old role left behind. The collector now reports memory.swap_persistence
(swap.target masked?, .swap units the generators emitted this boot, which of them
are masked, leftover active fstab swap lines — all world-readable, no root, no
systemctl parsing), and the check WARNs on "clean now, swapping after the next
reboot".

Rendered against every fact shape (swap active + unmasked, swap off + unmasked,
after the fix, a dev VM with no swap partition at all, per-unit mask only,
persistence unreadable, an older collector's output with the key absent, and
/proc/meminfo unreadable) and verified live on k3s-prod-1 post-reboot:
  [PASS] memory.swap — no swap active and none can be activated: swap.target is masked

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 22:35:31 -05:00
jakman85andClaude Opus 5 0e92d258dd fix(node_baseline): swap-off now survives a reboot — mask swap.target
The role's `swapoff -a` + commented fstab line held only until the next boot,
on every prod node. Measured 2026-09-07: all three came back from a reboot with
977 MiB of swap ACTIVE (k3s-prod-1 /dev/sda4, k3s-prod-3 /dev/sda4, k3s-prod-2
/dev/nvme0n1p4) while their fstab swap entry was commented out by this role. The
role reported success and the setting silently expired.

CAUSE. systemd-gpt-auto-generator claims any GPT partition typed
0657fd6d-a4ab-43c4-84e5-0933c84b4f4f ("Linux swap") from the PARTITION TABLE,
synthesises a .swap unit into /run/systemd/generator.late/ and links it into
swap.target.wants/. It never reads /etc/fstab, so neutralising fstab cannot stop
it. Re-running the generator by hand on k3s-prod-1 says so verbatim:
  Probed fstype 'swap' on partition /dev/disk/by-diskseq/1-part4.
  Adding swap: /dev/disk/by-diskseq/1-part4

MECHANISM. Mask swap.target. `systemctl show` on the generated unit says its only
inbound dependency is `WantedBy=swap.target` — Requires/RequiredBy/BoundBy empty,
and the matching .device unit has an empty Wants=, so a device appearing does not
pull its swap unit in by itself. swap.target is the single choke point for every
activation path: gpt-auto, systemd-fstab-generator, systemd-cryptsetup-generator.
It is device-shape independent (no device name appears in the change, so one task
covers sda4, nvme0n1p4 and dev's vda VMs), boot-safe (swap.target is Wants=/After=
from sysinit.target and After= from tmp.mount; RequiredBy= is EMPTY on all three
prod and all three dev nodes, and a masked unit reached only via Wants= is skipped,
not failed), and inert if the device is absent or renamed. The generated units are
masked too, as a second independent barrier.

Rejected: `systemd.gpt_auto=0` (disables GPT auto-discovery wholesale, needs a GRUB
edit + reboot to become true, and on DEV would switch off root auto-discovery — dev's
vda1 carries the discoverable GUID 4f68bce3-e8cd-4db1-96e7-fbcaf984b709); clearing
the partition's GPT type GUID (a partition-table write to the live boot disk of a node
serving etcd and Longhorn replicas, and an irreversible decision about that 977 MiB
that belongs to the operator); `systemd.swap=0` (systemd-fstab-generator only — this
swap does not come from fstab).

`swapoff -a` is now conditional on `swapon --show` returning something, so the task
reports honestly instead of hiding a command that ran anyway behind changed_when: false.

PROVEN on k3s-prod-1 (all 18 Longhorn volumes healthy before the reboot):
  run 1 changed=2, run 2 changed=0
  after `reboot`: /proc/swaps empty, `free` Swap 0B, swap.target masked, the
  generated unit still emitted by the generator and still masked, `systemctl
  --failed` empty, `systemctl is-system-running` = running.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 22:35:16 -05:00
jakman85andClaude Opus 5 08fbe85638 fix(ansible): move the founder designation to node 2
Node 1's SSD failed and was replaced; Debian was reinstalled, so node 1 is now a
JOINING node with no cluster state. It was still listed in [k3s_prod_primary],
and that group decides two things at once:

  1. which play gets k3s_server_mode: primary -> --cluster-init, which FOUNDS A
     NEW CLUSTER, and
  2. what k3s_api_host resolves to -- the --server join target.

So re-provisioning node 1 unchanged would have pointed a rebuilt, empty node at
ITSELF and founded a second cluster beside the live one. The role's
`creates: /etc/rancher/k3s/k3s.yaml` guard does not help: on a fresh install that
file is absent, so cluster-init WOULD have run.

Node 2 is the senior surviving server (up since 2026-09-01) and takes the founder
slot. Its own cluster-init task is inert precisely because k3s.yaml already exists
there. Verified after the change: k3s_api_host renders 192.168.2.11, and
groups[k3s_primary_group][0] is k3s-prod-2.

This is a latent gap the role documented but did not handle -- its comment assumes
the founder is always alive ('On a WARM single-node rejoin the founder is up and
answering too'). Rebuilding the founder itself was not covered.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 21:41:54 -05:00
jakman85 863af01062 docs: the §2 shared-device measurement came back, and it does not hold
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
infra/PROD-PROVISIONING.md §2 recorded the etcd/Longhorn shared-partition risk as
"accepted to be MEASURED, not assumed", with an exit condition. The measurement
arrived on 2026-09-07 in production, 2 d 7 h after bring-up, as an outage.

§2 now carries "THE MEASUREMENT CAME BACK": the journal evidence, the three-node
IO PSI table (11.36 / 0.75 / 1.29), and the honest verdict that the section's own
premise — "every prod node is SSD or NVMe, where rebuild I/O costs latency rather
than collapsing it" — was right about node 2 and wrong about node 1. A 2011 C400
with 61,440 reallocated sectors and no power-loss protection is not the "SSD" that
reasoning assumed, and §0 had already named the three-way drive asymmetry as the
fault line. The exit condition is OPEN and the second bullet (dedicated device)
applies; that remains the operator's.

Also records what the three software mitigations in this branch do and do not buy,
including the point that no cgroup or scheduler control arbitrates the ext4 journal
both tenants share — and the gap that etcd's own fsync histograms, which the exit
condition literally asks for, are not collected because kubeEtcd is disabled for
k3s. PSI is the signal that exists.

DECISIONS-LOG gains the full entry: the four I/O mechanisms found INERT on these
nodes (IOSchedulingClass, IOSchedulingPriority, io.latency, bare IOWeight), the
finding that concurrentReplicaRebuildPerNodeLimit was already set and already held
during the outage, the third instance of config that looked applied and was not
(orphanAutoDeletion), and the three things still requiring the operator.

infra/ansible/README.md: CORRECTED a stale claim that k3s-prod-2/-3 "stay commented
out" of the inventory. They have been live in [k3s_prod_secondary] since
2026-09-03. That sentence is exactly the kind used to reason about blast radius —
`-e k3s_hosts=k3s_prod` targets THREE hosts, not one — so it is replaced with the
--list-hosts command that settles it. The new etcd-io-priority play is added to the
run order, the layer table and the prod command list.

Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit a00a27a2567ea6a805c8c196b75df2c92a7c63a4)
2026-09-07 02:46:26 -05:00
jakman85 f7a9229d81 fix(ansible): the NVMe node would have died on an undefined variable
Self-review of the role I had just written. On k3s-prod-2 the scheduler path is
deliberately skipped, so the `stat` on /sys/fs/cgroup/system.slice/k3s.service is
skipped too — and a SKIPPED registration has no `.stat` key at all. Ansible
templates `loop:` BEFORE it evaluates `when:`, so

    loop: "{{ ['system.slice'] + (['system.slice/k3s.service']
              if _eio_k3s_cgroup.stat.exists else []) }}"

raises on that host even though the task would never have run. The role would have
failed on the one node it is designed to leave alone — and it would have failed at
the very end, after doing nothing, which reads like a broken mitigation rather than
an intentional skip.

`| default(false)` on both uses. Exercised offline against a simulated skipped
registration; `--syntax-check` re-run.

Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit 5ccf5111a54857abb934a64c41332b7aa5046323)
2026-09-07 02:46:26 -05:00
jakman85 a819c50ffe feat(ansible): arbitrate the shared etcd/Longhorn device — and prove it landed
roles/etcd_io_priority + playbooks/etcd-io-priority.yml, enabled for [k3s_prod]
only. Mitigation for the 2026-09-07 k3s-prod-1 outage: etcd's WAL fdatasync took
3.4-6.7 s (expects 1 s), the node went NotReady while the host stayed up, and
/proc/pressure/io full avg300 read 11.36 there against 0.75 (node 2, NVMe) and
1.29 (node 3).

WHAT IT DOES, on the SATA nodes only:

  1. k3s-etcd-io-priority.service (oneshot, Before=k3s.service) switches the
     shared device to BFQ and drops nr_requests 64 -> 32.
  2. IOWeight=10000 drop-ins on system.slice AND k3s.service, which BFQ turns
     into io.bfq.weight=1000 — a 10:1 share against the default-weighted
     kubepods.slice where Longhorn's instance-managers live.
  3. Reads all of it back out of the kernel and ASSERTS. If the scheduler did
     not switch or the weight did not land, the play fails.

THE THREE OBVIOUS ANSWERS ARE INERT ON THESE NODES, ALL CHECKED LIVE, ALL
DELIBERATELY ABSENT:

  * IOSchedulingClass=realtime on k3s.service. mq-deadline DOES honour the
    priority class, so this looks right — but ioprio is inherited across fork
    and kept across exec (block/blk-ioc.c __copy_io copies ioc->ioprio) and
    CONTAINERD IS A CHILD OF THE K3S SERVER PROCESS. Verified on k3s-prod-1:
    containerd pid 84553 has ppid 84333 = k3s-server, and
      ionice -c2 -n0 -- sh -c 'ionice -p $$; sh -c "ionice -p \$\$"'
    prints best-effort prio 0 at both levels. Realtime for k3s is realtime for
    every container on the node, Longhorn included. No relative gain.
  * IOSchedulingPriority (the level inside a class): mq-deadline buckets by
    CLASS only, so BE-0 vs BE-4 is a no-op.
  * io.latency, which would be the ideal scheduler-independent protection:
    CONFIG_BLK_CGROUP_IOLATENCY is not built in Debian trixie's 7.1.8 kernel
    (only IOCOST is) and no io.latency file exists under /sys/fs/cgroup.

AND ONE NON-OBVIOUS TRAP: bare IOWeight= is DECORATION here. systemd writes it
to io.weight (blk-iocost — compiled in but disabled, io.cost.qos is empty) and
io.bfq.weight (BFQ — not loaded). Neither reader exists on the stock nodes, so
the scheduler change and the weights have to be one role with one verification.
The value is 10000 rather than a "reasonable" 1000 for the same reason: systemd
rescales 1..10000 onto io.bfq.weight's 1..1000, so 1000 maps to ~101 against
BFQ's unweighted default of 100. The role asserts the result is >= 500.

k3s-prod-2 IS SKIPPED ON PURPOSE. The role discovers the device under
k3s_data_dir per node (verified: prod-1/3 -> /dev/sda5 -> sda, prod-2 ->
/dev/nvme0n1p5 -> nvme0n1) and writes NOTHING on NVMe rather than leaving inert
files behind — BFQ is a poor fit for NVMe and node 2 measured 0.75 IO PSI.

DEV IS UNTOUCHED. Master switch defaults false and is set true only in
group_vars/k3s_prod; dev has a dedicated etcd device (docs/plans/ssd-etcd-
isolation.md) and nothing to arbitrate. Verified with --list-hosts: no extra
vars -> the three dev hosts (where the role is a no-op); -e k3s_hosts=k3s_prod
-> the three prod hosts; --limit k3s-prod-3 -> one.

NOTHING IS RESTARTED. The oneshot writes two sysfs files; the weights go in
through daemon-reload. If daemon-reload turns out not to re-realize a running
k3s.service's cgroup, the verification FAILS and names the rolling restart as
the fix rather than quietly leaving the weight unapplied.

WHAT IT CANNOT DO, stated in the role README and worth repeating: it does not
separate the two tenants. Both still write one ext4 filesystem, and no cgroup
control arbitrates the shared jbd2 journal that etcd's fdatasync waits on. It is
also UNPROVEN on this hardware — it fails loudly if the settings do not land, it
cannot tell you they helped. Apply to k3s-prod-3 first and watch
NodeIOPressureHigh. Replacing the C400 / giving etcd its own device remains the
real fix and the operator's (PROD-PROVISIONING.md §2 exit condition).

Syntax-checked, and the fact-derivation and assert expressions were exercised
offline against all three node shapes plus negative controls (a node still on
mq-deadline fails the gate; IOWeight=1000 -> 101 fails the floor).

Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit de8c1f832724c011b03466842e18296de07a6776)
2026-09-07 02:46:26 -05:00
jakman85 efc6db6c1f feat(obs): alert on I/O pressure — the metric that missed the prod-1 outage
On 2026-09-07 06:05:21Z k3s-prod-1 went NotReady ("Kubelet stopped posting node
status") with the host up, pingable and SSH-able. The cause was disk I/O
starvation of the embedded etcd, from its own journal:

  slow fdatasync               took=3.56s / 6.71s / 4.95s / 3.40s (expects 1s)
  apply request took too long  took=8.30s                         (expects 100ms)
  etcdserver: too many requests

/proc/pressure/io FULL avg300 read 11.36 on k3s-prod-1 against 0.75 on
k3s-prod-2 (NVMe) and 1.29 on k3s-prod-3 — all three nodes put /data/k3s
(etcd's WAL) and /data/longhorn (every replica) on ONE partition, and node 1's
is a 2011 consumer SATA SSD with 61,440 reallocated sectors.

The stack alerted on CPU-PSI and memory and had NO I/O alert at all, so the
first signal the operator got was a node already out of the cluster. Adds:

  NodeIOPressureHigh      full IO PSI > 5%  for 10m  warning
  NodeIOPressureCritical  full IO PSI > 10% for 5m   critical

Thresholds are measured, not guessed: rate() is a 0-1 fraction, so the outage's
11.36% is 0.1136 and the healthy steady state on the same node is ~0.01.
Verified live that node-exporter already exports
node_pressure_io_{stalled,waiting}_seconds_total for all three prod nodes, so
these rules fire on arrival — no new exporter, no scrape change. (etcd's own
fsync histograms are NOT available: kube-prometheus-stack's kubeEtcd scrape job
is deliberately disabled for k3s, so PSI is the signal we actually have.)

PSI *full* (every runnable task blocked on disk), not *some*: `some` is noisy on
a storage node and did not separate the failing node from its peers.

Dashboard gains the matching I/O row and is retitled accordingly. Not scoped to
prod — dev shares this file and its k3s VMs are HDD-backed, where the same alert
is if anything more likely to earn its keep.

Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit cb28fc80e7e2085d2539530d55f90fc92e3b1bba)
2026-09-07 02:46:26 -05:00
jakman85 09265e0381 merge: OIDC issuer/callback URLs become per-environment
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
2026-09-07 02:12:15 -05:00
jakman85andClaude Opus 5 e65a76dc51 docs(log): record the prod OIDC-URL outage and the rule it establishes
Non-secret, environment-specific values must not live inside an encrypted
Secret: Flux's ${BASE_DOMAIN} substitution rewrites rendered manifest text and
cannot reach inside ciphertext, so such a value silently freezes at whatever
one environment happened to need. Dev was correct only by coincidence (dev IS
the apex); prod was wrong from its first reconcile, for both Headlamp and
Semaphore. StrictPostBuildSubstitutions cannot catch this — there is no
unresolved ${...} to complain about.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 02:10:10 -05:00
jakman85andClaude Opus 5 bc9c93d2ab fix(keycloak): the reconciler could not read its own Secret
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
Shipped in the previous delivery and failing on every run since:

  FATAL: cannot read /etc/keycloak-realm-clients (PermissionError)
  BackoffLimitExceeded

The client-secrets volume used defaultMode 0400 — owner-only — but the owner of a
projected Secret volume is root, while the pod runs runAsUser/runAsGroup 65532.
The sibling script volume worked only because it is 0444. So the healer that
exists to correct client-secret drift was itself inert.

It failed CLOSED (no writes attempted, no drift introduced), and the two alerts
shipped alongside it are exactly what surfaces this: KeycloakClientSecretReconciler
Failing fires on the failed Jobs. The observability caught its own subject.

Fixed with fsGroup 65532 + defaultMode 0440 rather than relaxing to world-readable
0444: fsGroup makes the projected volume group-owned by the running user, so the
secret stays unreadable to any other uid in the container.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 02:07:28 -05:00
jakman85andClaude Opus 5 6548f9a89c fix(semaphore): per-environment OIDC URLs — same latent break as Headlamp
Semaphore's whole SEMAPHORE_OIDC_PROVIDERS JSON blob lived inside the SOPS-encrypted
semaphore-bootstrap Secret (key `oidc-providers-json`) because it CONTAINS the confidential
client secret. That buried two environment-specific, NON-SECRET URLs in ciphertext —
provider_url (the realm issuer) and redirect_url — where Flux's ${BASE_DOMAIN} postBuild
substitution cannot reach. Both were frozen at `home.keithjackson.dev`, so prod's Semaphore
was pointed at dev's Keycloak and dev's own redirect URI: identical root cause to the
Headlamp login timeout, not yet reported only because nobody had tried prod SSO.

FIX: split the blob. The URLs are public and now live as a plaintext literal in
catalog/semaphore.yaml (rendered to the apps/wave-1 golden), ${BASE_DOMAIN}-substituted like
every other hostname. Only the client secret stays encrypted, in its own key
`oidc-client-secret`, and the KUBELET splices it back via dependent-env-var expansion:
$(SEMAPHORE_OIDC_CLIENT_SECRET) resolves against the secretKeyRef env var declared
immediately above it (order is load-bearing — the kubelet resolves env in list order and
leaves a forward reference as literal text).

Verified: the new key holds the same value as keycloak-realm-clients/semaphore-client-secret
(the mirror invariant), Semaphore v2.19.10's util.OidcProvider consumes client_secret as
before, and `flux envsubst --strict` leaves $(...) untouched while resolving ${BASE_DOMAIN} —
dev renders keycloak.home.keithjackson.dev, prod keycloak.prod.home.keithjackson.dev.

apps/wave-1/semaphore.yaml is a render golden; regenerated with the canonical
`catalog/render.py --all catalog -o apps --wave-layout --no-image-automation`.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 02:06:52 -05:00
jakman85andClaude Opus 5 bfa39bc5d0 fix(headlamp): per-environment OIDC URLs — prod was dialing DEV's Keycloak
The operator could not log in to Headlamp on prod:

  Get "https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration":
  dial tcp 192.168.1.53:443: i/o timeout

ROOT CAUSE: OIDC_ISSUER_URL and OIDC_CALLBACK_URL were keys INSIDE the SOPS-encrypted
headlamp-oidc-credentials.sops.yaml. Flux's ${BASE_DOMAIN} postBuild substitution rewrites
the RENDERED manifest text and can never reach inside ciphertext, so both values were frozen
at the apex `home.keithjackson.dev` in BOTH environments. Dev IS the apex, so dev looked
correct; prod (BASE_DOMAIN = prod.home.keithjackson.dev) therefore pointed at dev's Keycloak
at 192.168.1.53 — on the dev LAN behind pfSense, unroutable from prod's 192.168.2.0/24.

FIX: an issuer URL and a callback URL are PUBLIC. Move both out of the Secret into the
HelmRelease's spec.values.env, written with ${BASE_DOMAIN} so substitution resolves them per
environment. The Secret keeps only what is genuinely secret or environment-invariant
(OIDC_CLIENT_SECRET, OIDC_CLIENT_ID, OIDC_SCOPES).

Exactly ONE source per variable: the chart renders `env:` after `envFrom:`, and an explicit
env entry silently WINS over the same name from envFrom — so the two sets must not overlap,
and they no longer do. The five load-bearing names the chart's $(...) args require
(OIDC_CLIENT_ID/CLIENT_SECRET/ISSUER_URL/SCOPES/CALLBACK_URL) are all still delivered;
verified by rendering platform/headlamp per overlay, `flux envsubst --strict`, and
`helm template` of chart 0.41.0 against the substituted values.

Side effect, deliberate: naming OIDC_CALLBACK_URL in `env:` makes the chart finally EMIT
`-oidc-callback-url=$(OIDC_CALLBACK_URL)` — it gates that arg on the value being set via
`env:`/`config.oidc`, which a Secret key alone never tripped, so the Secret's callback URL
had been dead weight. The value is byte-identical to the single redirectUri registered on the
Keycloak `headlamp` client (also ${BASE_DOMAIN}-templated), the only URI Keycloak accepts.

[parity] holds: prod's kube-apiserver oidc-issuer-url is an install-time k3s flag from
k3s_base_domain=prod.home.keithjackson.dev, which is exactly what Headlamp now uses.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 02:06:38 -05:00
jakman85 3ca4ad6011 merge: R2 off-cluster backups + per-environment backup credentials
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
2026-09-07 01:55:14 -05:00
jakman85andClaude Opus 5 708a05dfc1 fix(devcontainer): restore multi-cluster kubeconfig selection
infra/devcontainer/kubeconfig.sh was never staged into the delivery-A commit, so
the branch built on main carried main's single-cluster tooling and `--context
k3s-prod` failed with 'context does not exist' mid-incident. Not Flux-reconciled,
so it never affected the cluster — but it breaks prod diagnosis from a checkout of
this branch, which is when you need it most.

Carries the concurrency fix: the merged kubeconfig holds both contexts and NO
current-context, with each invocation getting a per-invocation selector file
prepended to KUBECONFIG. Two concurrent dev.sh runs with different CLUSTER values
previously overwrote each other's current-context mid-command — reproduced live,
a CLUSTER=dev reader returned k3s-prod nodes on 3 of 7 iterations.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 01:55:14 -05:00
jakman85andClaude Opus 5 161430d915 feat(backup): the overlay picks the credential — BACKUP_CREDENTIALS_SUFFIX
The previous commit created the two halves. This one wires the choice, so dev renders MinIO
credentials and prod renders R2 credentials from the SAME base manifests (R-ENV-02: the
overlay patch is the entire difference between environments; no base manifest is forked).

    cluster-vars  BACKUP_CREDENTIALS_SUFFIX   base ""  |  dev ""  |  prod "-prod"

    velero HelmRelease  credentials.existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}
    3x CNPG Cluster     s3Credentials.*.name:      <db>-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}

ONE SUFFIX, NOT FOUR SECRET-NAME KNOBS. Four independent knobs can be updated three-quarters
of the way, leaving one consumer reading the OTHER environment's key with nothing rendering
wrong — and StrictPostBuildSubstitutions cannot catch that: strict mode fails on an UNDEFINED
variable, never on a defined-but-stale one. With one suffix the four move together or not at
all. The cost is that the rendered Secret name is not greppable from the consumer, so both
kustomizations spell the two names out in full beside the resources.

THE BASE DEFAULT IS "" AND THAT IS LOAD-BEARING. It renders the UNSUFFIXED names — objects
that exist in every render, and that dev is running today. The alternative failure mode is
strictly worse than the shared-Secret state this replaces: velero's pod hangs in
ContainerCreating forever when `existingSecret` names a missing Secret (the volume never
mounts), and a CNPG Cluster whose s3Credentials Secret is absent stops archiving WAL — which
under archive_mode=on is a filling pgdata PVC, not a missing backup. So an unresolvable
default breaks the referencing cluster HARDER than a wrong key does. Both overlays state the
value explicitly anyway, so neither inherits it silently.

Prod's overlay change is a cluster-vars op only. The velero Flux Kustomization patch is
untouched (it stays decryption-only, and velero stays suspended on prod).

VERIFIED — `kubectl kustomize --load-restrictor LoadRestrictionsNone` on clusters/dev and
clusters/prod for the cluster-vars values, then on each of the four affected tiers (platform,
platform-config, platform/velero, apps/wave-0) piped through `flux envsubst --strict` with
that environment's variables:

  strict envsubst              PASSES for all 8 (env x tier) combinations
  dev   velero existingSecret  velero-cloud-credentials
  dev   keycloak/nextcloud/semaphore-db  <db>-backup-credentials         (both key refs each)
  prod  velero existingSecret  velero-cloud-credentials-prod
  prod  keycloak/nextcloud/semaphore-db  <db>-backup-credentials-prod    (both key refs each)
  both  all EIGHT Secret objects present in the rendered output (dev and prod alike)
  no `${...}` survives in any rendered tier except gitea's three DELIBERATE `$${...}` shell
    escapes in apps/wave-0, which are pre-existing and unrelated.

Read alongside the endpoints the same overlays already set, the two environments are now
isolated on credential AND target:
  dev   velero s3://velero (root) + cnpg s3://cnpg      @ http://192.168.1.9:9000  region us-east-1
  prod  velero k3s-prod-backup/velero + barman prefix   @ https://<acct>.r2.cloudflarestorage.com  region auto

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 01:41:56 -05:00
jakman85andClaude Opus 5 e7c4a95c2d feat(backup): the four backup credentials become TWO objects each, one per environment
The four backup-credential SOPS Secrets are SHARED BASE manifests: platform/ and apps/ are
rendered by the same Flux tiers on both clusters, and the ONLY per-environment difference in
this repo is cluster-vars substitution (R-ENV-02). So writing the R2 token into all four
(0a6a862) did not "give prod credentials" — it REPLACED DEV'S. Merging that as-is would have
handed dev's Velero and all three dev CNPG Clusters an R2 key id to sign MinIO requests with:
403 on every WAL segment, and archive_mode=on turns that into a disk-full clock on the pgdata
PVC, not a degraded backup. Dev's PROVEN DR restore (R-STATE-03) is the thing that would have
broken.

This commit splits the material. Each credential is now TWO Secret objects, both applied to
BOTH clusters, differing only in name suffix:

    dev  -> velero-cloud-credentials              keycloak/nextcloud/semaphore-db-backup-credentials
    prod -> velero-cloud-credentials-prod         ...-db-backup-credentials-prod

The consumer picks one by name; that wiring is the NEXT commit, so this one changes no
rendered reference and is a no-op for both live clusters.

WHY DEV KEEPS THE UNSUFFIXED NAME. Dev is live. Renaming its Secret would change
`s3Credentials.accessKeyId.name` on three running CNPG Clusters — which projects into the
instance pods' environment and therefore ROLLS keycloak-db, nextcloud-db and semaphore-db —
and would roll the velero Deployment, all to rename an object whose contents do not change.
This repo already makes exactly this call for dev's live identifiers (EXTERNALDNS_TXT_OWNER_ID
"default", VELERO_BACKUP_PREFIX ""): the live cluster keeps the name it is running, and the
NEW environment takes the new one. The asymmetry is deliberate and documented at each
kustomization.

WHERE THE MATERIAL CAME FROM. Neither side is invented:
  - the four unsuffixed files are restored byte-for-byte from `main` (the MinIO material as
    it stood before 0a6a862), so dev's Secrets are IDENTICAL to what dev is running;
  - the four `-prod` files carry the R2 token from 0a6a862 (already on this branch as
    9739e9b), re-encrypted only because metadata.name changed.

RE-ENCRYPTION, AND WHY A COPY+SED IS NOT ENOUGH. `.sops.yaml` sets
`encrypted_regex: ^(data|stringData)$`, so metadata.name stays cleartext — but SOPS MACs the
WHOLE document (there is no `mac_only_encrypted` here), so editing that cleartext line
invalidates the MAC and `sops -d` then refuses the file. The `-prod` files were therefore
produced inside the dev toolbox (infra/devcontainer, which has sops 3.9.4 and the age key
mounted) by decrypt -> rename -> re-encrypt, with the plaintext never leaving a container
tmpfile. Note for the next person: `sops -e --output NEW OLD` matches creation rules against
the INPUT path, so encrypting a tmpfile into a repo path needs `--filename-override NEW` or
it dies with "no matching creation rules found".

Verified: all eight files `sops -d` cleanly; the four dev files hash to dev's material and the
four prod files to R2's, and the two sets differ. `.sops.yaml` needed no new rule — the
existing `(platform|apps|catalog)/.*(secret|sops)\.ya?ml$` rule already matches
`*-prod.sops.yaml`, so the new files encrypt to the same age recipient.

apps/wave-0/kustomization.yaml is GENERATED, so render.py learns the `-prod` sidecar rather
than the golden being hand-edited: `<db>-backup-credentials-prod.sops.yaml` now places into
the DB's wave alongside its unsuffixed twin (both must exist before the Cluster that names
one), and the selftest asserts it.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 01:38:22 -05:00
jakman85andClaude Opus 5 0a0527e7db feat(backup): prod's off-cluster DR lands on R2 — and stops the WAL clock
Prod has been archiving WAL to `TBD-PROD-S3-BACKUP-ENDPOINT` since bootstrap.
Verified live 2026-09-07 on all three CNPG Clusters: ContinuousArchiving=False,
"unexpected failure invoking barman-cloud-wal-archive: exit status 4". Because
archive_mode is ON, that is not a degraded backup — it is a DISK-FULL CLOCK.
Postgres will not recycle a segment it has not archived, so keycloak-db's pgdata
PVC is at 579 MiB used / 4360 MiB free and climbing monotonically. When it fills,
the primary PANICs and takes SSO down, and with it kubectl OIDC, Grafana,
Headlamp, Gitea and forward-auth. Wiring the endpoint is what lets Postgres drain
the whole accumulated backlog in one pass.

THE KNOB LAYOUT (three new, one repurposed) and why it is shaped this way.

The operator created exactly ONE R2 bucket, `k3s-prod-backup`, but the two backup
tools want differently-SHAPED values: the Velero chart takes `bucket` and `prefix`
as separate fields, while CNPG's barmanObjectStore.destinationPath is one full
s3:// URL whose first path segment IS the bucket. So:

  S3_BACKUP_ENDPOINT     dev http://192.168.1.9:9000   prod https://<acct>.r2...
  S3_BACKUP_REGION       dev us-east-1                 prod auto
  VELERO_BACKUP_BUCKET   dev velero                    prod k3s-prod-backup
  VELERO_BACKUP_PREFIX   dev ""                        prod velero
  CNPG_BACKUP_PATH       dev s3://cnpg                 prod s3://k3s-prod-backup/barman

There is deliberately NO separate `S3_BACKUP_BUCKET`: CNPG's bucket already lives
inside CNPG_BACKUP_PATH, and a second knob holding the same bucket name twice is a
rotation waiting to drift. Three knobs, each shaped for its own consumer, and no
two of them can contradict each other.

TENANCY. Inside k3s-prod-backup, velero and barman get disjoint TOP-LEVEL
prefixes — s3://k3s-prod-backup/velero/ and s3://k3s-prod-backup/barman/<cluster>/
— with the bucket ROOT left empty so a misrendered prefix lands somewhere
obviously wrong instead of silently colliding. This is not tidiness: Velero ADOPTS
every backup under its location and TTL-DELETES it on retention, so an overlapping
prefix would let it reap WAL segments; and interleaving two Postgres servers' WAL
into one prefix corrupts the chain outright. Dev↔prod isolation is doubly
independent — different endpoint AND different bucket names — so neither cluster
can reach the other's objects even with a mis-substituted prefix.

DEV IS UNCHANGED, BYTE-FOR-BYTE, and that is asserted rather than assumed. Base
defaults are dev's LIVE values (verified against the running cluster: BSL
bucket=velero with NO prefix, region us-east-1; all three CNPG Clusters
ContinuousArchiving=True at s3://cnpg on the MinIO), and the dev overlay now
restates all five explicitly for the same reason LONGHORN_DATA_PATH is restated —
a silent drift in base would ORPHAN the backups dev's PROVEN restore depends on.
The empty-prefix case was the one real risk, and it is settled empirically: with
prefix "" the chart omits objectStorage.prefix from the rendered
BackupStorageLocation ENTIRELY (helm template, chart 12.1.0), so dev's live BSL is
unchanged, not merely equivalent.

R2's TWO GOTCHAS, both wired up front:
  - region must be exactly `auto`. head-bucket answers {"BucketRegion":"WNAM"},
    which is a PHYSICAL LOCATION HINT, not a signing region. This was hardcoded
    us-east-1 inside the Velero HelmRelease — correct-by-accident for MinIO, wrong
    for anything else — so it is a knob now.
  - checksumAlgorithm: "". velero-plugin-for-aws v1.14.2 bundles an AWS SDK with
    flexible checksums on by default and sends x-amz-sdk-checksum-algorithm /
    x-amz-checksum-crc32 on every PUT; R2 rejects those with HTTP 400
    InvalidArgument, failing every backup and every kopia blob upload. It must be
    PRESENT-but-empty (the plugin branches on the key existing), and the chart does
    emit an empty value — verified by rendering it. Harmless against MinIO, so it
    is unconditional rather than a fourth knob.

WHAT THIS DOES NOT DO. Velero stays SUSPENDED on prod. `spec.suspend: true` is
owned by field manager `flux` (verified: kustomize-controller's managedFields do
not include f:suspend), so server-side apply will not strip it, and nothing here
declares suspension — which would also trip R-ENV-03, since a prod overlay may
patch only cluster-vars and a Flux Kustomization's `decryption`. Unsuspending is
the operator's, after `standup.sh --s3-endpoint <r2> s3check` proves a round trip.

NO DNS KNOB IS TOUCHED. BASE_DOMAIN stays prod.home.keithjackson.dev and every
EXTERNALDNS_* value is untouched. The apex inversion on work/prod-standup-campaign
was deliberately NOT carried: this branch ports that branch's R2 reasoning knob by
knob, and merges nothing.

VERIFIED: ci/l0.sh green (flux-postbuild-substitution 272 assertions, 0 hazards,
33 knobs defined / 28 referenced, every new token resolved in BOTH overlays);
render-golden, lint, schema, spec-immutability green; touched requirement
R-STATE-03 covered.

Refs R-STATE-03, R-IDP-05, R-ENV-02, R-ENV-03.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 01:14:29 -05:00
jakman85andClaude Opus 5 dc2d1ce17e fix(conformance): the R-NET-04 scan descended into nested agent worktrees
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
TRACKED_YAML rglob'd from the repo root excluding only .git and _artifacts, so it
also walked .claude/worktrees/agent-*/ — the isolated worktrees created for agent
runs. The check counted 5 CiliumEgressGatewayPolicy objects where exactly 1 is
authored, failing R-NET-04 for a reason unrelated to egress steering. Now skips
dot-DIRECTORIES (covering .git, .claude, .cache) while still matching dot-FILES
such as .sops.yaml. .claude/ is gitignored, so nothing beneath it is authored
surface.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 01:01:53 -05:00
jakman85andClaude Opus 5 6a8953e288 feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.

Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.

WHAT THIS FIXES

1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
   ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
   anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
   to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
   volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
   database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
   the hour, and evaluates to 3 firing series on prod today.
   Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
   nextcloud-db's exporter cannot log in to its own database (permission denied for
   database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
   for it — the obvious metric would have been blind to one database in three.

2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
   by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
   job="kubelet" (the apiserver copy of the same series reports namespace=default),
   and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
   available=0 forever and would otherwise have fired critical permanently.

3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
   firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
   a node carries the egress-gateway label, with dev's behaviour unchanged today.

4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
   LITERALLY and the import still reports success, then sits at Done=True and never
   re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
   $(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
   green. The CronJob is the level-triggered correction; two alerts make its silence
   observable. Repaired live already — this stops it recurring.

Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 00:59:22 -05:00
jakman85andClaude Opus 5 9739e9be91 chore(backup): R2 credentials for the off-cluster DR target
Written by platform/backup/set-r2-credentials.sh (operator, 2026-09-07T05:41:34Z)
into all four consumers, in the two shapes they need: velero wants an AWS INI
file, CNPG/barman wants discrete ACCESS_KEY_ID / ACCESS_SECRET_KEY. All four
verified to carry the SAME 32-character key id, so no consumer was missed.

Proven from a pod INSIDE prod against
https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com:
head-bucket on k3s-prod-backup returns BucketRegion WNAM, and PUT and DELETE both
succeed. Byte-integrity of GET is NOT yet proven (the aws-cli image has no cmp);
s3check is what will establish it.

Credentials only — no endpoint or bucket wiring. That is feat/r2-backup-prod.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 00:58:45 -05:00
jakman85andClaude Opus 5 d1a6628a97 feat(ha): the CNI dials the control-plane VIP — kube-vip becomes a CNI prerequisite
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
OPERATOR DECISION 2026-09-06: "use the VIP as intended — a virtual address to reach
the cluster and provide fault tolerant HA. Without a change it will not work in HA."

THE DEFECT. Cilium's kube-proxy replacement dialled a SINGLE NODE. Re-confirmed live
before changing anything: all three prod cilium-agent pods carried
KUBERNETES_SERVICE_HOST=192.168.2.10 — k3s-prod-1's own address — while the kube-vip
VIP 192.168.2.9 was Running on all three nodes with a held lease. Lose prod-1 and the
CNI's control-plane path is dead on an "HA" cluster. clusters/prod's own comment said
the value must be the VIP and NOT .10, then called it "Not yet decided" — I treated
that stale comment as an open question when the evidence already settled it.

SPLIT, NOT REPOINT, because k3s_api_host had two consumers with opposite needs:
  k3s_api_host            unchanged — the FOUNDER's IP: the cold-bootstrap join
                          target and a tls-san. infra/tofu/outputs.tf GENERATES this
                          name, so renaming would ripple into tofu and the runbooks.
  k3s_apiserver_endpoint  NEW = {{ kube_vip_address }} — the HA address; what Cilium
                          consumes. Derived, not restated, so an environment overrides
                          ONE line and the VIP, the SAN and the CNI endpoint move
                          together. Prod needed no new override.

THE SECOND HAZARD, WHICH MY BRIEF DID NOT NAME and the agent found: pointing Cilium at
the VIP breaks a COLD bootstrap for the same reason the join would — site.yml ran
node-baseline -> k3s -> cilium, so Cilium would install against an address nothing
announced yet. That is the 2026-09-01 deadlock shape. Fixed by SEQUENCE, not value:
site.yml now imports kube-vip.yml BETWEEN k3s and cilium. It cannot go earlier
(kube-vip is a k3s auto-deploy manifest, so it needs a cluster) and it works there
because the DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 — needing
neither CNI nor kube-proxy. roles/kube_vip gained a read-only `wait_for` on the VIP so
a cold run stops THERE with a named cause instead of surfacing as unreachable agents.

Also fixed a pre-existing bug that only mattered once this play joined site.yml: the
role's read-only interface probe ran without `check_mode: false`, so under --check it
was skipped and the assert fired with a misleading "set kube_vip_interface explicitly".

VERIFIED BY ME, not just reported:
  renders    dev K3S_API_HOST=192.168.1.49   prod=192.168.2.9  (each environment's VIP)
  cilium     template consumes k3s_apiserver_endpoint, not k3s_api_host
  cold join  still `--server https://{{ k3s_api_host }}:6443` — the founder
  order      site.yml: node-baseline -> k3s -> kube-vip -> cilium -> verify -> flux
  gate       full CI gate PASS
Agent-side, additionally: in-memory render of cilium-values.yaml.j2 per host showing
k8sServiceHost=the VIP with the join still on the founder; k3s_tls_sans renders
['.10','.9','127.0.0.1','.11','.12'] so NO cert reissue and NO k3s restart is involved
— .9 already answers under full CA verification; and --check --diff of kube-vip.yml
against prod reported 6 ok / 0 changed, i.e. the role edits change nothing about the
kube-vip prod already runs.

ha-control-plane-as-code 92 -> 112 assertions, turning two escalations into machine
checks: the Cilium template must use the endpoint var; the endpoint must derive from
kube_vip_address and k3s_api_host must not; per environment K3S_API_HOST must equal
that environment's VIP and be none of its node IPs (the [parity] contract, now
machine-checked rather than a comment); site.yml must order k3s -> kube-vip -> cilium.
Every new assertion mutation-tested and bites. One honest negative recorded: reverting
only the BASE default is not caught, because both overlays patch the key — the check
reads effective per-environment values, so the base default is unguarded.

APPLY ORDER, and this is the operator's step — nothing was applied:
  PROD FIRST, and it is the low-risk direction (prod is the broken one). Push and let
  Flux reconcile platform/cilium: the value change rolls the cilium-agent DaemonSet
  ONE NODE AT A TIME (maxUnavailable: 1 — the knob that bounded the 2026-09-01
  incident), it will not proceed past an unready pod, existing datapath survives an
  agent restart, and .9 is already live before the change lands.
  DEV SEPARATELY, operator present. Same mechanics but live dev agents dial
  192.168.1.50, and dev is mid-DNS-migration on dev-live. Do not fold it into that.
  ANSIBLE: nothing to run for either live cluster; the Ansible half matters only for a
  cold build. Remove the now-redundant standalone kube-vip.yml step from any runbook.

NEW NAMED GAP, unwritten: the L2 test cannot assert the running agents' live
KUBERNETES_SERVICE_HOST, which is the drift the L0 check cannot see. Recorded in that
test's gap list.

Refs R-PLAT-01, R-NET-01, R-ENV-02.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-06 22:17:30 -05:00
jakman85andClaude Opus 5 f178d85c71 fix(kyverno): unblock CDI imports by SATISFYING the label requirement, not exempting it
Prod's UniFi VM has no disk. `DataVolume unifi/unifi-rootdisk` stuck in
ImportScheduled for hours, because Kyverno denied CDI's own generated pod:

  validate.kyverno.svc-fail denied Pod/unifi/importer-prime-<uid>
    require-auth-mode-label     (R-IDP-02)
    require-egress-class-label  (R-NET-05)

Both policies match any Pod in an `app-namespace: "true"` namespace with no
exclusion, and CDI generates `importer-*` itself — it cannot carry repo labels.
A COLD-START defect, not a prod quirk: dev's identical import Succeeded 9 days
ago, before the policies enforced, so dev has never exercised it and would hit it
on any rebuild. Fourth bug of this shape in two days.

MECHANISM: a MUTATE ClusterPolicy that ADDS
`egress-class: internet` + `auth-mode: none` (add-if-absent anchor) to Pods on
CREATE in app namespaces WHEN the requesting subject is ServiceAccount
`cdi/cdi-sa`. Both validate policies are untouched — no exclude, no exception,
`background: true` preserved on them.

Why mutate beats exempting: the requirement stays literally true ("every pod in an
app namespace declares both labels") and the values are HONEST — the importer does
pull over the default WAN path, and `internet` is the least-reach class (it does
not grant the VPN path); a transient worker pod with no UI genuinely is
`auth-mode: none`.

THE MATCH IS IDENTITY, NOT A NAME. `subjects: ServiceAccount cdi/cdi-sa` is the
AdmissionReview userInfo, taken from the real prod denial
(`system:serviceaccount:cdi:cdi-sa`, name importer-prime-…, op CREATE). A name
prefix is attacker-choosable; a requesting ServiceAccount is not — and the test
proves it, with a fixture `importer-prime-forged-by-an-app` that copies both the
name prefix AND CDI's labels and is still DENIED. Covers every CDI worker kind
(importer/uploader/cloner) without enumerating names that change per release.
Confirmed on prod that cdi-sa is the only cdi-namespace SA that can create pods.

REJECTED, and the first one is a trap worth recording:
  * PolicyException — the CRD EXISTS (v2/v2beta1, from the pre-applied bundle) but
    all controllers run `--enablePolicyException=false` (verified live on prod:
    admission, background and reports controllers all carry the flag). An exception
    would apply cleanly through Flux and be SILENTLY IGNORED. Enabling the gate is a
    cluster-wide capability change for no gain over the mutate.
  * `exclude` scoped to cdi-sa — sound identity, worse consequences: any userInfo
    match forces `background: false`, which would drop background scanning and
    policy-report coverage of R-NET-05/R-IDP-02 for ALL EXISTING PODS to fix one pod
    class. It also leaves the importer with no labels at all — the unlabelled
    fall-through that require-egress-class-label's own header calls a fail-open leak.
  * name-prefix / CDI-label matching — attacker-choosable, and proven walkable.

TWO-WAY PROOF, non-vacuous by construction — same resource, different requester,
opposite outcomes (verified by me, not just reported):
  userinfo cdi-sa   -> mutate pass, require-egress-class pass, require-auth-mode pass
  userinfo ordinary -> mutate SKIP,  require-egress-class FAIL, require-auth-mode FAIL
Kyverno L0 went 17 -> 29 tests passed, 0 failed. The mutate policy is deliberately
kept OUT of the existing tests/kyverno-test.yaml: that suite declares no userinfo,
and without userinfo the offline engine does not discriminate on subjects — it
mutates everything, which would have silently turned its negative rows into passes.

platform-config/kustomization.yaml is the live apply path for these ClusterPolicies;
without it the policy never reaches a cluster.

RESIDUAL TRADE-OFF, reversible by deleting one file: anything created by cdi-sa in
an app namespace now has these labels supplied rather than being rejected for not
declaring them — including a pod that is not a CDI worker. That identity already
holds `pods create/delete` cluster-wide via ClusterRole `cdi`, so it is not a new
capability; what changes is that for that one identity the declaration is assumed
rather than demanded. restrict-host-namespaces, the namespace default-deny baseline
and the CiliumNetworkPolicies all still apply.

FIRST resource-mutating policy on this cluster: kyverno-resource-mutating-webhook-cfg
holds 0 webhooks today; with autoUpdateWebhooks=true Kyverno registers
mutate.kyverno.svc-fail for Pods within seconds of this applying. Prod's
forceFailurePolicyIgnore=false makes it failurePolicy: Fail — same posture as the
validating webhook already in that path.

CDI retries importer creation on a loop, so the stuck import should proceed once
this lands. The detached prime-be3f9d1c-… 40 GiB scratch PVC is separate operator
cleanup; untouched, as is the DataVolume.

VERIFIED: full CI gate PASS. Nothing applied to any cluster.

Refs R-IDP-02, R-NET-05, R-PLAT-01.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-06 22:17:30 -05:00
jakman85andClaude Opus 5 f21e89d9d2 fix(gitops): CDI cold-start deadlock repaired IN THE GRAPH, not by a pre-apply
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
`platform-config` applied the CDI CR *and* cdi-storageprofiles.yaml in ONE pass.
StorageProfile's CRD does not exist until cdi-operator reconciles that CR, and
Flux server-side dry-runs every object before applying any — so the unknown kind
aborted the tier, the CR was never applied, the CRD was never created, and the
tier could NEVER converge. Observed on prod: cdi-operator Available 12h, no CDI
CR, no storageprofiles CRD, every downstream tier blocked.

The profiles now live in their own tier ./platform-config-cdi with
dependsOn: platform-config and wait: true. Because dependsOn is REVISION-gated
and platform-config runs wait: true, the CDI CR must be applied AND healthy —
exactly the event that creates the CRD — before the profiles are applied. Cold
clusters order themselves. `standup.sh crds` is retained as a fallback for
clusters already stuck, and because it is idempotent.

THE GENERALISABLE RULE (recorded in clusters/README.md): a CR whose CRD is
created by a CONTROLLER AT RUNTIME is second-order and needs its own pass.
cdi-operator.yaml registers exactly ONE CRD (cdis.cdi.kubevirt.io); storageprofiles,
datavolumes, datasources and dataimportcrons appear only after it reconciles the
CDI CR. An audit found StorageProfile is the only second-order kind in
platform-config — KubeVirt, CDI, KeycloakRealmImport, CNPG Cluster,
VolumeSnapshotClass, ClusterIssuer, PrometheusRule, ServiceMonitor and the Cilium
L2 CRs all have their CRD in git or in a chart.

Moving the CDI CR up into `platform` was rejected: that would put CDI's health on
the tier the CNI gates and everything waits on — the mistake the oauth2-proxy and
headlamp splits exist to undo.

apps-vms gains dependsOn: platform-config-cdi. LOAD-BEARING, not cosmetic:
apps/vms/unifi/20-datavolume.yaml provisions on storageClassName: longhorn-vm and
its comment claimed platform-config guaranteed the profiles — which this move
would have silently falsified.

THE PRUNE TRANSIENT, MEASURED RATHER THAN ASSUMED. platform-config has
prune: true, so it deletes both profiles once, CDI recreates them with empty spec,
and the new tier re-applies the overrides ~30-60s later. Compared our override
against CDI's unaided inference for the same Longhorn provisioner on dev:
  longhorn-vm (we override)     spec RWX/Block  -> status ReadWriteMany/Block
  longhorn-static (no override) spec unset      -> status ReadWriteOnce/Block
Both infer cloneStrategy: snapshot. So cloneStrategy is NOT at risk; accessModes
IS — unaided CDI gives RWO. A DataVolume provisioned on longhorn-vm inside the
window would be stamped RWO/Block and be PERMANENTLY non-live-migratable. Flux can
no longer cause it (apps-vms gates on the new tier); a human must not do it by hand.
Corroborating evidence that the recreate happens at all: dev has 5 StorageClasses
and 5 StorageProfiles, three with fluxOwner=NONE, spec unset and status populated
— CDI creating and inferring them with no Flux involvement.

MERGE PRECONDITION: `flux get kustomization platform-config` must be Ready on the
target first. If it is not, the profiles are pruned and sit at CDI defaults until
it recovers — and prod was in that state hours earlier. Fallback is the two-step
the Cilium move used (merge with prune: false, verify the ownership label moved,
restore prune: true).

substituteFrom: cluster-vars is declared even though the manifest has no ${TOKEN},
deliberately: conformance/tests/l0-static/flux-postbuild-substitution only audits
paths whose Kustomization declares it, so omitting it would move the file OUT of
the audited set — a blind spot for a future literal-token ship. Gate confirms it
stayed in (path=./platform-config-cdi files=2 defined=0 hazards=0). No SOPS
decryption stanza: the moved file is two plain CRs, so no overlay change was needed.

VERIFIED INDEPENDENTLY OF THE AGENT: platform-config renders 0 StorageProfile
while keeping the CDI and KubeVirt CRs; platform-config-cdi renders exactly the
two profiles; the graph is byte-identical across dev and prod with apps-vms
carrying both edges and no cycle. Full CI gate PASS.

KNOWN FOLLOW-UP: bootstrap/standup.sh TIER_ORDER does not yet include
platform-config-cdi, so `resume` will not wait on it. Another agent holds that
file; the PM will add it.

Refs R-GITOPS-01, R-ENV-02.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-06 22:14:19 -05:00
jakman85andClaude Opus 5 2d7d302c7e feat(prod): cut prod's GitOps source over to prod's OWN in-cluster Gitea
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
Prod reconciled from http://192.168.2.132:3000 — a podman container on the
operator's WORKSTATION, holding a DHCP lease on WIFI. Production's source of truth
was a laptop: a lease change or a closed lid stalls all reconciliation. Dev was
pointed there too (deliberately, to break a strand risk during its DNS migration),
so BOTH clusters depended on it simultaneously.

Prod now reconciles from https://gitea.prod.home.keithjackson.dev/... — the Gitea
prod itself runs, which apps-wave-0 has been serving for 28h.

THE SELF-REFERENCE IS THE WHOLE DIFFICULTY, and it is why this is a commit rather
than a kubectl patch. The root Kustomization renders ./clusters/prod, which CONTAINS
the patch that sets GitRepository.spec.url — so a live patch is reverted on the next
reconcile unless the AUTHORED value moves with it. Dev hit the identical trap on
2026-09-05. The ordered cutover:
  1. push the repo to prod's Gitea (18 branches, refs verified identical, 499 commits
     of history — a real move, not a shallow copy);
  2. fast-forward prod Gitea's `main` to c8c1d29, the EXACT commit prod was already
     running, so swapping the source changed nothing else. Its main had been at
     4809199, and repointing to that would have ROLLED PROD BACKWARDS, undoing the
     per-environment BASE_DOMAIN work and sending prod at the apex — the collision
     with live dev that two days of work exists to prevent;
  3. live-patch spec.url to open a window (source changed, content identical:
     Ready=True at the same revision on the first poll);
  4. land THIS commit through that window so the authored value agrees.

VERIFIED FROM INSIDE PROD, not from the workstation: a throwaway pod resolved
gitea.prod.home.keithjackson.dev through in-cluster CoreDNS to the Gateway VIP
(192.168.2.20) and cloned main over the public Let's Encrypt cert with NO CA
supplied. That exercises split-horizon DNS, the Gateway, Gitea and TLS on the pod
network — the path Flux actually uses. A workstation clone proves none of it.

The bootstrap Gitea is retained as a recovery path, not deleted
(bootstrap/gitea/bootstrap-gitea.sh). Dev still reconciles from it on branch
`dev-live` and returns to its own Gitea at step D4 of the §10a migration runbook.

Refs R-GITOPS-01, R-ENV-02.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-06 22:04:07 -05:00
jakman85andClaude Opus 5 c8c1d2935d feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).

THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.

Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.

PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.

BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.

TWO THAT CANNOT BE TOKENIZED, flagged not forced:
  - headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
    over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
    so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
    anyway — client secrets are realm-scoped.
  - kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
    reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
    overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
    prod.home.

EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.

I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.

DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.

L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.

PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.

Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-04 22:43:54 -05:00
jakman85andClaude Opus 5 9b8979a427 feat(prod): temp Gitea source + wireless disabled + flux bootstrap path
Autonomous prod bring-up, operator away. Three changes that together clear the last
blockers before `flux bootstrap`.

=== 1. TEMPORARY BOOTSTRAP GITEA — the last hard blocker, cleared ===
clusters/prod's source URL could not work: Gitea runs ON the dev cluster (the one slated for
decommission) and split-horizon DNS resolves gitea.home.keithjackson.dev to PROD's OWN
gateway. Stood up the existing bootstrap/gitea/bootstrap-gitea.sh on the operator's
workstation with HOST_NAT_IP=192.168.2.132; all three prod nodes verified reachable
(Gitea 1.22.6 via the API), no firewall change needed — prod is on the same /24, no NAT.

clusters/prod now points at http://192.168.2.132:3000/... under a boxed TEMPORARY banner
with a 5-step cutover checklist ending 'delete this whole block — dev's stanza is what done
looks like'. Repo verified PUBLIC with anonymous read two independent ways (API
"private":false; git ls-remote unauthenticated with GIT_TERMINAL_PROMPT=0), so NO secretRef
is needed — documented so a future auth failure is not misread as a network fault.

bootstrap-gitea.sh's header corrected: it documented HOST_NAT_IP=10.73.0.1, the RETIRED
libvirt NAT, and forbade agent use; the operator has now directed this for prod.

=== 2. WIRELESS DISABLED ON PROD (operator-reported) ===
k3s-prod-3 was on ethernet AND wireless, and the wifi DHCP client was overwriting
/etc/resolv.conf. Root cause of the silence: node_baseline's DNS `contended` detection
checked resolvconf and NetworkManager but NOT dhcpcd, so the role wrote resolvers, reported
success, and dhcpcd reverted them on lease renewal — the exact 'green run that fixed
nothing' the contended branch exists to prevent.

Now IaC, not a one-off: survey wireless hardware, **REFUSE to disable it if the path to the
node runs over the wireless interface** (the guard that meant an agent dying mid-apply cost
nothing), blacklist the driver so it never reloads, stop dhcpcd only where it holds a
wireless interface and nothing else, take links down and unload modules with NO reboot, then
report the resulting state. dhcpcd is now in the contended set without weakening fail-closed.

Applied and verified: node 3's wlp1s0 is GONE from /sys/class/net, dhcpcd inactive and its
unit not-found, ONE default route per node, all three on 8.8.8.8/1.1.1.1 resolving public
names. Safe and idempotent on nodes 1 and 2, whose wifi hardware was already down.

=== 3. flux_bootstrap PATH PARAMETERISED ===
bootstrap/flux/bootstrap.sh already read FLUX_ENV_PATH but the role never exported it, so it
ALWAYS bootstrapped clusters/dev — prod was impossible. Two behaviour-bearing lines: role
default clusters/dev + `environment: FLUX_ENV_PATH`, overridden to clusters/prod in
group_vars/k3s_prod. Proven per host with a read-only Ansible probe: prod hosts resolve
clusters/prod, localhost/hypervisor resolves clusters/dev. Only two definitions repo-wide.

DEV PROVEN BYTE-IDENTICAL for the source change: all 12 substituting Flux paths rendered for
both trees, 27/27 SHA-256 artefacts identical, with TWO POSITIVE CONTROLS (an overlay-level
canary and a deep-path content canary) confirming the harness detects change — and both
canaries proven restored. Honest scope note recorded: kustomize strips comments, so the
harness proves the built stream, not source comments.

L0 substitution check: 106 assertions, 0 hazards. ci/l0.sh and ci/lint.sh exit 0.

=== STILL BLOCKING A FULL PROD RECONCILE (operator decisions) ===
- S3_BACKUP_ENDPOINT remains TBD-PROD-*: prod CANNOT reach dev's versitygw at
  192.168.1.9:9000 (verified UNREACHABLE from all three — the pfSense boundary), and a prod
  backup target is a deferred operator decision. Deliberately NOT invented: a DR target on a
  workstation container would create false confidence. Velero/CNPG backups will fail loudly.
- Prod's public resolvers mean the API server's oidc-issuer-url resolves PUBLICLY, so kubectl
  SSO will not work on prod until split-horizon is solved for it (§4).
- The temp Gitea is a single container on the operator's workstation — a single point of
  failure for prod reconciliation, acceptable for bring-up only.

Serves: R-GITOPS-01, R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-04 18:53:17 -05:00
jakman85andClaude Opus 5 05baae90ec feat(prod): kernel fix + disk layout — prod nodes ready for k3s
Prod is being stood up now, without waiting for node 2's RAM upgrade. Three coupled
changes, all applied and verified on all three nodes.

=== 1. KERNEL >= 6.13 — now a PREREQUISITE, not a nice-to-have ===
A k3s SERVER that intermittently boots with 2.6 GiB is not a degraded node; it fails under
etcd plus its workload share, unpredictably, on any reboot. Fixed before k3s goes on.

roles/node_baseline gains a `kernel`-tagged block: asserts the suite matches the node's
Debian release, writes a deb822 trixie-backports source, installs, reads back the exact
version, writes a provenance/rollback note to the node. Scoped by
node_baseline_backports_kernel (default FALSE, true only in group_vars/k3s_prod) — chosen
over a hostname test so a fourth prod node inherits it from group membership. Verified:
--check against dev skips all 8 tasks on all 3 hosts. Two deliberate calls: `state: latest`
not `present` (linux-image-amd64 was already installed from main, so `present` would have
silently fixed nothing), and NO preferences.d pin (backports' NotAutomatic +
ButAutomaticUpgrades already gives 100/500; an explicit Pin-Priority: 100 would switch OFF
automatic security upgrades for the very kernel being installed).

Installed 7.1.8+deb13-amd64 on all three; 6.12.107 kept bootable.

THE DECISIVE PROOF: node 2 booted at MEMATTR=0xbeeb0018 — the EXACT address that gave it
2.6 GiB twice — and now reserves 634,904K instead of 13,958,980K. MemTotal 15.4 GiB. Same
address, handled correctly; not a lucky draw.

§0b's claim that a fixed kernel prints 'FW_BUG Corrupted...' is CONDITIONAL and was
corrected: >=6.13 has three rejection gates and only the third prints that line; all three
nodes hit gate 1 (version > 2). Proof instead is boot-independent — decompressing
/boot/vmlinuz on all three yields the FW_BUG and descriptor-size strings verbatim; neither
exists in 6.12.

TWO MORE BUGS IN MY OWN HEALTH CHECK, found because the fix couldn't otherwise be shown:
  - MEMATTR_VERSION_RE matched (\d+) but the kernel prints that field with %d — SIGNED. Node
    2's real value was -1569980416, which did not match, so state fell through to `clean` and
    the node reported PASS "the kernel accepted it without complaint". A FALSE PASS on the
    exact bug the check exists to find. Now (-?\d+).
  - bad_version was WARN unconditionally, so a FIXED node would have stayed WARN forever.
    Now keys on memattr_kernel_fixed: same line, WARN below 6.13, PASS at/above.

=== 2. DISK LAYOUT — Longhorn claimed, per-node paths DISCOVERED not hardcoded ===
Operator decision: reuse the ~424 GB partition; accept the shared-device I/O risk (dev's
etcd/Longhorn contention was on a spinning HDD, prod is SSD/NVMe).

Device paths differ per node (sda5 / nvme0n1p5 / sda5), so the role DISCOVERS the partition
via `findmnt --mountpoint` on a declared claim mountpoint rather than trusting inventory.
Per-host host_vars would have frozen today's hardware into inventory, where it goes stale on
the first re-image and the stale value's failure mode is mkfs on the WRONG partition.
Eleven fail-closed asserts before any write: real block device, not backing / /boot /usr
/var /home /tmp or active swap (built from the node's OWN gathered facts, never a hardcoded
list), already ext4, no foreign label, >= 100 GiB, and empty. Relabel, never reformat.
/srv verified empty first on every node (ls -A -> lost+found only; du 20K; the "5.1% used"
was ext4's default 5% root reserve).

=== 3. k3s DATA-DIR MOVED — /var was 11.8 GiB and would have killed etcd ===
Measured on the fully-deployed dev cluster: containerd 21-25 GB/node, /var total 58-70 GB.
Prod's /var is 11.8 GiB and held BOTH etcd and the image store, so prod would have filled
/var during its first image pulls — and a full /var means etcd cannot write its WAL. Cluster
death, not degradation, arriving during bring-up. Invisible when the disk decision was made,
because "etcd on root" sounded like the 30 GiB / rather than a separate 12 GiB /var.

Now: p5 -> /data (LABEL=k3s-data), with /data/longhorn and /data/k3s. Neutral names chosen
deliberately — the partition now has two tenants, and reusing /srv would make "claimed" and
"unclaimed" indistinguishable to the idempotency pivot, while a bare `data` label would make
a stranger's `data` volume look already-claimed and weaken an assert.

The transition off the existing LABEL=longhorn mount is DECLARED, not inferred
(node_baseline_data_retired_fs_label/_retired_mount, defaults ""), because changing the label
makes a claimed node answer "no" to the pivot and walk into the greenfield path. Self-retiring
once blkid -L k3s-data succeeds. THREE independent gates keep live dev out: retired vars
default ""; gated on longhorn_source == claim-mountpoint (dev is `device`); and the emptiness
assert fails closed on dev's 18 volumes even if both were defeated.

k3s_data_dir is one variable read by two roles (k3s_server --data-dir; node_baseline derives
the etcd mount). Role default is k3s's own default, so dev is unchanged and keeps its
dedicated etcd device. node_baseline_etcd_source becomes `shared-data-partition` and is
LOAD-BEARING: the role asserts per run that the filesystem backing the etcd dir really IS the
data partition — had the mount silently failed, etcd plus 21-25 GB of images would land back
on the OS filesystem invisibly until the disk filled.

LONGHORN_DATA_PATH is a new cluster-vars knob (R-ENV-02): base and DEV /var/lib/longhorn,
prod /data/longhorn.

DEV PROVEN UNCHANGED, which is the property that mattered — the live cluster has 18 Longhorn
volumes on that path. All 12 substituting Flux paths rendered through
`kubectl kustomize | flux envsubst --strict` for both trees: 12/12 per-file SHA-256 identical
and an identical aggregate SHA over every rendered dev byte. With a POSITIVE CONTROL showing
the same harness detects prod changing on exactly one line — an all-identical result is
worthless if the harness cannot see change. Live check after: dev's default-data-path setting
still /var/lib/longhorn, 18 volumes attached, 0 Kustomizations unready.

Idempotent: run 2 is changed=0 on all three nodes. Also fixed a real tag bug — include_tasks
does NOT propagate its own tags to included tasks (unlike import_tasks), so `--tags disks`
selected the include and ran nothing inside it while reporting success.

OPEN DEFECTS, tracked not forgotten: node 2's warm-reboot hang (one clean reboot since, NOT a
fix); and node 3 runs dhcpcd for its wifi interface, which rewrites /etc/resolv.conf on lease
renewal — the role's `contended` detection checks resolvconf and NetworkManager only, so node
3's decided resolvers will silently revert.

Gate: MERGE GATE: PASS. L0 substitution check 106 assertions.

Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-STATE-01, R-STATE-02, R-PLAT-01. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-04 00:39:10 -05:00
jakman85andClaude Opus 5 8e913fdd7b fix(node_health): "Failed to map" does NOT mean memory lost — plus survey results
THE MODEL WAS WRONG AND MY CHECK WITH IT. A live counter-example: k3s-prod-2 after its power
cycle shows MEMATTR=0xbeeb0018 and 'Failed to map ... @ 0xbeeb0018' — the SAME address and SAME
line as both of its 2.6 GiB boots — with MemTotal 15857732 kB, i.e. 15.1 GiB of 16 GB. FULL
MEMORY. The check reported FAIL 'THIS NODE HAS LOST MEMORY TO THE BUG' on a healthy node while
memory.physical_total correctly PASSed at 4.2%. The two checks contradicted each other.

REFINED MECHANISM — two kernel functions were conflated:
  efi_memattr_init() does the memblock_reserve(). Catastrophic loss needs tbl_size NEGATIVE
    (signed int -> sign-extended -> clamped to PHYS_ADDR_MAX). A garbage-but-POSITIVE tbl_size
    reserves a modest wrong amount and costs nothing noticeable.
  efi_memattr_apply_permissions() does the later memremap(), and THAT is what prints
    'Failed to map'. It fails for ANY sufficiently wrong size, negative or merely large-positive.
So the line proves the table was corrupt AND TRUSTED, not that memory was lost. The cost is set
by the garbage VALUE, which is why one address gave 2.6 GiB twice and 15.1 GiB once.

THIS ALSO REFUTES THE ADDRESS->OUTCOME HYPOTHESIS OUTRIGHT — not untested, disproven.

STRUCTURAL FIX so the contradiction cannot recur: the shortfall is computed ONCE into a shared
fact (node_health_mem_shortfall) with worst-wins sub-verdicts; memory.physical_total IS that
verdict and firmware.efi_memattr READS it, measuring nothing itself. FAIL is gated on the shared
verdict. 'Failed to map' + normal memory is now WARN ('the bug fired and this boot survived it').
No honesty guard was weakened; one was strengthened (the measurement clause falls back to the DIMM
shortfall when zoneinfo is unreadable, and only claims 'could NOT be measured' when neither was
read). All five paths incl. the two unproducible-live SKIPs exercised against synthetic facts.
Verdicts: prod-1 WARN, prod-2 WARN (was FAIL), prod-3 PASS, dev unaffected.

SECOND FAULT, arguably worse for prod: NODE 2 HANGS ON A WARM REBOOT, STAYING POWERED. Its
journal ends cleanly mid-shutdown ('eno1: NIC Link is Down') then stops; the box sat powered but
off-LAN for ~38 min until the operator held the power button. A node that cannot survive an
unattended warm reboot cannot be rolled in an HA cluster, and node_health cannot see a machine
that is off the network. Leading candidate: node 2 is the ONLY node with 'Recover OS after Boot
Failure = Enable' — promoted from 'coincidence' in the BIOS diff to a live availability defect.
The reboot survey is NOT viable unattended on node 2 until this is understood.

SURVEY RESULTS (40 boots): prod-1 0/20, prod-3 0/20, prod-2 0 ROUNDS (hung on its first reboot),
pooled 0/40 -> p(bad) 95% CI 0.000-0.088. NOT elimination: 0/20 still permits 17% per node, and
node 2's prior 2-of-3 is unchanged. What it did establish: addresses are node-specific and highly
concentrated (node 1 used 3 distinct addresses in 20 boots, 15 at one; no address ever shared
between nodes); MEMATTR -> parse was a PERFECT function across all 7 repeated addresses over 37
boots; and the corrupting bytes are stable PER MACHINE (node 1 always 1030, node 3 always
1463964749), so the 'shared OS boot chain' inference needs refining.

Adds playbooks/memattr-survey.yml + roles/memattr_survey — reboots opt-in, resumable, artifacts
gitignored. Documents an Ansible defect worth knowing repo-wide: a looped include_tasks under the
default 'linear' strategy is flattened per host, so hosts run SERIALLY; the first attempt rebooted
node 1 twenty times while the others idled. Fixed with strategy: free + a forks preflight assert.

Serves: R-ENV-01, R-ENV-03, R-OBS-04, R-TEST-02. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-03 23:50:59 -05:00
jakman85andClaude Opus 5 b19c5db9ae feat(prod): node 3 onboarded — and it is NOT immune; BIOS-setting hypothesis disproven
k3s-prod-3 online at the planned 192.168.2.12 (serial 8CC015307N, same chassis/BIOS/CPU),
enabled in inventory, vaulted become confirmed, package floor installed. Health: 0 FAIL /
4 WARN / 14 PASS / 1 SKIP — the only node with zero FAILs. WD Blue WDS500G2B0A SATA SSD at
100% life, 3213 h, 0 grown bad blocks, 0 erase fails. Matched Crucial 2x16 GB pair — the only
node with symmetric memory.

*** NODE 3 IS NOT IMMUNE TO THE §0b BUG. *** Its firmware DOES publish a MAT
(MEMATTR=0xbeece018); the ABSENCE of a memattr diagnostic line means the kernel read a VALID
table, which is the opposite of immunity. There are THREE outcomes, not two: bail-out on a
garbage version >2, a valid parse with a sane size, and the catastrophic path. Seven boots across
three machines show the MAT address MOVES BOOT-TO-BOOT on every node with settings constant, and
each node has been seen on more than one path. Blast radius widens from two nodes to ALL THREE;
all three need the >= 6.13 kernel.

THE CORRUPTING BYTES ARE THE OS BOOT CHAIN, not firmware and not DRAM: garbage version
1463964749 = 0x5742544D appeared on node 2 AND node 3 — two different machines, identical bytes.
The overwriter is the shared shim/GRUB/EFI-stub/initrd placement writing into the page the
firmware pointed at. So initrd size, module set and boot device are part of the causal chain.

BIOS-SETTING HYPOTHESIS: DISPROVEN, and the three-way diff cannot support it even in principle.
All 240 attributes read on all three; firmware identical fleet-wide. The discriminating set (both
working nodes agree, broken differs) is seven attributes, of which:
  - Fast Boot is the ONLY live candidate, and only as a probability-shifter — HP documents it
    'skips some preboot initialization steps', which changes the heap top and hence where the MAT
    lands. Different mechanism from the previously-killed MRC-caching theory; that stays dead.
  - Legacy Boot Options is AFFIRMATIVELY INERT: 'Configure Legacy Support and Secure Boot' =
    'Legacy Support Disable and Secure Boot Enable' on ALL THREE, Option ROM Launch Policy =
    'All UEFI' on all three. No CSM is built on node 2 either. A naive diff gets this wrong.
  - the rest are coincidence. After Power Loss is ELIMINATED by node 3, which works and shares
    node 2's Power Off — though it is now a §1 checklist violation on TWO nodes.
FATAL CONFOUND: the 1=3!=2 partition is perfectly collinear with node 2 being the only
single-DIMM AND the only NVMe-booting node. Adding node 3 did not break the confound, it DOUBLED
it. No setting can be the cause anyway: node 2 booted good once and bad twice with settings held
constant.

EXPERIMENT DESIGN RECORDED because the obvious approach is statistically worthless: p(bad) is
known only to n=3, 95% CI ~0.09-0.99, so even 8 clean boots prove nothing. Elimination needs ~20
boots/arm; an odds shift needs ~50. Zero-touch first step: 20 reboots of node 2 unchanged, logging
MEMATTR address vs outcome — both failures were at exactly 0xbeeb0018 and all five successes
elsewhere, so if outcome maps onto a narrow address range the mechanism is settled without
touching anything.

ROLE BUG FIXED: node 3's WD drive reports a SMART threshold for only 1 of 25 attributes, and
smartctl OMITS the key rather than emitting 0 — summary.txt.j2 dereferenced a.thresh bare in five
places and the report task died, so no artifact was written. Added a thr() macro rendering 'n/a'
(deliberately not 0, which would read as a real ATA cannot-trip threshold). Would have hit any
drive of this class.

Also: three nodes, three different disk models, only one NVMe, and every node has a single disk
with no unpartitioned spare — so §2's identical-layout requirement is unmet on all three.

Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-03 22:32:25 -05:00
jakman85 6e4d311545 docs(prod): will 6.12.y get the memattr fix? Researched — not on its own
Operator asked whether waiting is viable or whether a backports kernel is permanent. Answer: it
will NOT arrive on its own, but it CAN be made to, and the ask is unusually strong.

VERIFIED:
- NOT in linux-6.12.y. Newest commit touching memattr.c on that branch is 2024-07-12. In mainline
  since v6.13 (2025-01-19) — ~19 months, ~97 point releases.
- NO stable tag and NO Fixes: tag on either commit. Structural, not an oversight: the unvalidated
  arithmetic dates to the file's creation in 2016, so it is not a regression and there is no bad
  commit to point Fixes: at. BOTH automatic feeds — Fixes:-scanning and AUTOSEL — had nothing to
  grab. AUTOSEL never surfaced it for any branch.
- NOBODY has requested it: nothing on [email protected], nothing in all 8,027 open+archived
  Debian bugs against src:linux.
- 6.12 LTS EOL is December 2028 — outlives Debian 13's support window, so not time-barred.
- Debian REBASES: trixie carries 6.12.94-1 in trixie-security ('New upstream stable update'), so an
  upstream pickup would reach us via a normal apt upgrade.

THE DECISIVE CARD: the cherry-pick is BYTE-CLEAN. Debian trixie 6.12.94-1's memattr.c hashes to
164203429fa7a..., identical to the upstream patch's pre-image blob — the file has not diverged
between v6.12 and the commit's parent. cherry-pick applies with zero conflicts and zero adjustment,
which removes the usual reason such requests get declined. The commit also meets every
stable-kernel-rules criterion: mainline, 17 insertions, 'hardware quirk', 'a real bug that bothers
people'.

PLAN: backports now as explicitly provisional; email [email protected] (rules Option 2) CC
linux-efi and Ard Biesheuvel who authored AND committed it as EFI maintainer, leading with the
clean-apply proof and the three-boot reproduction with byte-identical e820; Debian BTS as secondary
under handbook §9.1.7; make the deviation permanent only if there is no movement in 4-6 weeks.

Only 8fbe4c49c0cc is needed by hand — 5d0faa8e8369 loosens a heuristic that was rejecting
LEGITIMATE tables and is not required for this symptom.

Serves: R-ENV-01, R-ENV-03. Task: T-7.1.
2026-09-01 23:04:56 -05:00
gitea-admin 4809199da6 Merge pull request 'Node 2 memory: solved — Linux 6.12 EFI memattr bug, not hardware' (#15) from docs/node2-memory-refine into main
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
2026-09-02 03:45:30 +00:00
jakman85 0a733294ff docs(prod): node 1's DIMMs are not a matched pair
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
Surfaced by the new node_health DIMM reporting, unrelated to the memattr bug and NOT a fault:
DIMM1 is a Micron 8ATF2G64HZ-3G2E1 (DDR4-3200), DIMM3 a Micron 16ATF2G64HZ-2G6E1 (DDR4-2667).
Both 16 GB so capacity is right, but different rank organisation and rated speed mean the pair
runs at the slower 2667.

Recorded in §0 where the symmetry decision gets made, because §2 requires the three nodes to end
up with identical layouts and the previous '2x16 GB Micron' note hid the mismatch.

Serves: R-ENV-03. Task: T-7.1.
2026-09-01 22:31:26 -05:00
jakman85andClaude Opus 5 cf33338448 feat(node_health): detect the EFI memattr bug on every run, two independent ways
The fault is BOOT-TO-BOOT NON-DETERMINISTIC — the same machine booted correctly two boots
earlier — so it has to be detected on every health run, not diagnosed once. And node 1 is
exposed to the identical lottery on every reboot while currently looking healthy.

firmware.efi_memattr — reads the kernel ring buffer and distinguishes six states:
  Failed to map EFI Memory Attributes table @ 0x..   -> FAIL (memory lost NOW)
  Unexpected EFI Memory Attributes table version <N> -> WARN (corrupt table still published;
                                                        intact only by luck)
  [Firmware Bug]: Corrupted EFI ... detected!        -> PASS (the >=6.13 fix working)
  no memattr line, log reaches early boot            -> PASS (no table / accepted / not UEFI)
  no memattr line, log does NOT reach early boot     -> SKIP
  dmesg unreadable                                   -> SKIP

memory.physical_total — deliberately a SEPARATE check, not folded in: it measures the SYMPTOM
with no kernel log at all, so an unreadable-dmesg node is still not unmeasured. Two sub-verdicts,
worst wins: /proc/zoneinfo present-vs-managed (world-readable, works on metal and VMs — sum of
managed IS MemTotal) and MemTotal vs the DIMM total from dmidecode. Thresholds 10% WARN / 25%
FAIL, set from measured baselines: 2.0% prod-1, 2.1% workstation, 2.2% dev VM, vs 83.4% prod-2.

THREE GUARDS AGAINST A FALSE PASS, because "no memattr line" must never silently mean "fine":
1. dmesg failure yields available:false with the exact reason (rc, uid, kernel.dmesg_restrict) -> SKIP
2. the ring buffer WRAPS and every memattr line prints in the first fraction of a second, so the
   collector records a boot_covered anchor (Linux version / BIOS-provided physical RAM map); no
   memattr line WITHOUT that anchor is a SKIP, not a PASS
3. memory.physical_total needs no kernel log at all
Verified by forcing the unprivileged path: SKIP with the EPERM reason while zoneinfo still PASSed.

VERDICTS, confirmed live:
  k3s-prod-1  WARN "EXPOSED, not immune ... survived only because its garbage version field read
                    1030, which is > 2 ... one unlucky boot from the failure state" + PASS 2.0%
  k3s-prod-2  FAIL "THIS NODE HAS LOST MEMORY TO THE BUG ... at 0xbeeb0018" + FAIL 83.4%/83.6%
  k3s-dev-*   PASS both, NO false finding: "no UEFI config-table line ... did not boot via UEFI.
                    Log reaches back to early boot, so that absence is measured, not assumed."
Both failure texts carry the remediation verbatim, including that there is NO kernel cmdline
workaround because efi_memattr_init() runs before any efi= bail-out.

Root-gating follows the role's existing pattern: --privileged re-runs THE SAME collector under
become, only when the unprivileged pass actually failed to read something. Reimplementing the log
grep and SMBIOS parse in Jinja would have been a second source of truth for the same fact.

LATENT BUG FIXED in passing: the collector's run() used bare shutil.which(), so dmidecode (in
/usr/sbin) was invisible under the unprivileged k3s PATH and produced a FALSE "not installed"
skip reason. Now shares the sbin-aware which() collect_tools() already had.

INCIDENTAL FINDING recorded in §0b: node 1's DIMMs are NOT a matched pair — DIMM1 Micron
8ATF2G64HZ-3G2E1 (DDR4-3200), DIMM3 Micron 16ATF2G64HZ-2G6E1 (DDR4-2667). Both 16 GB so capacity
is right, but different rank organisation and rated speed, so the pair runs at 2667. Not a fault;
recorded because §2 requires the three nodes to end up symmetric and "2×16 GB Micron" hid it.

Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-01 22:30:45 -05:00
jakman85 47f02c6e06 docs(prod): second independent confirmation + no cmdline workaround exists
A second research pass reached the same root cause by a different route and pinned the
arithmetic. Recording it because convergence from two independent lines is what turns this from
a good hypothesis into a settled diagnosis.

MECHANISM TO THE BYTE:
- the garbage 'version' node 2 read was 1463964749 = 0x5742544D = ASCII 'MTBW'. The kernel is
  reading unrelated data. HP publishes the MEMATTR pointer into memory /sys/firmware/memmap calls
  plain System RAM, so the pre-Linux boot chain overwrites it.
- tbl_size is a SIGNED int; num_entries * desc_size wraps mod 2^32 and goes NEGATIVE when bit 31
  lands. memblock_reserve sign-extends it and memblock_cap_size clamps to PHYS_ADDR_MAX - base,
  so EVERYTHING above 0xbeeb0018 is reserved.
- memblock_free_all then hands the allocator nothing above that line while present_pages (from
  memblock.memory) is untouched -> present 3340288 / managed 0.
- 'Failed to map EFI Memory Attributes table' is emitted LATER by efi_memattr_apply_permissions();
  it is positive PROOF the reservation already happened, a consequence rather than the cause.

CLEANEST A/B TELL, one dmesg line: NODE_DATA(0) on prod-2 ends at 0xbeeb0000 — pgdat forced
immediately BELOW the reservation — while on prod-1 it ends at 0x82f800000, the top of RAM.
Identical 0x2a980 allocation, opposite placement.

THERE IS NO KERNEL CMDLINE WORKAROUND. efi_memattr_init() is called from
efi_config_parse_tables(), and x86's efi_init() calls efi_config_init() BEFORE the
efi_runtime_disabled() bail-out — so noefi / efi=noruntime / efi=novamap do NOT prevent the
reservation. A fixed kernel is the only real fix.

EFI_MEMORY_SP / soft-reserved was the strongest competing theory and is EXCLUDED BY MEASUREMENT:
/sys/firmware/memmap/10/type reads 'System RAM'. Soft-reserved would print as 'Soft Reserved' and
is never memblock_add()ed, which would have left present at ~0 rather than 3340288.

Also records that the hardening landed exactly one release late for Debian 13 (bug present since
2016; 8fbe4c49c0cc first in v6.13; follow-up 5d0faa8e8369) and that a Debian BTS report against
src:linux is warranted — 6.12 is Debian 13's kernel for the release lifetime.

Serves: R-ENV-01, R-ENV-03. Task: T-7.1.
2026-09-01 22:16:22 -05:00
jakman85 49d82491dd docs(prod): node 2 memory SOLVED — Linux 6.12 EFI memattr bug, NOT a hardware fault
I framed this as an OPEN HARDWARE FAULT and blamed, in order: the DIMM, BIOS memory remapping,
and stale Fast Boot memory-training data. ALL THREE WERE WRONG. The hardware is fine.

ROOT CAUSE: HP's R21 firmware publishes a CORRUPT EFI Memory Attributes Table, and Linux 6.12
fails to validate it before reserving memory from its contents. drivers/firmware/efi/memattr.c
never validates num_entries or desc_size:

    if (tbl->version > 2) { pr_warn(...); goto unmap; }
    tbl_size = sizeof(*tbl) + tbl->num_entries * tbl->desc_size;
    memblock_reserve(efi_mem_attr_table, tbl_size);

Garbage version > 2 -> kernel bails, memory intact. Garbage version <= 2 -> kernel multiplies two
garbage fields and reserves GIGABYTES from the table address to the top of RAM.

THE EVIDENCE THAT SETTLES IT: the same machine — same DIMM, same BIOS, same kernel, same cmdline —
booted with the FULL 15.9 GiB two boots earlier, with a byte-identical BIOS-e820 map. Boot -2:
15,917,744K available, 615,208K reserved, 'Unexpected EFI Memory Attributes table version
1463964749'. Boots -1/0: 2.6 GiB, 13.9 GiB reserved, 'Failed to map ... @ 0xbeeb0018'. No hardware
fault is intermittent in that shape or fixed by a reboot. MemTotal = DMA(3840) + DMA32(684481)
pages = 2,753,284 kB exactly, Normal contributing zero.

BOTH PROD NODES ARE AFFECTED. k3s-prod-1 prints 'Unexpected EFI Memory Attributes table version
1030' — also garbage, but >2, so the kernel bails and it keeps its 32 GiB BY LUCK, NOT IMMUNITY.
It is one unlucky boot from the identical fault. Dev nodes publish no memattr table at all
(Proxmox OVMF) and are unaffected.

FIX: kernel >= 6.13 — upstream 8fbe4c49c0cc 'efi/memattr: Ignore table if the size is clearly
bogus', whose commit message describes this verbatim. Verified absent from v6.12 and from current
linux-6.12.y. On Debian 13: linux-image-amd64 from trixie-backports.

DO NOT buy RAM, move the DIMM, RMA the board, or run memtest expecting failures. Rebooting,
disabling Fast Boot, clearing CMOS and unplugging USB are all the same lottery — they move where
the table lands and may LOOK like fixes.

Records every killed hypothesis with its disproof so none of them get re-run.

Serves: R-ENV-01, R-ENV-03. Task: T-7.1.
2026-09-01 22:14:14 -05:00
jakman85 5ea9c93f58 docs(prod): BIOS password answer + CORRECT the 0x4 'auth failure' reading
Operator asked for the default BIOS password. ANSWER: there is none, and no factory default
exists to look up. HP business desktops ship with no Setup and no Power-On password; the
is_enabled=0 we measured IS the factory state. Recovery for a forgotten password is the PSWD
jumper — and 'stringent security' must NOT be enabled, since HP documents that it removes the
physical bypass and turns a 5-minute fix into a signed per-serial HP service case.

CORRECTION, and the kernel driver misled me: I recorded that the firmware refused our write for
lack of BIOS-admin authentication. Wrong. hp-bioscfg passes the return of HP's WMI SetBIOSSetting
through hp_wmi_error_and_message(), which is THE ERROR TABLE FOR A DIFFERENT INTERFACE (the
"SECU" commands under HP_WMI_BIOS_GUID). Per HP's Client Management Interface white paper,
SetBIOSSetting 4 = 'Failed' (generic) and 6 = 'Access Denied'. We got 4, not 6 — and the driver
has a separate BIOS_ADMIN_NOT_SET = 0x38 -> -EPERM for the no-password case, which we also did
not get. 'Invalid command type' is the driver looking 4 up in the wrong table: a genuine upstream
bug worth reporting to platform-driver-x86.

Most likely real cause (inferred, testable): with an empty password the driver emits an UNTAGGED
empty UTF-16 string, while every HP-documented caller passes Password="<utf-16/>" even when
blank — the driver only prepends that prefix for a non-empty string. Setting a password may fix
writes, but by flipping onto the tagged code path, NOT because 0x4 meant unauthorised.

Also verified from source: hp-bioscfg CANNOT set a password. current_password and new_password
both funnel into store_password_instance(), which only strscpy's into driver RAM; new_password is
write-only into a buffer nothing ever reads.

Count corrected: 241 attributes exposed, 239 with a current_value, 215 non-empty. I had quoted
215 as the total; it was the non-empty subset.

Recommendation recorded: land read-only drift detection now (zero risk, no new credential, and it
already caught real drift); treat a Setup password as a deliberate experiment on node 3 only;
hp-repsetup as fallback but it needs Secure Boot off or a self-signed module; skip Sure Admin/BCU
at this scale.

Serves: R-ENV-03, R-SEC-01. Task: T-7.1.
2026-09-01 22:10:07 -05:00
jakman85 3a88abb958 docs(prod): iGPU ruled out as the cause; CORRECT my last_pfn misreading
Operator asked whether node 2's missing memory could be reserved for integrated graphics.
TESTED, and no: /proc/iomem shows 'c8800000-cc7fffff : Graphics Stolen Memory' = EXACTLY 64 MiB,
matching the BIOS Video Memory Size=64 MB setting, and it sits BELOW the 4 GiB line. The UHD 630
(i915 bound) takes precisely what it is configured to take. It cannot account for 13.3 GiB and is
not even in the affected address range.

CORRECTION to my own analysis, prompted by checking that hypothesis properly: I cited the
'last_pfn revised DOWN from 0x42f800 to 0xc3c10' line as evidence of a memory cap. That was a
MISREADING. x86 boot prints last_pfn twice — first max_pfn from the full e820, then max_low_pfn,
the end of RAM below 4 GiB. The e820 low usable region ends at 0xc3c0ffff, so the next PFN is
exactly 0xc3c10, which is the second value observed. It matches to the byte and is normal on any
machine with more than 4 GiB. I presented a routine boot message as a smoking gun, and the
'memory remapping above 4 GB not being applied' signature claim rested partly on it.

The real anomaly is narrower than first stated: the Normal zone is 'present' but 'managed 0',
13.3 GiB is reserved, and nothing in the boot log explains by what. Ruled out so far: DIMM health,
firmware inventory, memory-remap settings (none exist on this platform), MTRR trim, iomem
registration, and now integrated graphics. Leading remaining hypothesis is stale memory-training
data cached by Fast Boot — testable with one F10 toggle.

Serves: R-ENV-03. Task: T-7.1.
2026-09-01 21:54:12 -05:00
gitea-admin 086e9091c9 Merge pull request 'feat(prod): node 2 online — open hardware fault (2.6 GiB of 16 GB) + BIOS drift' (#13) from feat/prod-node-2 into main 2026-09-02 02:49:10 +00:00
jakman85 9f7052ab2b docs(prod): fill in node-2 inventory; drop stale investigate bullets
Records node 2's as-found hardware alongside node 1 and removes two bullets that my earlier
edit left duplicated/obsolete (the BIOS comparison is done; clear-CMOS was listed twice).

Makes the hardware ASYMMETRY explicit in the table rather than buried in prose: node 1 is a
~2011 Micron C400 SATA SSD at 69% life with 15 grown bad blocks and 187 erase failures; node 2
is a Samsung PM9A1 NVMe at 96% life, PASSED. Same chassis, same BIOS, same CPU, same partition
layout — very different storage. §2 requires an identical disk layout across all three nodes
because Longhorn replica placement assumes symmetry, so this belongs in the inventory where the
disk decision will be made.

Serves: R-ENV-03. Task: T-7.1.
2026-09-01 21:45:37 -05:00
jakman85 3c8f809aaf docs(prod): CORRECTION — BIOS attributes are read-only in practice, not writable
I recorded that the hp-bioscfg attributes were 'writable from the OS' because current_value
has mode rw and no BIOS passwords are set. That was WRONG, and the firmware said so when the
write was actually attempted:

  hp_bioscfg: Returned error 0x4, "Invalid command type"   (surfaces as printf: I/O error)

HP requires BIOS-admin authentication for programmatic changes and neither credential is
provisioned on these nodes — Setup Password is_enabled=0 (role bios-admin), SPM
key_mechanism=not provisioned. Secure Boot is also enabled. File mode is not authority; the
firmware is.

What survives the correction, and it is the useful half: READS WORK. Diffing 215 attributes
between node 1 and node 2 is exactly how the drift in §0b was found — nobody spots
'After Power Loss=Power Off' by eye. So the §1 firmware checklist IS machine-CHECKABLE today
with reads alone, and node_health should assert it.

What does NOT survive: machine-ENFORCEABLE. Converging BIOS settings automatically would
require provisioning a BIOS Setup Password on every node — a stored credential that can rewrite
firmware. That is a real security decision and the operator's call, not something to default to.

Consequence for the node-2 memory fault: setting Fast Boot=Disable and After Power Loss=Power On
must be done in F10 on the box. Updated §0b and §1 accordingly.

Serves: R-ENV-03. Task: T-7.1.
2026-09-01 21:45:15 -05:00
jakman85 11b21f7347 docs(prod): BIOS settings compared programmatically via hp-bioscfg
Both nodes load the hp-bioscfg driver, exposing 215 BIOS attributes read/write at
/sys/class/firmware-attributes/. No BIOS passwords are set, so they are writable FROM THE OS.

RULED OUT as the cause of node 2's 2.6 GiB limit: firmware inventory is correct (BIOS itself
reports Memory Size=16384 MB); no memory-remap/above-4G attribute exists on this platform at
all; MTRRs are effectively identical with no trim warning; /proc/iomem registers the full
100000000-42f7fffff as System RAM and 127 memory blocks report online.

REAL DIFFERENCES FOUND (node1 works / node2 broken):
  After Power Loss          Power On  / Power Off   <- VIOLATES this document's own §1
  Fast Boot                 Disable   / Enable      <- leading hypothesis
  Legacy Boot Options       Disable   / Enable
  TPM Activation Policy     F1 to Boot / Allow user to reject
  MS+Windows UEFI CA 2023   Enable    / Disable

Leading hypothesis: HP Fast Boot reuses CACHED MEMORY-TRAINING DATA rather than retraining each
boot. A DIMM fitted while Fast Boot was on is a plausible route to memory being present,
inventoried, handed to the kernel — and then reserved. It is also the only memory-relevant
setting differing from the working node. NOT TESTED: changing it is a firmware write plus a
reboot and needs operator sanction.

BROADER FINDING for T-7.1: the §1 firmware checklist is now MACHINE-CHECKABLE rather than an
eyeball-at-F10 exercise, and machine-ENFORCEABLE too. That is precisely how node 2 came to have
After Power Loss=Power Off while node 1 has Power On — nobody could see the drift. Follow-up
recorded: extend node_health to assert the firmware checklist, add a role to converge it.

Raw BIOS dumps deliberately NOT committed — they carry serials, UUIDs and MAC addresses, same
reason node-health artifacts are gitignored. Only the findings are recorded.

Serves: R-ENV-03. Task: T-7.1.
2026-09-01 21:04:12 -05:00
jakman85andClaude Opus 5 cb69a53825 feat(prod): enable node 2 + HARDWARE FAULT — it addresses 2.6 GiB of its 16 GB
k3s-prod-2 (192.168.2.11) came online 2026-09-01. Enabled in the inventory, baseline package
floor installed, first health run completed. Vaulted become works on it unchanged.

*** OPEN FAULT: the node can only address ~2.6 GiB of a healthy, correctly-installed 16 GB
DIMM. *** Recorded in PROD-PROVISIONING §0b.

  dmidecode        16 GB Samsung M471A2K43DB1-CTD in DIMM1 @ 2667 MT/s (DIMM3 empty)
  BIOS e820        reports ~16 GiB usable, incl. 0x100000000-0x42f7fffff
  kernel           Memory: 2568544K/16546088K available (13958980K RESERVED)
  last_pfn         revised DOWN 0x42f800 (~16.7 GiB) -> 0xc3c10 (~3.06 GiB) in early boot
  zones            Normal present 3340288 pages (~12.7 GiB) but MANAGED 0
  MemTotal         2,753,284 kB = 2.6 GiB

The DIMM is NOT the problem: it is present, correctly identified, and at rated speed. The
kernel sees all 16 GB and then reserves 13.3 GiB. Everything below ~3 GiB usable, everything
above 4 GiB present-but-unmanaged — the signature of memory remapping above 4 GB not being
applied, not of failing memory. Node 1 is the control: identical chassis, identical BIOS R21
Ver. 02.26.00, identical CPU, and it addresses its full 31 GiB. Per-machine state, not a
firmware-version defect.

WARNING RECORDED: do NOT assume the incoming 32 GB fixes this. If the fault is the slot, the
board or a BIOS setting rather than the DIMM, 32 GB presents identically and the money is
wasted. Verify MemTotal after any RAM change.

ALSO FOUND — the prod nodes are NOT hardware-symmetric:
  node 1  Micron C400 512GB SATA SSD (~2011) — 69% life, 15 grown bad blocks, 187 erase
          failures, 32 unexpected power losses
  node 2  Samsung PM9A1 512GB NVMe        — 96% life, 2142 power-on hours, 34 unexpected
          power losses, overall-health PASSED
Same chassis, same BIOS, same CPU, same partition layout — very different storage. §2 requires
an identical disk layout across all three nodes because Longhorn replica placement assumes
symmetry, so this asymmetry is now a live input to the disk decision rather than a footnote.
Node 2's NVMe is the drive node 1 should have had.

Node 2 otherwise: 0 FAIL / 3 WARN / 12 PASS / 2 SKIP. Same three WARNs as node 1 (swap still
active pending full baseline, single IPv4 resolver, and the NVMe's 34 unexpected power losses),
plus the same unknowable-ECC SKIP.

Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-01 20:56:58 -05:00
gitea-admin f1daa8cae0 Merge pull request 'PROVEN: 1 → 2 — R-STATE-03 DR restore executed and proven' (#12) from feat/r-state-03-proven into main 2026-09-02 00:24:57 +00:00
jakman85andClaude Opus 5 380ea98a38 feat(conformance): R-STATE-03 PROVEN — DR restore from off-cluster backup, executed
The second requirement ever proven on this project, and the one the operator had attested
was working but the suite could not attest to. Test authored 2026-08-31 carrying a
`conformance.home.k3s/never-executed` annotation; executed for the first time tonight.

  test      dr-restore                              run   20260902T001846Z-fdc3265
  req       R-STATE-03 (safety-critical, DR)        junit passed / 0 failed / 0 skipped
  sentinel  DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK  proof proven
  sha       fdc3265 (clean tree)                    kube  v1.36.4+k3s1

WHAT IT ACTUALLY PROVED, in three steps:
1. The backup target is genuinely OFF-CLUSTER. BSL velero/default phase=Available at
   http://192.168.1.9:9000 — endpoint resolved and asserted to be none of: loopback, any
   node Internal/ExternalIP, any Service ClusterIP, any Pod IP, inside any node podCIDR, or
   inside any ServiceCIDR.
2. Backups are SCHEDULED, not merely possible — velero-daily and
   velero-daily-resources-only, both phase=Enabled, last run 2026-09-01T03:00:27Z.
3. The drill: seed a nanosecond-unique canary onto a longhorn PVC; Velero Backup to
   Completed with a kopia PodVolumeBackup carrying a real snapshotID (volume DATA reached
   the remote repository, not just resource JSON); ANNIHILATE the source — delete the
   namespace, poll until gone, force-delete the PV, FAIL if the original PV survived;
   Restore into a clean namespace; assert the restored PVC bound a DIFFERENT PV and the
   canary bytes are byte-identical.
The bytes could only have come from outside the cluster.

VALIDATES THE PARSER FIX FROM 57d4df8. This log contains SEVEN occurrences of
DR_RESTORE_SKIPPED_OK — every one of them inside a `=== COMMAND` script echo — and the pass
token twice in real `=== STDOUT`. Under the pre-fix scanner this genuine pass would have been
recorded as `skipped`, exactly as node-loss was. The section-parsing rule is doing real work,
not theoretical work.

CLEANUP VERIFIED against a pre-run baseline: Velero backups back to 7 (its own
chainsaw-*-dr-backup deleted via DeleteBackupRequest), 0 chainsaw backups, 0 restores, 0
orphan Released/Available PVs, both scratch namespaces gone, 13/13 Kustomizations Ready, 0
pods not Running. It wrote into and cleaned out of the SAME bucket and kopia repository the
live nightly schedule uses, and left it as it found it.

SEPARATE FINDING, cleaned up: `chainsaw-stable-worm` was still Active from 2026-08-25 —
empty residue from an earlier conformance session that never tore its namespace down. Deleted.
Worth knowing that Chainsaw namespace leaks can outlive a run by a week unnoticed.

NOT proven by this test, stated in its own header rather than implied: off-HOST DR (the dev
nodes are VMs on the same Proxmox host as the S3 target, so this does not survive hypervisor
loss); no direct S3 client read; real service data (Nextcloud/Gitea/CNPG drills remain
operator exercises); bare-metal rebuild DR; encryption-at-rest; retention/GC.

Serves: R-STATE-03, R-TEST-03, R-TEST-05.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-01 19:23:09 -05:00
gitea-admin acae2ad36c Merge pull request 'PROVEN: 0 → 1 — R-STATE-02 proven, plus two ledger bugs that made proof impossible' (#11) from feat/ledger-section-parsing into main 2026-09-02 00:10:06 +00:00
jakman85andClaude Opus 5 fdc3265585 fix(conformance): the ledger counted as evidence FOR the requirements it proves
Committing the first PROVEN row immediately degraded it to `stale-proof`. Cause: a
self-defeating loop.

`IMPL_EXCLUDE_PREFIXES` excluded `conformance/tests/` and `ci/tests/` but NOT
`conformance/runs/`. The ledger and the preserved run logs CONTAIN the requirement IDs they
are evidence FOR — `ledger.jsonl` literally holds `"requirements":["R-STATE-02"]` — so they
were scanned as IMPLEMENTING evidence for R-STATE-02. That made `last_change` for the
requirement move to the ledger commit itself, which the row's `git_sha` necessarily predates,
so the freshness check failed.

The consequence was total, not marginal: **recording a proof destroyed it, so NO ledger row
could ever have been non-stale.** The ledger would have reported `PROVEN: 0 (stale N)` forever
while looking like it was working — the same shape of silent-uselessness as the COMMAND-echo
bug in the previous commit, and found the same way: by actually running the thing and
disbelieving a green-looking result.

Fix: exclude `conformance/runs/`. The ledger is the proof RECORD; it must not participate in
the freshness computation about itself.

Verified: `PROVEN: 1 (stale 0, self-skipped 0, failed 0, unknown 0) from 1 ledger row(s)`;
R-STATE-02 reads **proven**. status.py --selftest PASS. The staleness machinery is otherwise
untouched — a genuinely stale row (impl changed after the run) still degrades, which the
selftest continues to assert.

Serves: R-TEST-01..04.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-01 19:08:38 -05:00
jakman85 4cb5e7a8d4 feat(conformance): first PROVEN row — R-STATE-02 node-loss
Records the first requirement this project has ever PROVEN, in the only sense its own
definition of done accepts: a green, requirement-tagged, reproducible behaviour test,
executed against a real cluster, with the evidence committed.

  test      node-loss                      run   20260901T235117Z-e9a4b66
  req       R-STATE-02 (safety-critical)   junit passed
  sentinel  NODE_LOSS_VOLUME_AVAILABLE_OK  proof proven
  sha       57d4df8 (clean tree)           kube  v1.36.4+k3s1

What actually ran: seed a 3-replica longhorn volume, cordon the node the writer landed on,
evict only its own pod, assert the volume stays available and a reader on another node
re-attaches and reads the data back. 131s. It deliberately does NOT drain, so kubelet and
the etcd member stay up and quorum is never touched — which is what retired the old
'defer node-loss to a maintenance window' reason as unfounded rather than by fiat.

Recorded against the clean tree of the preceding commit so git_sha resolves and the
freshness check passes. The test was NOT re-run for this: the preserved run.log is the
evidence, and re-running would have proved the parser rather than preserving the run.

Serves: R-STATE-02, R-TEST-03, R-TEST-05.
2026-09-01 19:07:39 -05:00
jakman85andClaude Opus 5 57d4df8c4f fix(conformance): ledger read tokens from Chainsaw's COMMAND echo, not real output
The run ledger mis-classified the FIRST real L3 execution — a genuine PASS recorded as
`skipped`. Diagnosis, from the preserved log, section-classified:

  line  51  COMMAND   echo NODE_LOSS_SKIPPED_OK        <- script SOURCE
  line 190  STDOUT    NODE_LOSS_VOLUME_AVAILABLE_OK    <- the only REAL emission

Chainsaw echoes each script into a `=== COMMAND` block before running it, and every
skip-gated test contains `echo <TEST>_SKIPPED_OK` as source text. record_run.py scanned the
raw log without distinguishing sections, so it matched the skip token in the echo.
NODE_LOSS_SKIPPED_OK never appeared in output at all.

SEVERITY: since every skip-gated test embeds its own skip token as source, and skip outranks
pass, THE LEDGER COULD NEVER HAVE RECORDED `proven` FOR ANY TEST WITH A SKIP GATE. It would
have silently recorded nothing forever while appearing to work. It failed CLOSED — the right
direction — but a ledger that always fails closed is an expensive way to write "unknown".

FIX: classify_sections() splits the log on indented `=== NAME` markers (a section runs to the
first non-blank line indented LESS than its marker). Verified against the real log's actual
markers, including that Go's column-0 `=== RUN/PAUSE/CONT` lines carry a test name and must
never open a section. Then:
  PASS tokens are read from STDOUT ONLY — exactly where Chainsaw's own contains($stdout,…)
    looks, so a source echo or stderr can never manufacture a proof.
  SKIP tokens are read from every section that is NOT COMMAND, so an unknown future section
    name fails closed in BOTH directions.
  Lines outside any section prove nothing — this also closes a second false-positive vector:
    a FAILING Chainsaw run prints the assertion expression, which contains both tokens.
  A log with no markers is scanned whole exactly as before (such logs have no COMMAND echo).

SECOND BUG, same root: _LOG_HEADER_RE never matched a real Chainsaw log — the lines are
`    l.go:52: <BS>| 23:51:18 | <ANSI>node-loss<ANSI> | …`, not `| … |`. EVERY genuine run was
silently falling into DEGRADED whole-log mode. Now strips ANSI/backspace noise first and
accepts the Go source prefix, so runs attribute per-testcase as designed.

Safety properties re-asserted, not assumed: skip still beats pass WITHIN real output; unknown
layers still require a sentinel; JUnit failed/error/skipped still outrank everything; missing
JUnit or log still exits 3. Negative control: with scan_texts forced section-blind in a
throwaway copy, exactly the new cases fail — they do catch this bug.

Includes the preserved run directory 20260901T235117Z-e9a4b66 (report.junit.xml + run.log)
for R-STATE-02 / node-loss — 131s, cordon -> evict -> volume stayed available -> reader
re-attached -> clean teardown, the first L3 fault-injection test ever executed on this
cluster. The ledger row is recorded in the FOLLOWING commit, deliberately: git_sha is sampled
at record time and a dirty tree records `-dirty`, which is unresolvable and correctly degrades
to stale-proof. Recording against this clean commit is the honest way to get a resolvable sha
— NOT a --git-sha override, which would be a lever for recording a clean sha over a dirty
tree and was deliberately not added.

Serves: R-TEST-01..04, R-STATE-02.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-01 19:07:26 -05:00
gitea-admin 14494e1947 Merge pull request 'feat(conformance): committed run ledger — prerequisite for moving PROVEN off zero' (#10) from feat/conformance-run-ledger into main 2026-09-01 23:50:28 +00:00
jakman85andClaude Opus 5 e9a4b669d4 feat(conformance): committed run ledger — the only thing that can move PROVEN off zero
status.py has always reported PROVEN: 0 for all 51 requirements, and named the fix in its
own header: "a committed, machine-readable run ledger". There was none — _artifacts/ is
gitignored, the root .gitignore kills *.junit.xml, and a stray JUnit was surfaced only as
a quarantined mtime advisory. Built BEFORE the first real L3 execution so that run is
captured rather than reconstructed from memory.

THE TRAP THIS EXISTS TO AVOID. These tests exit 0 and Chainsaw records `passed` on a
SELF-SKIP exactly as on a real run, and Chainsaw's JUnit carries no stdout — only
properties. So committing the JUnit as proof would let DR_RESTORE_SKIPPED_OK be recorded
as proof of R-STATE-03: authoritative-looking and wrong, which is worse than no ledger.
The sentinel token is the ONLY thing distinguishing proven from skipped, and it lives
solely in the run log. The ledger therefore records the LOG alongside the JUnit and
derives proof from the sentinel, never from the JUnit result.

Sentinel tokens are DERIVED, not hand-listed: record_run.py parses every
contains($stdout,'TOKEN') out of each test's own chainsaw-test.yaml, so the list cannot
drift from the tests. 23 of 35 tests declare no sentinel at all.

decide() is the single place `proven` is emitted, ordered WORST-FIRST:
  JUnit failure/error -> failed
  JUnit <skipped/>    -> junit-skipped
  ANY declared skip token in the log -> skipped   (beats a green JUnit AND beats a pass
                                                   token present in the same log)
  no sentinel declared -> proven ONLY for L0/L1 (whose JUnit is truthful on its own, since
                          run_l1_offline.py maps exit 3 to <skipped/>); written as an
                          explicit exempt-set so an unknown/blank layer fails CLOSED
  EVERY declared pass token present -> proven
  otherwise -> unknown-sentinel-absent
Log attribution is a state machine over Chainsaw's per-test headers; a log with no
recognisable header falls back to whole-log scanning restricted to GLOBALLY-UNIQUE tokens,
so DNS_OK (owned by two tests) can never prove either. record_run.py exits 3 if the JUnit
or the log is missing — nothing recorded, never green.

STALENESS: a proven row is honoured only while `git merge-base --is-ancestor` holds
between the requirement's last implementing/test change and the row's git_sha; otherwise
it degrades to stale-proof. A -dirty sha, an unknown rev, or an uncommitted test path all
fail ancestry the same way. A requirement reads proven only when EVERY verifying test
does (worst-of), and the latest row per test wins — not best-ever.

REFUSAL POSTURE PRESERVED: with an empty ledger the report is unchanged in intent — every
requirement not-derivable, headline "PROVEN: 0 — NOT DERIVABLE". This replaces "not
derivable" with real evidence only where evidence exists, never with optimism.

--record is explicit, never implicit (a run that silently records itself is how a bad run
becomes permanent false proof), is rejected for --status/--coverage-only/--lint, and
re-execs under tee returning ${PIPESTATUS[0]} so recording can never soften a gate. It
rm -f's a stale _artifacts/report.junit.xml first — a hazard hit live during development,
where a stale JUnit got paired with a fresh log.

Selftests: record_run.py 23 checks (incl. end-to-end Chainsaw-passed + NODE_LOSS_SKIPPED_OK
-> proof "skipped"), status.py +16 against a throwaway git repo (self-skip -> skipped;
proven-at-older-sha -> stale-proof; dirty/unresolvable -> stale-proof; worst-of;
latest-row-wins; malformed lines dropped). No regressions: --coverage-only, --status,
--lint, --strict-coverage and the merge-gate shape all rc=0 and unchanged; nothing in ci/
references status.py.

Root .gitignore needed no change — conformance/runs/ is not itself excluded, so the deeper
negations override the root's global *.junit.xml. Verified with git check-ignore.

Serves: R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-01 18:49:01 -05:00
gitea-admin aa0a50a143 Merge pull request 'chore(flux): restore platform prune:true — Cilium split merge 2 of 2' (#9) from chore/cilium-split-merge2 into main 2026-09-01 23:41:51 +00:00
jakman85andClaude Opus 5 22174186ff chore(flux): restore platform prune:true — MERGE 2 of 2 (Cilium split cleanup)
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
Closes the transition opened in merge 1 (63f98aa). `prune: true` is restored on the
`platform` Kustomization now that the ownership transfer is VERIFIED on the live cluster,
not merely expected:

  13/13 Kustomizations Ready, `cilium` among them, at main@sha1:23aa2854
  Namespace/kube-system          label kustomize.toolkit.fluxcd.io/name = cilium
  HelmRelease/cilium             label ... = cilium
  HelmRepository/cilium          label ... = cilium
  platform  inventory: 0 entries matching kube-system__Namespace | kube-system_cilium_*
  cilium    inventory: exactly those three objects
  Cilium pods UNCHANGED from the pre-merge baseline —
    cilium-2f8h8 r=0 2026-09-01T05:55:27Z
    cilium-425m8 r=1 2026-08-30T13:25:10Z
    cilium-54cww r=0 2026-09-01T05:55:26Z
  HelmRelease still v11

So the relabel deterministically preceded any prune, exactly as the revision-gated
dependsOn edge predicted, and the ownership move cost zero pod restarts.

The in-file comment now records the lesson rather than the temporary state: before moving
ANY platform/* component in future, check whether it is the sole declarer of a shared
Namespace. platform/cilium/namespace.yaml was the only declaration of Namespace/kube-system
under ./platform, which would have turned a component-scoped prune into
`delete namespace kube-system` — taking coredns, metrics-server, local-path and Cilium.

L0 substitution gate: 103 assertions, 0 hazards.

Serves: R-PLAT-01, R-GITOPS-01.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-01 18:39:20 -05:00
gitea-admin 23aa2854f1 Merge pull request 'feat(flux): Cilium gets its own Kustomization — merge 1 of 2' (#8) from feat/cilium-own-kustomization into main 2026-09-01 23:35:04 +00:00
jakman85andClaude Opus 5 63f98aab26 feat(flux): give Cilium its own Kustomization — MERGE 1 of 2 (transition)
Cilium was one of ~12 components inside a single `platform` Kustomization with wait:true,
so the CNI's health was indistinguishable from eleven unrelated components in the tier's
Ready condition, and the only emergency valve was freezing the whole platform tier. This
repo already set the precedent in reverse (oauth2-proxy and headlamp were split OUT so
unhealthy leaves would not gate a tier); Cilium is the mirror image — the one component
that legitimately SHOULD gate everything.

New clusters/base/cilium.yaml: path ./platform/cilium, wait:true, prune:true, interval 1h
(a CNI does not need a 10m re-render cadence; kustomize-controller is source-triggered so
a push still reconciles immediately — the interval is only the drift-correction floor),
timeout 10m (a 3-node DaemonSet rollout with wait exceeds the 5m default),
dependsOn:[flux-system], and postBuild.substituteFrom: cluster-vars — the last is
LOAD-BEARING, not decorative: helmrelease.yaml references ${K3S_API_HOST} and
kustomize-controller now runs --feature-gates=StrictPostBuildSubstitutions=true, so
without it this Kustomization would fail immediately.

*** THE HAZARD WAS FAR WORSE THAN "the CNI gets pruned" ***
platform/cilium/namespace.yaml is the ONLY declaration of Namespace/kube-system anywhere
under ./platform, and the live object carries kustomize.toolkit.fluxcd.io/name=platform.
With prune:true, the moment ./platform/cilium leaves platform's path the stale set is not
"the Cilium HelmRelease" — it is `kubectl delete namespace kube-system`, taking coredns,
metrics-server, local-path AND Cilium. A full cluster kill, not a CNI kill.

TWO INDEPENDENT PROTECTIONS, deliberately not trusting either alone:
1. prune:false on `platform` for this transition merge — Flux's documented Procedure 1 for
   moving resources between Kustomizations. Chosen over the per-resource
   `kustomize.toolkit.fluxcd.io/prune: disabled` annotation because that needs THREE merges
   and leaves the new Kustomization owning resources it can never prune.
2. The pruner's own owner-label filter. In v1.4.0, prune() passes Inclusions:
   GetOwnerLabels(obj.Name...), so platform only deletes objects whose LIVE labels still
   say name=platform. The new cilium Kustomization relabels all three to name=cilium on
   apply, and platform.dependsOn:[cilium] is REVISION-GATED, so platform provably cannot
   reconcile revision N until cilium has APPLIED revision N. The relabel deterministically
   precedes the prune.
Protection 2 alone would probably have sufficed. Not betting kube-system on "probably".

NOTED FOR THE FUTURE, in-file: anyone moving a platform/* component must first check
whether it is the sole declarer of a shared Namespace — that turns a component-scoped
prune into a cluster-scoped one.

Verified: `kubectl kustomize platform` goes 88 -> 85 documents, 3 removed / 0 added /
0 CHANGED, and `kubectl kustomize platform/cilium` reproduces those 3 byte-identical
(sha256-matched). A pure ownership move; no Cilium value touched. L0 substitution gate
103 assertions (was 102), new cilium row, 0 hazards. Server-side dry-run against the live
cluster accepted both CRs. clusters/prod renders clean off the same base.

MERGE 2 (restore prune:true) follows only after verifying: 13/13 Ready; kube-system, the
HelmRelease and the HelmRepository all relabelled to name=cilium; platform's inventory no
longer lists them; and Cilium pod ages/restarts UNCHANGED from
  cilium-2f8h8 r=0 2026-09-01T05:55:27Z / cilium-425m8 r=1 2026-08-30T13:25:10Z /
  cilium-54cww r=0 2026-09-01T05:55:26Z, HelmRelease v11.

Serves: R-PLAT-01, R-GITOPS-01.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-01 18:33:20 -05:00
gitea-admin 62968c9ee9 Merge pull request 'docs: P0/P1/P2 landed + postStart crashloop incident' (#7) from docs/poststart-incident into main 2026-09-01 22:02:56 +00:00
319 changed files with 40526 additions and 1054 deletions

No files matched your search

+43 -20
View File
@@ -1,25 +1,44 @@
# .gitea/workflows/gate.yml — the ENFORCED PR merge gate (Phase 6, WS-GITOSS).
#
# This is the LIVE gate wired to the in-cluster Gitea Actions runner (ns gitea-actions,
# Docker executor + DinD). It invokes the AGGREGATE merge gate — `make -C ci gate` — so CI
# runs BYTE-FOR-BYTE the same command a human runs in the devcontainer (R-TEST-02): the
# runner just installs the pinned tools from ci/versions.env first. `make -C ci gate`
# (ci/gate.sh) runs every stage in order — spec-immutability, lint, Kyverno L0,
# render==golden, L1 offline render — and applies the R-TEST-03 decision, printing
# "MERGE GATE: PASS" / "MERGE GATE: BLOCKED". It is the single required status check on
# `main` (see branch protection). A non-zero exit fails the job, so a BLOCKED gate blocks
# the merge.
# Docker executor + DinD). It invokes the AGGREGATE merge gate — `./ci/gate.sh`, which is
# exactly what `make -C ci gate` runs, so CI and the devcontainer gate the same way
# (R-TEST-02): the runner just installs the pinned tools from ci/versions.env first (and see
# below for why the script, not `make`, is invoked here). It runs every stage in order —
# spec-immutability, lint, Kyverno L0, render==golden, L1 offline render — and then applies the
# R-TEST-03 decision, printing
# "MERGE GATE: PASS" (0) / "BLOCKED" (1) / "UNDETERMINED" (3) / "NOT APPLICABLE" (0). It is the
# single required status check on `main` (see branch protection). A non-zero exit fails the job,
# so a BLOCKED gate blocks the merge.
#
# WHY THIS STEP DECLARES GATE_STRICT_SKIPS AND TOLERATES EXIT 3 (2026-09-13). A SKIPPED stage is
# never green, and since 2026-09-13 gate.sh refuses to print PASS when one did not run (it used to
# print "— not green" and then "MERGE GATE: PASS", exit 0 — fail-open in the gate itself). Two
# kinds of non-run must be told apart HERE, in the caller, because only the caller knows which
# environment it is:
# * a pinned tool missing or a seam unwired ON THIS RUNNER is a DEFECT — the toolchain is
# installed for us two steps up. GATE_STRICT_SKIPS=true DECLARES that (no env sniffing, no
# "does this look like CI?" heuristic) and makes those skips exit 1: untolerated, job fails.
# * spec-immutability CANNOT be verified here at all: the spec repo is a separate repo and is
# not checked out into this runner (SPEC_REPO_SLUG unwired; blueprint in merge-gate.yaml). It
# is the ONLY skip that can still reach exit 3 under strict skips, which is what makes
# `|| [ "$?" -eq 3 ]` a narrow, reviewed tolerance rather than a blanket one. The log still
# says UNDETERMINED and names R-TEST-06 as unverified. Wire the spec checkout (or set
# REQUIRE_SPEC_VERIFICATION=true) and this tolerance should be deleted.
# Called DIRECTLY, not through `make`: GNU make reports any recipe failure as its own exit 2, so
# `make -C ci gate || [ "$?" -eq 3 ]` would see 2 and fail the job. Same reason the
# spec-immutability pre-step below calls the script directly.
#
# The cluster-gated L2/L3 conformance stage SKIPS honestly here (exit 3, surfaced not faked
# — HARD RULE 4) because this runner declares no cluster: `make -C ci gate` does not set
# CLUSTER_AVAILABLE, so gate.sh reports the touched requirements UNVERIFIED without blocking
# Phase-0/6 static work. L2/L3 run as an IN-CLUSTER Chainsaw Job (its own scoped SA; CI holds
# — HARD RULE 4) because this runner declares no cluster: nothing here sets CLUSTER_AVAILABLE,
# so gate.sh reports the touched requirements UNVERIFIED — and NAMES that declared gap in its
# PASS headline — without blocking Phase-0/6 static work. L2/L3 run as an IN-CLUSTER Chainsaw Job (its own scoped SA; CI holds
# no broad kubeconfig) driven by the separate `conformance` job below.
#
# NOTE (R-TEST-06 / gate-semantics): spec-immutability is ALSO surfaced as its own explicit
# early step (below) so its result is a distinct, tolerant status in the Gitea UI; gate.sh
# re-runs it as its first stage, which is the authoritative one. `make -C ci gate` here is
# the byte-identical laptop gate; the extra pre-step is a read-only git-diff check and is
# re-runs it as its first stage, which is the authoritative one. The gate step below is the
# same script the laptop runs; the extra pre-step is a read-only git-diff check and is
# purely additive. The full pipeline blueprint (spec-checkout, per-layer jobs, decision job)
# is retained dormant in merge-gate.yaml.
---
@@ -69,13 +88,17 @@ jobs:
- name: Spec immutability (R-TEST-06, skips if spec repo absent)
run: ./ci/spec-immutability.sh || [ "$?" -eq 3 ]
# Aggregate merge gate: byte-for-byte the command a human runs in the devcontainer
# (`make -C ci gate` -> ci/gate.sh). Runs spec-immutability, lint, Kyverno L0,
# render==golden and L1 render, then the R-TEST-03 decision. Exits non-zero on a
# BLOCKED gate, which fails this job. The cluster-gated L2/L3 stage SKIPS here (no
# CLUSTER_AVAILABLE) — surfaced as UNVERIFIED, never faked green (see header).
- name: Merge gate — make -C ci gate
run: make -C ci gate
# Aggregate merge gate: the same script a human runs in the devcontainer via
# `make -C ci gate`. Runs spec-immutability, lint, Kyverno L0, render==golden and L1
# render, then the R-TEST-03 decision. Exits non-zero on a BLOCKED gate, which fails this
# job. The cluster-gated L2/L3 stage SKIPS here (no CLUSTER_AVAILABLE) — surfaced as
# UNVERIFIED and NAMED in the PASS headline, never faked green. GATE_STRICT_SKIPS + the
# tolerated exit 3 are explained in the file header; the tolerance covers exactly one
# thing, the unverifiable spec repo, and nothing else can produce a 3 here.
- name: Merge gate — ci/gate.sh (what `make -C ci gate` runs)
env:
GATE_STRICT_SKIPS: "true"
run: ./ci/gate.sh || [ "$?" -eq 3 ]
# NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4
# ("not supported on GHES" — Gitea does not implement the artifact-v2 backend), which
+17 -2
View File
@@ -39,6 +39,21 @@ install_chainsaw() {
curl -fsSL "$url" | tar -xz -C "$bindir" chainsaw
}
# promtool — the alert-rule UNIT-TEST engine the L0 check
# conformance/tests/l0-static/alert-rules-unit-tested runs. Installed for the l0 stage because
# without it that check exits 3 (SKIPPED, never green) on the runner and the alert unit tests
# would be enforced nowhere — the exact "coverage that looks present" failure it exists to end.
# Pinned from ci/versions.env to the Prometheus version prod runs (promtool IS the rule engine);
# the check asserts `promtool --version` equals that pin, so a drifted install FAILS, not skips.
install_promtool() {
have promtool && return 0
log "promtool ${PROMTOOL_VERSION} (from the prometheus release tarball)"
local tarball="prometheus-${PROMTOOL_VERSION}.${os}-${arch}"
curl -fsSL "https://github.com/prometheus/prometheus/releases/download/v${PROMTOOL_VERSION}/${tarball}.tar.gz" \
| tar -xz -C "$bindir" --strip-components=1 "${tarball}/promtool"
chmod +x "${bindir}/promtool"
}
install_kubectl() {
have kubectl && return 0
log "kubectl ${KUBECTL_VERSION}"
@@ -50,10 +65,10 @@ have() { command -v "$1" >/dev/null 2>&1; }
case "$stage" in
lint) install_yamllint ;;
l0) install_yamllint; install_check_jsonschema; install_kyverno ;; # schema (check-jsonschema) + Kyverno L0
l0) install_yamllint; install_check_jsonschema; install_kyverno; install_promtool ;; # schema (check-jsonschema) + Kyverno L0 + alert unit tests
l1) install_chainsaw ;; # L1 render via conformance runner (Chainsaw)
conformance) install_chainsaw; install_kubectl ;; # trigger the in-cluster Job
all) install_yamllint; install_check_jsonschema; install_kyverno; install_chainsaw; install_kubectl ;;
all) install_yamllint; install_check_jsonschema; install_kyverno; install_promtool; install_chainsaw; install_kubectl ;;
*) echo "unknown stage: $stage" >&2; exit 2 ;;
esac
log "stage '${stage}' tools ready"
+57
View File
@@ -0,0 +1,57 @@
A|*.keithjackson.dev|136.51.115.252
A|auth.home.keithjackson.dev|192.168.1.53
A|brickbreak.keithjackson.dev|20.236.242.14
A|crafty-mc.home.keithjackson.dev|192.168.1.55
A|crafty.home.keithjackson.dev|192.168.1.53
A|dev.brickbreak.keithjackson.dev|20.236.242.14
A|gitea.home.keithjackson.dev|192.168.1.53
A|grafana.home.keithjackson.dev|192.168.1.53
A|headlamp.home.keithjackson.dev|192.168.1.53
A|home.keithjackson.dev|192.168.1.53
A|jackett.home.keithjackson.dev|192.168.1.53
A|keycloak.home.keithjackson.dev|192.168.1.53
A|nextcloud.home.keithjackson.dev|192.168.1.53
A|pihole.home.keithjackson.dev|192.168.1.53
A|plex.home.keithjackson.dev|192.168.1.53
A|qbittorrent.home.keithjackson.dev|192.168.1.53
A|radarr.home.keithjackson.dev|192.168.1.53
A|seerr.home.keithjackson.dev|192.168.1.53
A|semaphore.home.keithjackson.dev|192.168.1.53
A|sonarr.home.keithjackson.dev|192.168.1.53
A|unifi.home.keithjackson.dev|192.168.1.54
TXT|a-auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
TXT|a-crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
TXT|a-crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
TXT|a-gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
TXT|a-grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
TXT|a-headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
TXT|a-home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
TXT|a-jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
TXT|a-keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
TXT|a-nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
TXT|a-pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
TXT|a-plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
TXT|a-qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
TXT|a-radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
TXT|a-seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
TXT|a-semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
TXT|a-sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
TXT|a-unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
TXT|auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
TXT|crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
TXT|crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
TXT|gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
TXT|grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
TXT|headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
TXT|home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
TXT|jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
TXT|keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
TXT|nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
TXT|pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
TXT|plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
TXT|qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
TXT|radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
TXT|seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
TXT|semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
TXT|sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
TXT|unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
+95
View File
@@ -0,0 +1,95 @@
A|*.keithjackson.dev|136.51.115.252
A|auth.home.keithjackson.dev|192.168.1.53
A|auth.prod.home.keithjackson.dev|192.168.2.20
A|brickbreak.keithjackson.dev|20.236.242.14
A|conformance-dns.home.keithjackson.dev|192.168.1.56
A|crafty-mc.home.keithjackson.dev|192.168.1.55
A|crafty-mc.prod.home.keithjackson.dev|192.168.2.22
A|crafty.home.keithjackson.dev|192.168.1.53
A|crafty.prod.home.keithjackson.dev|192.168.2.20
A|dev.brickbreak.keithjackson.dev|20.236.242.14
A|gitea.home.keithjackson.dev|192.168.1.53
A|gitea.prod.home.keithjackson.dev|192.168.2.20
A|grafana.home.keithjackson.dev|192.168.1.53
A|grafana.prod.home.keithjackson.dev|192.168.2.20
A|headlamp.home.keithjackson.dev|192.168.1.53
A|headlamp.prod.home.keithjackson.dev|192.168.2.20
A|home.keithjackson.dev|192.168.1.53
A|jackett.home.keithjackson.dev|192.168.1.53
A|jackett.prod.home.keithjackson.dev|192.168.2.20
A|keycloak.home.keithjackson.dev|192.168.1.53
A|keycloak.prod.home.keithjackson.dev|192.168.2.20
A|nextcloud.home.keithjackson.dev|192.168.1.53
A|nextcloud.prod.home.keithjackson.dev|192.168.2.20
A|pihole.home.keithjackson.dev|192.168.1.53
A|pihole.prod.home.keithjackson.dev|192.168.2.20
A|plex.home.keithjackson.dev|192.168.1.53
A|plex.prod.home.keithjackson.dev|192.168.2.20
A|prod.home.keithjackson.dev|192.168.2.20
A|qbittorrent.home.keithjackson.dev|192.168.1.53
A|qbittorrent.prod.home.keithjackson.dev|192.168.2.20
A|radarr.home.keithjackson.dev|192.168.1.53
A|radarr.prod.home.keithjackson.dev|192.168.2.20
A|seerr.home.keithjackson.dev|192.168.1.53
A|seerr.prod.home.keithjackson.dev|192.168.2.20
A|semaphore.home.keithjackson.dev|192.168.1.53
A|semaphore.prod.home.keithjackson.dev|192.168.2.20
A|sonarr.home.keithjackson.dev|192.168.1.53
A|sonarr.prod.home.keithjackson.dev|192.168.2.20
A|unifi.home.keithjackson.dev|192.168.1.54
A|unifi.prod.home.keithjackson.dev|192.168.2.21
TXT|a-auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
TXT|a-auth.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
TXT|a-conformance-dns.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/chainsaw-growing-grouse/dns-probe-svc
TXT|a-crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
TXT|a-crafty-mc.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/crafty/crafty-l4-tcp
TXT|a-crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
TXT|a-crafty.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/crafty/crafty
TXT|a-gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
TXT|a-gitea.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/gitea/gitea-3000
TXT|a-grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
TXT|a-grafana.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/monitoring/grafana
TXT|a-headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
TXT|a-headlamp.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/headlamp/headlamp
TXT|a-home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
TXT|a-jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
TXT|a-jackett.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/jackett/jackett-9117
TXT|a-keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
TXT|a-keycloak.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/keycloak/keycloak
TXT|a-nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
TXT|a-nextcloud.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/nextcloud/nextcloud-80
TXT|a-pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
TXT|a-pihole.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/pihole/pihole-80
TXT|a-plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
TXT|a-plex.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/plex/plex-32400
TXT|a-prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/homepage/homepage
TXT|a-qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
TXT|a-qbittorrent.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
TXT|a-radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
TXT|a-radarr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/radarr/radarr-7878
TXT|a-seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
TXT|a-seerr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/seerr/seerr-5055
TXT|a-semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
TXT|a-semaphore.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/semaphore/semaphore-3000
TXT|a-sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
TXT|a-sonarr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/sonarr/sonarr-8989
TXT|a-unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
TXT|a-unifi.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/unifi/unifi-l4-tcp
TXT|auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
TXT|crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
TXT|crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
TXT|gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
TXT|grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
TXT|headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
TXT|home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
TXT|jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
TXT|keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
TXT|nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
TXT|pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
TXT|plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
TXT|qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
TXT|radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
TXT|seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
TXT|semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
TXT|sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
TXT|unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
+93
View File
@@ -0,0 +1,93 @@
A|*.keithjackson.dev|136.51.115.252
A|auth.home.keithjackson.dev|192.168.1.53
A|auth.prod.home.keithjackson.dev|192.168.2.20
A|brickbreak.keithjackson.dev|20.236.242.14
A|crafty-mc.home.keithjackson.dev|192.168.1.55
A|crafty-mc.prod.home.keithjackson.dev|192.168.2.22
A|crafty.home.keithjackson.dev|192.168.1.53
A|crafty.prod.home.keithjackson.dev|192.168.2.20
A|dev.brickbreak.keithjackson.dev|20.236.242.14
A|gitea.home.keithjackson.dev|192.168.1.53
A|gitea.prod.home.keithjackson.dev|192.168.2.20
A|grafana.home.keithjackson.dev|192.168.1.53
A|grafana.prod.home.keithjackson.dev|192.168.2.20
A|headlamp.home.keithjackson.dev|192.168.1.53
A|headlamp.prod.home.keithjackson.dev|192.168.2.20
A|home.keithjackson.dev|192.168.1.53
A|jackett.home.keithjackson.dev|192.168.1.53
A|jackett.prod.home.keithjackson.dev|192.168.2.20
A|keycloak.home.keithjackson.dev|192.168.1.53
A|keycloak.prod.home.keithjackson.dev|192.168.2.20
A|nextcloud.home.keithjackson.dev|192.168.1.53
A|nextcloud.prod.home.keithjackson.dev|192.168.2.20
A|pihole.home.keithjackson.dev|192.168.1.53
A|pihole.prod.home.keithjackson.dev|192.168.2.20
A|plex.home.keithjackson.dev|192.168.1.53
A|plex.prod.home.keithjackson.dev|192.168.2.20
A|prod.home.keithjackson.dev|192.168.2.20
A|qbittorrent.home.keithjackson.dev|192.168.1.53
A|qbittorrent.prod.home.keithjackson.dev|192.168.2.20
A|radarr.home.keithjackson.dev|192.168.1.53
A|radarr.prod.home.keithjackson.dev|192.168.2.20
A|seerr.home.keithjackson.dev|192.168.1.53
A|seerr.prod.home.keithjackson.dev|192.168.2.20
A|semaphore.home.keithjackson.dev|192.168.1.53
A|semaphore.prod.home.keithjackson.dev|192.168.2.20
A|sonarr.home.keithjackson.dev|192.168.1.53
A|sonarr.prod.home.keithjackson.dev|192.168.2.20
A|unifi.home.keithjackson.dev|192.168.1.54
A|unifi.prod.home.keithjackson.dev|192.168.2.21
TXT|a-auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
TXT|a-auth.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
TXT|a-crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
TXT|a-crafty-mc.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/crafty/crafty-l4-tcp
TXT|a-crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
TXT|a-crafty.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/crafty/crafty
TXT|a-gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
TXT|a-gitea.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/gitea/gitea-3000
TXT|a-grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
TXT|a-grafana.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/monitoring/grafana
TXT|a-headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
TXT|a-headlamp.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/headlamp/headlamp
TXT|a-home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
TXT|a-jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
TXT|a-jackett.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/jackett/jackett-9117
TXT|a-keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
TXT|a-keycloak.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/keycloak/keycloak
TXT|a-nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
TXT|a-nextcloud.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/nextcloud/nextcloud-80
TXT|a-pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
TXT|a-pihole.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/pihole/pihole-80
TXT|a-plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
TXT|a-plex.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/plex/plex-32400
TXT|a-prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/homepage/homepage
TXT|a-qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
TXT|a-qbittorrent.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
TXT|a-radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
TXT|a-radarr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/radarr/radarr-7878
TXT|a-seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
TXT|a-seerr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/seerr/seerr-5055
TXT|a-semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
TXT|a-semaphore.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/semaphore/semaphore-3000
TXT|a-sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
TXT|a-sonarr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/sonarr/sonarr-8989
TXT|a-unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
TXT|a-unifi.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/unifi/unifi-l4-tcp
TXT|auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
TXT|crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
TXT|crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
TXT|gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
TXT|grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
TXT|headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
TXT|home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
TXT|jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
TXT|keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
TXT|nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
TXT|pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
TXT|plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
TXT|qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
TXT|radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
TXT|seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
TXT|semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
TXT|sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
TXT|unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
+1284
View File
File diff suppressed because it is too large. Load diff
+1 -1
View File
@@ -102,7 +102,7 @@ spec:
userData: |-
#cloud-config
hostname: unifi
fqdn: unifi.home.keithjackson.dev
fqdn: unifi.${BASE_DOMAIN}
package_update: true
package_upgrade: false
packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl, jq, cloud-guest-utils]
+1 -1
View File
@@ -46,7 +46,7 @@ metadata:
lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
# external-dns (source: service) publishes A unifi.home.keithjackson.dev -> the pinned VIP.
# This is the browser + device target; UI is https://unifi.home.keithjackson.dev:11443.
external-dns.alpha.kubernetes.io/hostname: "unifi.home.keithjackson.dev"
external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
spec:
type: LoadBalancer
externalTrafficPolicy: Local
+25 -4
View File
@@ -90,9 +90,9 @@ spec:
name: gitea-bootstrap
key: secret-key
- name: GITEA__server__DOMAIN
value: gitea.home.keithjackson.dev
value: gitea.${BASE_DOMAIN}
- name: GITEA__server__ROOT_URL
value: https://gitea.home.keithjackson.dev/
value: https://gitea.${BASE_DOMAIN}/
- name: GITEA__actions__ENABLED
value: 'true'
- name: GITEA__actions__DEFAULT_ACTIONS_URL
@@ -127,7 +127,7 @@ spec:
command:
- /bin/sh
- -c
- "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n i=$((i + 1))\n [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n n=0\n until su-exec git gitea admin user create \\\n --username \"$GITEA_ADMIN_USERNAME\" \\\n --password \"$GITEA_ADMIN_PASSWORD\" \\\n --email \"[email protected]\" \\\n --admin --must-change-password=false; do\n n=$((n + 1))\n [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n sleep 3\n done\n echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not\n# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS TRUE\n# \u2014 making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on\n# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was\n# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.\n# Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator forms by design.\n# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.\n# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why the\n# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n d=0; disc_ok=0\n while [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1;\
- "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n i=$((i + 1))\n [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n n=0\n until su-exec git gitea admin user create \\\n --username \"$GITEA_ADMIN_USERNAME\" \\\n --password \"$GITEA_ADMIN_PASSWORD\" \\\n --email \"[email protected]\" \\\n --admin --must-change-password=false; do\n n=$((n + 1))\n [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n sleep 3\n done\n echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not\n# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS TRUE\n# \u2014 making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on\n# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was\n# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.\n# Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator forms by design.\n# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.\n# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why the\n# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n d=0; disc_ok=0\n while [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1;\
\ break\n fi\n d=$((d + 1)); sleep 2\n done\n [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n r=0; reg_ok=0\n while [ \"$r\" -lt 10 ]; do\n oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n if [ -n \"$oidc_id\" ]; then\n su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n else\n su-exec git gitea admin auth add-oauth \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n fi\n # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\n done\n if [ \"$reg_ok\" = 1 ]; then\n echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n else\n gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n fi\nfi\n# A postStart hook MUST exit 0 \u2014 a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff). This hook already ends 0 on both paths, but\n# it is made EXPLICIT so the guarantee does not depend on whatever statement happens to\n# be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,\n# the moment its OIDC guard was fixed and the block actually ran for the first time.\nexit 0\n"
volumes:
- name: data
@@ -170,6 +170,27 @@ spec:
- port: '3000'
protocol: TCP
---
# CiliumNetworkPolicy in-cluster client ingress allow (R-NET-02): gitea-allow-flux-ingress
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: gitea-allow-flux-ingress
namespace: gitea
labels:
app.kubernetes.io/name: gitea
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: gitea
ingress:
- fromEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: flux-system
toPorts:
- ports:
- port: '3000'
protocol: TCP
---
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): gitea-allow-internet-egress
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
@@ -207,7 +228,7 @@ spec:
namespace: gateway
sectionName: https
hostnames:
- gitea.home.keithjackson.dev
- gitea.${BASE_DOMAIN}
rules:
- matches:
- path:
+9 -8
View File
@@ -74,7 +74,7 @@ spec:
homelab.keithjackson.dev/auth-mode: forward-auth
annotations:
homelab.keithjackson.dev/auth-mode: forward-auth
homelab.keithjackson.dev/auth-groups: media-users
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
spec:
containers:
- name: jackett
@@ -83,7 +83,7 @@ spec:
- containerPort: 9117
volumeMounts:
- name: config
mountPath: /data/config
mountPath: /config
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
imagePullPolicy: IfNotPresent
@@ -91,16 +91,17 @@ spec:
- --http-address=0.0.0.0:4180
- --reverse-proxy=true
- --provider=oidc
- --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
- --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
- --upstream=http://127.0.0.1:9117
- --email-domain=*
- --scope=openid email profile
- --oidc-groups-claim=groups
- --allowed-group=media-users
- --allowed-group=platform-admins
- --skip-provider-button=true
- --redirect-url=https://jackett.home.keithjackson.dev/oauth2/callback
- --cookie-domain=.home.keithjackson.dev
- --whitelist-domain=.home.keithjackson.dev
- --redirect-url=https://jackett.${BASE_DOMAIN}/oauth2/callback
- --cookie-domain=.${BASE_DOMAIN}
- --whitelist-domain=.${BASE_DOMAIN}
- --cookie-secure=true
- --pass-access-token=true
- --set-xauthrequest=true
@@ -266,14 +267,14 @@ metadata:
app.kubernetes.io/name: jackett
annotations:
homelab.keithjackson.dev/auth-mode: forward-auth
homelab.keithjackson.dev/auth-groups: media-users
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
spec:
parentRefs:
- name: shared
namespace: gateway
sectionName: https
hostnames:
- jackett.home.keithjackson.dev
- jackett.${BASE_DOMAIN}
rules:
- matches:
- path:
+2
View File
@@ -11,6 +11,7 @@ resources:
- jackett-oauth2-proxy-credentials.sops.yaml
- jackett.yaml
- nextcloud-db-app.sops.yaml
- nextcloud-db-backup-credentials-prod.sops.yaml
- nextcloud-db-backup-credentials.sops.yaml
- nextcloud-db.yaml
- pihole-oauth2-proxy-credentials.sops.yaml
@@ -19,5 +20,6 @@ resources:
- qbittorrent-oauth2-proxy-credentials.sops.yaml
- qbittorrent.yaml
- semaphore-db-app.sops.yaml
- semaphore-db-backup-credentials-prod.sops.yaml
- semaphore-db-backup-credentials.sops.yaml
- semaphore-db.yaml
@@ -0,0 +1,29 @@
apiVersion: v1
kind: Secret
metadata:
name: nextcloud-db-backup-credentials-prod
namespace: nextcloud-db
type: Opaque
stringData:
ACCESS_KEY_ID: ENC[AES256_GCM,data:4lx6et4HcXDegulyIoa7jZR/lVgapGPEkukajfCKjvk=,iv:p6V7tizbikk55tOvpPOsPHTe/p52QLX5A5xrwuB14rA=,tag:oNJuipT/Lb6zynJ2K2Q71A==,type:str]
ACCESS_SECRET_KEY: ENC[AES256_GCM,data:V5bHPoPeBOVyLp1n1Jilyso4i5P2aPSH0hrryfweg2q/SctWZxRbLBQ/FLzOigbJsAhlwOGu069mQPma6xjfdw==,iv:8u4UOvI8dC/cm6mEWDlyN5E3SfRZmwKY9uZY/DjPEsQ=,tag:tw7tGs29TmUvMPBCtCszAA==,type:str]
sops:
kms: []
gcp_kms: []
azure_kv: []
hc_vault: []
age:
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBiSUt5N2k2N2lDU2ZsMy9m
M1NFS0M2QjFsNFhnR0hCMzFyaTdiQWFZRFJZCmhBTWRXNmxLMGtsc3Rxd2FSMDZp
aktjcTE2c1hmU1EzM3RWNksyb3Q1YVUKLS0tIDBFS1I5aDBER2cvcWtLZ3BXTEp5
ZGI5ZmZIcStheUNQamxIVVVpc0ZMOHMKlE5aHewtHJpeU24F5sodc9CEm9KTjHQQ
8ThEQ/8vyazL9V4ojlICxSHU/SHkHpD1eBaTKcAQd/LZzujIcJ+QfA==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-09-07T06:36:33Z"
mac: ENC[AES256_GCM,data:8awz1Jh8t4zjDMyd93/v+6on9iiIaSi01293A7uYhxj9L1+g63zeO+DPmSn6+KbZvHdqU2A7up09H1NzfXAEdGuvVbB36cdGq0zKank3PUi8c7IT6qNuWydxRAcqLnbbD+c30Z6GB+gAuZHstNOe4Rgc4aJvMN6FuP6x6Z0870c=,iv:KGGKJq2QA5xztvoHs/SViXW72DYwT16i+W+dJBUeIJQ=,tag:3UpYqx2+RIK0QdMH/BSIpA==,type:str]
pgp: []
encrypted_regex: ^(data|stringData)$
version: 3.9.4
+304 -14
View File
@@ -37,16 +37,53 @@ metadata:
name: nextcloud-db
namespace: nextcloud-db
spec:
# Single Postgres instance on 3-replica Longhorn is acceptable at home scale (ADR-0014, same as
# keycloak-db). The storage is already 3-way replicated by Longhorn; raise to 3 for Postgres-level
# HA as a future hardening.
instances: 1
# THREE instances — one primary + two hot standbys (2026-09-07), same as keycloak-db; see
# platform/keycloak/postgres-cluster.yaml for the full rationale. Short version: Longhorn's 3-way
# replication saves the DATA on node loss, not the SERVICE, and with `instances: 1` all three CNPG
# primaries had drifted onto k3s-prod-2, so one node took Nextcloud down alongside SSO and
# Semaphore. A streaming standby lets CNPG promote in seconds and repoint the -rw Service.
# Gated on the Barman archive below actually working (ContinuousArchiving=True with a real
# firstRecoverabilityPoint) — standbys replicate corruption, they are not a backup.
#
# Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by
# pg_basebackup from the running primary, which is neither restarted nor disconnected.
instances: 3
imageName: ghcr.io/cloudnative-pg/postgresql:16.4
# Stamp app.kubernetes.io/name=nextcloud-db onto every generated pod/service so the rendered
# nextcloud egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).
inheritedMetadata:
labels:
app.kubernetes.io/name: nextcloud-db
# HARD anti-affinity — the three instances MUST land on three DIFFERENT nodes; and roll the
# primary by SWITCHOVER rather than CNPG's default in-place `restart`, which would delete and
# recreate the primary pod. Full rationale in platform/keycloak/postgres-cluster.yaml. Short
# version: CNPG's default `preferred` is a scheduler hint, so when CNPG recreates an instance
# after a node loss it may stack two instances on one node — which is how `instances: 3` decays
# back into the single-node failure domain this change removes. `required` makes it a constraint;
# the cost is that an instance whose node is gone stays Pending, which is correct (2 of 3 still
# serve). Satisfiable on both clusters: prod and dev each have 3 schedulable, untainted nodes.
#
# APPLYING THIS CHANGES THE POD SPEC, so CNPG rolls the instances: standbys first, then one
# switchover. Expect a single brief connection reset for Nextcloud.
affinity:
enablePodAntiAffinity: true
podAntiAffinityType: required
primaryUpdateMethod: switchover
# SYNCHRONOUS REPLICATION — quorum of ONE standby, PREFERRED durability (2026-09-07). Full
# rationale in platform/keycloak/postgres-cluster.yaml. Short version: async replication loses the
# last few transactions when a primary dies before its standbys catch up, and `method: any` +
# `number: 1` makes COMMIT wait for one standby to confirm. dataDurability MUST stay `preferred`,
# never `required` — `required` blocks writes outright when fewer than `number` healthy standbys
# are available, which on a 3-node cluster sharing nodes with etcd and the CNI turns one degraded
# standby into a total write outage. `preferred` relaxes to async instead.
#
# NON-DISRUPTIVE: compiles to `synchronous_standby_names`, a reload-only PostgreSQL parameter —
# no restart, no switchover, no client disconnect. Land it after the anti-affinity roll.
postgresql:
synchronous:
method: any
number: 1
dataDurability: preferred
storage:
storageClass: longhorn
size: 5Gi
@@ -68,25 +105,38 @@ spec:
memory: 256Mi
limits:
memory: 512Mi
# --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → the on-Proxmox MinIO `cnpg` bucket ---
# --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → object storage ---
# Continuous WAL archiving + periodic base backups = proper Postgres PITR (realises the SD's
# backup.policy daily-offcluster). destinationPath is the bucket ROOT; CNPG appends serverName
# (defaults to the Cluster name), so objects land under s3://cnpg/nextcloud-db/. Path-style HTTP.
# Credentials: the scoped key from the in-namespace SOPS Secret nextcloud-db-backup-credentials
# (same-ns only), kept in sync with platform/backup/minio-credentials.sops.yaml.
# endpointURL is ${S3_BACKUP_ENDPOINT} (cluster-vars, R-ENV-02; dev http://192.168.1.9:9000),
# substituted by the `apps-wave-0` Flux Kustomization. Prod supplies its own off-cluster target.
# backup.policy daily-offcluster). CNPG appends serverName (defaults to the Cluster name) to
# destinationPath, so objects land under ${CNPG_BACKUP_PATH}/nextcloud-db/ — dev
# s3://cnpg/nextcloud-db/ on the on-Proxmox MinIO, prod
# s3://k3s-prod-backup/barman/nextcloud-db/ on Cloudflare R2. Both endpointURL and
# destinationPath are cluster-vars knobs (R-ENV-02), substituted by `apps-wave-0`.
# Credentials: PER-ENVIRONMENT (2026-09-07). dev reads nextcloud-db-backup-credentials (the
# scoped MinIO key), prod reads nextcloud-db-backup-credentials-prod (the Cloudflare R2
# token). BOTH Secrets are applied to BOTH clusters by apps/wave-0/kustomization.yaml (a
# generated golden — render.py places the -prod sidecar in this DB's wave); the
# cluster-vars knob ${BACKUP_CREDENTIALS_SUFFIX} (dev "" | prod "-prod") chooses which
# one this Cluster reads. Both are in-namespace, which is all a pod can read. They were
# ONE shared Secret until the split, so giving prod its R2 token would have left dev
# signing MinIO requests with an R2 key id — 403 on every archive command, i.e. the
# disk-full clock below rather than a merely missing backup.
#
# archive_mode is ON, so a FAILING archive command is a disk-full clock, not a degraded backup:
# Postgres refuses to recycle a WAL segment it has not archived, and the pgdata PVC fills until
# the primary PANICs. Prod demonstrated this from bootstrap to 2026-09-07 with a placeholder
# endpoint. Treat ContinuousArchiving=False here as an availability incident.
backup:
retentionPolicy: "30d"
barmanObjectStore:
destinationPath: s3://cnpg
destinationPath: ${CNPG_BACKUP_PATH}
endpointURL: ${S3_BACKUP_ENDPOINT}
s3Credentials:
accessKeyId:
name: nextcloud-db-backup-credentials
name: nextcloud-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
key: ACCESS_KEY_ID
secretAccessKey:
name: nextcloud-db-backup-credentials
name: nextcloud-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
key: ACCESS_SECRET_KEY
wal:
compression: gzip
@@ -108,3 +158,243 @@ spec:
cluster:
name: nextcloud-db
method: barmanObjectStore
---
# =============================================================================
# CNPG METRICS-EXPORTER CONNECT RECONCILER (R-OBS-01)
#
# THE BUG THIS CLOSES (true positive, verified live on prod 2026-09-12, and the
# condition platform/cnpg/alerts.yaml already recorded on BOTH clusters 2026-09-06):
# the `nextcloud` database's ACL had been narrowed out-of-band so that PUBLIC held
# nothing, and CNPG's metrics exporter role therefore could not even CONNECT:
#
# nextcloud-db-1/-2/-3, every scrape:
# FATAL: permission denied for database "nextcloud" (SQLSTATE 42501)
# DETAIL: User does not have CONNECT privilege.
# msg="Error collecting user query" targetDatabase=nextcloud
#
# CONSEQUENCE: `cnpg_last_error` = 1 on all three instances and the ENTIRE
# cnpg_pg_stat_* / pg_settings / pg_postmaster family MISSING for this database,
# while the scrape target still reported up=1 — a blind spot that looks like health.
# Measured before the fix: cnpg_pg_settings_setting had 765 series for keycloak and
# 765 for semaphore-db and ZERO for nextcloud-db. Nextcloud's database had never
# been monitored.
#
# ROOT CAUSE, AND WHY ONLY THIS DATABASE. All three Cluster specs are identical in
# their bootstrap; the difference is entirely in the live ACL:
#
# nextcloud : datacl = {nextcloud=CTc/nextcloud} <-- explicit, no PUBLIC entry
# keycloak : datacl = NULL
# semaphore : datacl = NULL
#
# A NULL datacl means "PostgreSQL defaults", which include CONNECT and TEMP for
# PUBLIC — so keycloak and semaphore let the exporter in only by inheriting PUBLIC's
# implicit CONNECT. On `nextcloud` something ran the equivalent of
# `REVOKE ALL ON DATABASE nextcloud FROM PUBLIC`, which MATERIALISES datacl and
# drops the implicit PUBLIC entry with it. That REVOKE is not in this repo and not
# in DECISIONS-LOG.md — it was applied out of band, which is exactly why a
# git-declared fix is needed rather than another hand-run psql.
#
# WHY THE GRANT IS EXPLICIT RATHER THAN "RE-GRANT CONNECT TO PUBLIC": restoring
# PUBLIC's CONNECT would undo whatever hardening that REVOKE intended and would
# leave the exporter depending on an implicit default again — the precise fragility
# that broke here and that keycloak/semaphore are still relying on. Naming the
# exporter role makes the dependency explicit and survives a future REVOKE FROM
# PUBLIC on any of the three databases.
#
# WHY A CronJob AND NOT A CNPG FIELD — CNPG 1.30.0's API GENUINELY CANNOT EXPRESS
# THIS. Checked against the live CRDs on prod:
# * `Database` (databases.postgresql.cnpg.io) has no database-ACL field at all.
# `allowConnections` is the pg_database.datallowconn FLAG (open/closed to
# everyone), not a per-role GRANT; its only grant/revoke verbs are the `usage`
# lists on `fdws` and `servers`.
# * `spec.managed.roles` and `DatabaseRole` manage ROLE ATTRIBUTES (login,
# superuser, connectionLimit, ...) plus `inRoles` MEMBERSHIP. Membership is the
# wrong tool: the only local role holding CONNECT here is the OWNER, so
# `inRoles: [nextcloud]` would hand the exporter the owner's CREATE and TEMP on
# the database and read/write on every application table — the opposite of
# least privilege, to buy one CONNECT.
# * `bootstrap.initdb.postInitApplicationSQL` runs ONCE, at bootstrap. It cannot
# repair these three already-bootstrapped instances (the actual bug), and on a
# FRESH cluster the grant is unnecessary anyway (datacl starts NULL). Adding it
# would be a second mechanism that fixes nothing this one does not.
# So the SQL has to be applied by something, and this repo already has the pattern
# for state Flux cannot reach: the level-triggered CronJob reconciler in
# platform/keycloak/client-secret-reconciler.yaml, whose reasoning transfers
# verbatim — the drifted state lives inside PostgreSQL, not in a Kubernetes object,
# so no admission or mutation policy can see it. A one-shot Job is deliberately NOT
# used: a completed Job is never re-run, so a DB rebuild or a PITR restore would
# silently reopen the blind spot. A loop converges instead.
#
# PRIVILEGE GRANTED — EXACTLY ONE, AND NOTHING ELSE:
# GRANT CONNECT ON DATABASE nextcloud TO cnpg_metrics_exporter;
# Resulting ACL entry is `cnpg_metrics_exporter=c/nextcloud` — the `c` is CONNECT;
# there is no `C` (CREATE) and no `T` (TEMPORARY). No ownership, no superuser, no
# write, no table or schema grants. Nothing further is needed because CNPG already
# creates this role as a member of the built-in `pg_monitor` role (verified on all
# three prod clusters), and pg_monitor already carries the read access every
# collector query wants — they read pg_catalog views (pg_stat_database,
# pg_stat_archiver, pg_settings, pg_postmaster) and no application tables. Proof
# that CONNECT alone is sufficient: after this grant, `Error collecting user query`
# went to ZERO on all three pods, i.e. every collector query now succeeds.
#
# IT RUNS AS THE DATABASE OWNER, NOT AS A SUPERUSER. `enableSuperuserAccess` is off
# on this Cluster (there is no nextcloud-db-superuser Secret), and it stays off: the
# owner of a database holds the grant option on that database's privileges, so
# `nextcloud` can issue this GRANT itself. Verified live before committing, by
# running the statement under `SET ROLE nextcloud`. Credentials come from the
# in-namespace SOPS Secret nextcloud-db-app that already exists for the app.
#
# NETWORKING: nextcloud-db is intentionally NOT an app-namespace (see this file's
# header), so Kyverno generates no default-deny baseline here and this same-namespace
# pod -> nextcloud-db-rw:5432 connection needs no rendered allow.
#
# THE POD LABEL IS DELIBERATELY *NOT* app.kubernetes.io/name=nextcloud-db. That
# label is what `inheritedMetadata` stamps on the CNPG instance pods and what the
# rendered nextcloud egress CiliumNetworkPolicy selects; reusing it here would quietly
# make this reconciler a permitted destination for the Nextcloud app. It gets its own
# name so that selector keeps meaning "the database".
apiVersion: batch/v1
kind: CronJob
metadata:
name: nextcloud-db-exporter-grant
namespace: nextcloud-db
labels:
app.kubernetes.io/name: nextcloud-db-exporter-grant
app.kubernetes.io/part-of: nextcloud-db
spec:
# Every 15 minutes, matching the keycloak reconciler's cadence. A converged
# database makes this one idempotent GRANT and exits; the cost is negligible, and
# after a rebuild or PITR restore monitoring is blind for at most one interval
# (well inside the alert's `for: 6h`).
schedule: "*/15 * * * *"
# A run still waiting on the primary must not be lapped by the next tick.
concurrencyPolicy: Forbid
startingDeadlineSeconds: 300
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 3
jobTemplate:
spec:
backoffLimit: 2
activeDeadlineSeconds: 300
ttlSecondsAfterFinished: 3600
template:
metadata:
labels:
app.kubernetes.io/name: nextcloud-db-exporter-grant
app.kubernetes.io/part-of: nextcloud-db
spec:
restartPolicy: Never
# Talks only to PostgreSQL; its one input is a mounted Secret, so it needs
# no Kubernetes API access at all (R-SEC-02, least privilege).
automountServiceAccountToken: false
securityContext:
runAsNonRoot: true
# uid/gid 26 is `postgres` in the CNPG PostgreSQL image.
runAsUser: 26
runAsGroup: 26
seccompProfile:
type: RuntimeDefault
containers:
- name: grant
# The SAME image the instances above run, pinned to the same tag. psql
# is already in it and it is already present on every node, so this
# adds no new image and no new supply-chain surface. Keep the tag in
# step with spec.imageName above when Renovate bumps it.
image: ghcr.io/cloudnative-pg/postgresql:16.4
imagePullPolicy: IfNotPresent
command:
- psql
# -w never prompts; -X skips ~/.psqlrc; ON_ERROR_STOP makes the Job
# FAIL LOUDLY rather than reporting success on a refused grant.
- "-w"
- "-X"
- "-v"
- "ON_ERROR_STOP=1"
- "-f"
- /etc/grant/grant.sql
env:
# libpq connection, assembled from the owner Secret. Target the -rw
# Service: GRANT is a write and must reach the primary.
- name: PGHOST
value: nextcloud-db-rw.nextcloud-db.svc.cluster.local
- name: PGPORT
value: "5432"
# Connecting to the application database itself is deliberate: the
# owner has CONNECT on it, and a successful connection is the same
# check the exporter was failing.
- name: PGDATABASE
value: nextcloud
- name: PGUSER
valueFrom:
secretKeyRef:
name: nextcloud-db-app
key: username
# Read by libpq from the environment and never logged; psql echoes
# no connection parameters.
- name: PGPASSWORD
valueFrom:
secretKeyRef:
name: nextcloud-db-app
key: password
- name: PGCONNECT_TIMEOUT
value: "10"
volumeMounts:
- name: sql
mountPath: /etc/grant
readOnly: true
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
resources:
requests:
cpu: 10m
memory: 32Mi
limits:
memory: 128Mi
volumes:
- name: sql
configMap:
name: nextcloud-db-exporter-grant
defaultMode: 0444
---
# The desired state, as SQL. Idempotent: re-granting an existing privilege is a
# no-op, so a converged run changes nothing. The DO block is the ASSERTION — it
# turns "the grant did not take" into a failed Job instead of a silent success.
apiVersion: v1
kind: ConfigMap
metadata:
name: nextcloud-db-exporter-grant
namespace: nextcloud-db
labels:
app.kubernetes.io/name: nextcloud-db-exporter-grant
app.kubernetes.io/part-of: nextcloud-db
data:
grant.sql: |
-- The ONE privilege the CNPG metrics exporter is missing on this database.
-- CONNECT only: no CREATE, no TEMPORARY, no ownership, no table/schema grants.
-- Read access for the collector's pg_catalog queries already comes from the
-- built-in pg_monitor role, which CNPG grants this role when it creates it.
GRANT CONNECT ON DATABASE nextcloud TO cnpg_metrics_exporter;
-- Fail the Job if the privilege is somehow still absent, so a broken
-- reconciler surfaces as a failed Job rather than as a quiet blind spot.
-- NO PL/pgSQL, AND NOT ONE DOLLAR SIGN IN THIS FILE — DELIBERATE, including in these
-- comments. This assertion began life as a dollar-quoted DO block, which cannot survive
-- being shipped this way: the ConfigMap is rendered through Flux's postBuild
-- substitution, and envsubst collapses a doubled dollar into a single one. The block
-- reached Postgres with its opening quote halved and every Flux-reconciled run died
-- with a syntax error pointing at that character.
-- It PASSED when hand-applied with kubectl, because that path runs no envsubst — so
-- the defect was invisible until Flux owned the object, and three runs of Error pods
-- were the first sign. Escaping by doubling again would work and would also be the
-- next person's trap; plain SQL leaves substitution nothing to touch.
--
-- has_database_privilege() returns boolean; ::int is 1 when granted and 0 when not, so
-- the division raises division_by_zero and ON_ERROR_STOP=1 fails the Job. The divisor
-- is not a constant, so it cannot be folded away at plan time.
\echo 'asserting cnpg_metrics_exporter has CONNECT on database nextcloud'
SELECT 1 / has_database_privilege('cnpg_metrics_exporter', 'nextcloud', 'CONNECT')::int
AS exporter_can_connect;
+5 -5
View File
@@ -91,16 +91,16 @@ spec:
- --http-address=0.0.0.0:4180
- --reverse-proxy=true
- --provider=oidc
- --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
- --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
- --upstream=http://127.0.0.1:80
- --email-domain=*
- --scope=openid email profile
- --oidc-groups-claim=groups
- --allowed-group=platform-admins
- --skip-provider-button=true
- --redirect-url=https://pihole.home.keithjackson.dev/oauth2/callback
- --cookie-domain=.home.keithjackson.dev
- --whitelist-domain=.home.keithjackson.dev
- --redirect-url=https://pihole.${BASE_DOMAIN}/oauth2/callback
- --cookie-domain=.${BASE_DOMAIN}
- --whitelist-domain=.${BASE_DOMAIN}
- --cookie-secure=true
- --pass-access-token=true
- --set-xauthrequest=true
@@ -229,7 +229,7 @@ spec:
namespace: gateway
sectionName: https
hostnames:
- pihole.home.keithjackson.dev
- pihole.${BASE_DOMAIN}
rules:
- matches:
- path:
+52 -1
View File
@@ -35,6 +35,34 @@ spec:
- protocol: TCP
port: 53
---
# PersistentVolume plex-media - static external NFS (bulk tier)
apiVersion: v1
kind: PersistentVolume
metadata:
name: plex-media
labels:
app.kubernetes.io/name: plex
spec:
capacity:
storage: 9Ti
accessModes:
- ReadOnlyMany
persistentVolumeReclaimPolicy: Retain
storageClassName: ''
mountOptions:
- nfsvers=4.1
- hard
- noatime
- timeo=600
- retrans=2
nfs:
server: ${BULK_NFS_SERVER}
path: /mnt/Storage/plex
readOnly: true
claimRef:
namespace: plex
name: plex-media
---
# PVC plex-config (R-STATE)
apiVersion: v1
kind: PersistentVolumeClaim
@@ -51,6 +79,23 @@ spec:
requests:
storage: 8Gi
---
# PVC plex-media (R-STATE)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: plex-media
namespace: plex
labels:
app.kubernetes.io/name: plex
spec:
accessModes:
- ReadOnlyMany
storageClassName: ''
volumeName: plex-media
resources:
requests:
storage: 9Ti
---
# Deployment (R-PLAT-02)
apiVersion: apps/v1
kind: Deployment
@@ -90,10 +135,16 @@ spec:
volumeMounts:
- name: config
mountPath: /config
- name: media
mountPath: /media
readOnly: true
volumes:
- name: config
persistentVolumeClaim:
claimName: plex-config
- name: media
persistentVolumeClaim:
claimName: plex-media
---
# Service
apiVersion: v1
@@ -183,7 +234,7 @@ spec:
namespace: gateway
sectionName: https
hostnames:
- plex.home.keithjackson.dev
- plex.${BASE_DOMAIN}
rules:
- matches:
- path:
+66 -8
View File
@@ -35,6 +35,33 @@ spec:
- protocol: TCP
port: 53
---
# PersistentVolume qbittorrent-downloads - static external NFS (bulk tier)
apiVersion: v1
kind: PersistentVolume
metadata:
name: qbittorrent-downloads
labels:
app.kubernetes.io/name: qbittorrent
spec:
capacity:
storage: 9Ti
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: ''
mountOptions:
- nfsvers=4.1
- hard
- noatime
- timeo=600
- retrans=2
nfs:
server: ${BULK_NFS_SERVER}
path: /mnt/Storage/plex
claimRef:
namespace: qbittorrent
name: qbittorrent-downloads
---
# PVC qbittorrent-config (R-STATE)
apiVersion: v1
kind: PersistentVolumeClaim
@@ -51,6 +78,23 @@ spec:
requests:
storage: 5Gi
---
# PVC qbittorrent-downloads (R-STATE)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: qbittorrent-downloads
namespace: qbittorrent
labels:
app.kubernetes.io/name: qbittorrent
spec:
accessModes:
- ReadWriteMany
storageClassName: ''
volumeName: qbittorrent-downloads
resources:
requests:
storage: 9Ti
---
# Deployment (R-PLAT-02)
apiVersion: apps/v1
kind: Deployment
@@ -74,16 +118,24 @@ spec:
homelab.keithjackson.dev/auth-mode: forward-auth
annotations:
homelab.keithjackson.dev/auth-mode: forward-auth
homelab.keithjackson.dev/auth-groups: media-users
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
spec:
containers:
- name: qbittorrent
image: lscr.io/linuxserver/qbittorrent:5.1.2
ports:
- containerPort: 8080
env:
- name: PUID
value: '1000'
- name: PGID
value: '3000'
volumeMounts:
- name: config
mountPath: /data/config
mountPath: /config
- name: downloads
mountPath: /downloads
subPath: downloads
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
imagePullPolicy: IfNotPresent
@@ -91,16 +143,17 @@ spec:
- --http-address=0.0.0.0:4180
- --reverse-proxy=true
- --provider=oidc
- --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
- --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
- --upstream=http://127.0.0.1:8080
- --email-domain=*
- --scope=openid email profile
- --oidc-groups-claim=groups
- --allowed-group=media-users
- --allowed-group=platform-admins
- --skip-provider-button=true
- --redirect-url=https://qbittorrent.home.keithjackson.dev/oauth2/callback
- --cookie-domain=.home.keithjackson.dev
- --whitelist-domain=.home.keithjackson.dev
- --redirect-url=https://qbittorrent.${BASE_DOMAIN}/oauth2/callback
- --cookie-domain=.${BASE_DOMAIN}
- --whitelist-domain=.${BASE_DOMAIN}
- --cookie-secure=true
- --pass-access-token=true
- --set-xauthrequest=true
@@ -148,10 +201,15 @@ spec:
port: http
initialDelaySeconds: 5
periodSeconds: 15
securityContext:
fsGroup: 3000
volumes:
- name: config
persistentVolumeClaim:
claimName: qbittorrent-config
- name: downloads
persistentVolumeClaim:
claimName: qbittorrent-downloads
---
# Service
apiVersion: v1
@@ -266,14 +324,14 @@ metadata:
app.kubernetes.io/name: qbittorrent
annotations:
homelab.keithjackson.dev/auth-mode: forward-auth
homelab.keithjackson.dev/auth-groups: media-users
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
spec:
parentRefs:
- name: shared
namespace: gateway
sectionName: https
hostnames:
- qbittorrent.home.keithjackson.dev
- qbittorrent.${BASE_DOMAIN}
rules:
- matches:
- path:
@@ -0,0 +1,29 @@
apiVersion: v1
kind: Secret
metadata:
name: semaphore-db-backup-credentials-prod
namespace: semaphore-db
type: Opaque
stringData:
ACCESS_KEY_ID: ENC[AES256_GCM,data:OhWEEYqrvmDB40NmiETgeUisBvaO9ocevCZAsurNijw=,iv:kEjOIg9NHbRYm+Ul1WrBAzb3gLbIv+ipiYCxFK0QoVc=,tag:KAB1JAIw0ATwbjDEKZVT9A==,type:str]
ACCESS_SECRET_KEY: ENC[AES256_GCM,data:C0PNKv07LCYpFONLqd55sz+ioyU/i2YJUkbZ8OX/dCErM/DilqchyUAT5EqnT/r3YJmdaA1pX85S3fxlVVHkbg==,iv:4ha28b+xIebvX2JPITmtTovIe4vdmj2XfI3cgkUHH4c=,tag:xbMyflcHojytPzgaoOYcXQ==,type:str]
sops:
kms: []
gcp_kms: []
azure_kv: []
hc_vault: []
age:
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZVTJ3cXR6cks3Y0RNS2hx
K3pjZEduYUxmL0V3SmFUcU9XbGVHSVFITWh3Cjl1eHlTLytoOW5PaVNFVkJoeTBo
T1FnRHNDVzJ6dWpuR09WaElvbWF4YU0KLS0tIEJoQ1VsM1cxRlg0R1dtNGdOenNn
aVNjMjVBTytLdHlFdmZCS2tPelo1Y2MKYt1G5t+sDiHgcg+Q9ECBzJn2gS/paSEC
7aoX60SBrPgHvnGls5lzc3Tin9nH4jWYl6Nu8DAvIE0YPXDFr5a0UA==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-09-07T06:36:33Z"
mac: ENC[AES256_GCM,data:mhVP7tjdZK2FR2imvpJbTyT7BigvcIKEpLSnBS1oSITUirB8xZC7aHBfyFuIoKOrpOJgEe40nHUdDYCHRJtb4dLeVja8+7meVaQWzxC8BWVHsksW+73hsqhjAzlHeobx6Irbk4PAMqIY0/jnf4MHryjnRn3dUuwu9U1CqL30JbE=,iv:57oCcp6FW9EAIVKnYGtkaTrnbijyv54T+vmZxMbMqnw=,tag:agrRP/JLypU60KOeN0s/5w==,type:str]
pgp: []
encrypted_regex: ^(data|stringData)$
version: 3.9.4
+64 -14
View File
@@ -38,16 +38,53 @@ metadata:
name: semaphore-db
namespace: semaphore-db
spec:
# Single Postgres instance on 3-replica Longhorn is acceptable at home scale (ADR-0014, same as
# keycloak-db / nextcloud-db). The storage is already 3-way replicated by Longhorn; raise to 3 for
# Postgres-level HA as a future hardening.
instances: 1
# THREE instances — one primary + two hot standbys (2026-09-07), same as keycloak-db /
# nextcloud-db; see platform/keycloak/postgres-cluster.yaml for the full rationale. Short version:
# Longhorn's 3-way replication saves the DATA on node loss, not the SERVICE, and with
# `instances: 1` all three CNPG primaries had drifted onto k3s-prod-2, so one node took Semaphore
# down alongside SSO and Nextcloud. A streaming standby lets CNPG promote in seconds and repoint
# the -rw Service. Gated on the Barman archive below actually working (ContinuousArchiving=True
# with a real firstRecoverabilityPoint) — standbys replicate corruption, they are not a backup.
#
# Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by
# pg_basebackup from the running primary, which is neither restarted nor disconnected.
instances: 3
imageName: ghcr.io/cloudnative-pg/postgresql:16.4
# Stamp app.kubernetes.io/name=semaphore-db onto every generated pod/service so the rendered
# semaphore egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).
inheritedMetadata:
labels:
app.kubernetes.io/name: semaphore-db
# HARD anti-affinity — the three instances MUST land on three DIFFERENT nodes; and roll the
# primary by SWITCHOVER rather than CNPG's default in-place `restart`, which would delete and
# recreate the primary pod. Full rationale in platform/keycloak/postgres-cluster.yaml. Short
# version: CNPG's default `preferred` is a scheduler hint, so when CNPG recreates an instance
# after a node loss it may stack two instances on one node — which is how `instances: 3` decays
# back into the single-node failure domain this change removes. `required` makes it a constraint;
# the cost is that an instance whose node is gone stays Pending, which is correct (2 of 3 still
# serve). Satisfiable on both clusters: prod and dev each have 3 schedulable, untainted nodes.
#
# APPLYING THIS CHANGES THE POD SPEC, so CNPG rolls the instances: standbys first, then one
# switchover. Expect a single brief connection reset for Semaphore.
affinity:
enablePodAntiAffinity: true
podAntiAffinityType: required
primaryUpdateMethod: switchover
# SYNCHRONOUS REPLICATION — quorum of ONE standby, PREFERRED durability (2026-09-07). Full
# rationale in platform/keycloak/postgres-cluster.yaml. Short version: async replication loses the
# last few transactions when a primary dies before its standbys catch up, and `method: any` +
# `number: 1` makes COMMIT wait for one standby to confirm. dataDurability MUST stay `preferred`,
# never `required` — `required` blocks writes outright when fewer than `number` healthy standbys
# are available, which on a 3-node cluster sharing nodes with etcd and the CNI turns one degraded
# standby into a total write outage. `preferred` relaxes to async instead.
#
# NON-DISRUPTIVE: compiles to `synchronous_standby_names`, a reload-only PostgreSQL parameter —
# no restart, no switchover, no client disconnect. Land it after the anti-affinity roll.
postgresql:
synchronous:
method: any
number: 1
dataDurability: preferred
storage:
storageClass: longhorn
size: 5Gi
@@ -69,25 +106,38 @@ spec:
memory: 256Mi
limits:
memory: 512Mi
# --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → the on-Proxmox MinIO `cnpg` bucket ---
# --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → object storage ---
# Continuous WAL archiving + periodic base backups = proper Postgres PITR (realises the SD's
# backup.policy daily-offcluster). destinationPath is the bucket ROOT; CNPG appends serverName
# (defaults to the Cluster name), so objects land under s3://cnpg/semaphore-db/. Path-style HTTP.
# Credentials: the scoped key from the in-namespace SOPS Secret semaphore-db-backup-credentials
# (same-ns only), kept in sync with platform/backup/minio-credentials.sops.yaml.
# endpointURL is ${S3_BACKUP_ENDPOINT} (cluster-vars, R-ENV-02; dev http://192.168.1.9:9000),
# substituted by the `apps-wave-0` Flux Kustomization. Prod supplies its own off-cluster target.
# backup.policy daily-offcluster). CNPG appends serverName (defaults to the Cluster name) to
# destinationPath, so objects land under ${CNPG_BACKUP_PATH}/semaphore-db/ — dev
# s3://cnpg/semaphore-db/ on the on-Proxmox MinIO, prod
# s3://k3s-prod-backup/barman/semaphore-db/ on Cloudflare R2. Both endpointURL and
# destinationPath are cluster-vars knobs (R-ENV-02), substituted by `apps-wave-0`.
# Credentials: PER-ENVIRONMENT (2026-09-07). dev reads semaphore-db-backup-credentials (the
# scoped MinIO key), prod reads semaphore-db-backup-credentials-prod (the Cloudflare R2
# token). BOTH Secrets are applied to BOTH clusters by apps/wave-0/kustomization.yaml (a
# generated golden — render.py places the -prod sidecar in this DB's wave); the
# cluster-vars knob ${BACKUP_CREDENTIALS_SUFFIX} (dev "" | prod "-prod") chooses which
# one this Cluster reads. Both are in-namespace, which is all a pod can read. They were
# ONE shared Secret until the split, so giving prod its R2 token would have left dev
# signing MinIO requests with an R2 key id — 403 on every archive command, i.e. the
# disk-full clock below rather than a merely missing backup.
#
# archive_mode is ON, so a FAILING archive command is a disk-full clock, not a degraded backup:
# Postgres refuses to recycle a WAL segment it has not archived, and the pgdata PVC fills until
# the primary PANICs. Prod demonstrated this from bootstrap to 2026-09-07 with a placeholder
# endpoint. Treat ContinuousArchiving=False here as an availability incident.
backup:
retentionPolicy: "30d"
barmanObjectStore:
destinationPath: s3://cnpg
destinationPath: ${CNPG_BACKUP_PATH}
endpointURL: ${S3_BACKUP_ENDPOINT}
s3Credentials:
accessKeyId:
name: semaphore-db-backup-credentials
name: semaphore-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
key: ACCESS_KEY_ID
secretAccessKey:
name: semaphore-db-backup-credentials
name: semaphore-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
key: ACCESS_SECRET_KEY
wal:
compression: gzip
+159
View File
@@ -0,0 +1,159 @@
---
# Namespace (trust zone; default-deny + policies apply)
apiVersion: v1
kind: Namespace
metadata:
name: vm-demo
labels:
homelab.keithjackson.dev/app-namespace: 'true'
annotations:
homelab.keithjackson.dev/backup-policy: daily-offcluster
---
# Default-deny baseline (R-NET-01): deny all, allow only DNS
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: vm-demo
labels:
homelab.keithjackson.dev/baseline: default-deny
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress: []
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
---
# CDI DataVolume (standalone; survives rebuild, R-STATE-01)
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
name: vm-demo-rootdisk
namespace: vm-demo
spec:
source:
registry:
url: docker://quay.io/containerdisks/fedora:41
storage:
accessModes:
- ReadWriteOnce
storageClassName: longhorn
resources:
requests:
storage: 6Gi
---
# KubeVirt VirtualMachine (R-PLAT-03)
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: vm-demo
namespace: vm-demo
labels:
app.kubernetes.io/name: vm-demo
spec:
running: false
template:
metadata:
labels:
kubevirt.io/vm: vm-demo
homelab.keithjackson.dev/egress-class: internet
homelab.keithjackson.dev/auth-mode: none
spec:
domain:
cpu:
cores: 1
memory:
guest: 1536Mi
devices:
disks:
- name: rootdisk
disk:
bus: virtio
- name: cloudinit
disk:
bus: virtio
interfaces:
- name: default
masquerade: {}
resources:
requests:
memory: 1536Mi
networks:
- name: default
pod: {}
volumes:
- name: rootdisk
dataVolume:
name: vm-demo-rootdisk
- name: cloudinit
cloudInitNoCloud:
userData: "#cloud-config\npackage_update: true\npackages:\n - dnf-automatic\nwrite_files:\n - path: /etc/dnf/automatic.conf\n content: |\n [commands]\n upgrade_type = default\n random_sleep = 0\n download_updates = yes\n apply_updates = yes\nruncmd:\n - [ systemctl, enable, --now, dnf-automatic.timer ]\n"
---
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): vm-demo-allow-internet-egress
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: vm-demo-allow-internet-egress
namespace: vm-demo
labels:
app.kubernetes.io/name: vm-demo
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: vm-demo
egress:
- toEntities:
- world
---
# ServiceMonitor (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: vm-demo
namespace: vm-demo
labels:
release: kube-prometheus-stack
app.kubernetes.io/name: vm-demo
spec:
selector:
matchLabels:
app.kubernetes.io/name: vm-demo
namespaceSelector:
matchNames:
- vm-demo
endpoints:
- port: metrics
interval: 30s
---
# PrometheusRule SLI alerts (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: vm-demo-slis
namespace: vm-demo
labels:
release: kube-prometheus-stack
app.kubernetes.io/name: vm-demo
spec:
groups:
- name: vm-demo.slis
rules:
- alert: vm-demo-down
expr: up{job="vm-demo"} == 0
for: 5m
labels:
severity: critical
service: vm-demo
annotations:
summary: vm-demo is down
description: 'vm-demo has been unreachable for 5m (SLI: up).'
+1 -1
View File
@@ -165,7 +165,7 @@ spec:
exec /sbin/tini -- /opt/act/run.sh
env:
- name: GITEA_INSTANCE_URL
value: https://gitea.home.keithjackson.dev
value: https://gitea.${BASE_DOMAIN}
- name: GITEA_RUNNER_REGISTRATION_TOKEN
valueFrom:
secretKeyRef:
+228
View File
@@ -0,0 +1,228 @@
---
# Namespace (trust zone; default-deny + policies apply)
apiVersion: v1
kind: Namespace
metadata:
name: keycloak
labels:
homelab.keithjackson.dev/app-namespace: 'true'
gateway-access: shared
annotations:
homelab.keithjackson.dev/backup-policy: daily-offcluster
---
# Default-deny baseline (R-NET-01): deny all, allow only DNS
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: keycloak
labels:
homelab.keithjackson.dev/baseline: default-deny
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress: []
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
---
# PVC keycloak-db (R-STATE)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: keycloak-db
namespace: keycloak
labels:
app.kubernetes.io/name: keycloak
spec:
accessModes:
- ReadWriteOnce
storageClassName: longhorn
resources:
requests:
storage: 10Gi
---
# Deployment (R-PLAT-02)
apiVersion: apps/v1
kind: Deployment
metadata:
name: keycloak
namespace: keycloak
labels:
app.kubernetes.io/name: keycloak
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app.kubernetes.io/name: keycloak
template:
metadata:
labels:
app.kubernetes.io/name: keycloak
homelab.keithjackson.dev/egress-class: internet
homelab.keithjackson.dev/auth-mode: native
spec:
containers:
- name: keycloak
image: quay.io/keycloak/keycloak
ports:
- containerPort: 8080
volumeMounts:
- name: db
mountPath: /data/db
volumes:
- name: db
persistentVolumeClaim:
claimName: keycloak-db
---
# Service
apiVersion: v1
kind: Service
metadata:
name: keycloak
namespace: keycloak
labels:
app.kubernetes.io/name: keycloak
spec:
selector:
app.kubernetes.io/name: keycloak
ports:
- name: port-8080
port: 8080
targetPort: 8080
---
# CiliumNetworkPolicy egress allow (R-NET-02): keycloak-allow-keycloak-db-5432
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: keycloak-allow-keycloak-db-5432
namespace: keycloak
labels:
app.kubernetes.io/name: keycloak
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: keycloak
egress:
- toEndpoints:
- matchLabels:
app.kubernetes.io/name: keycloak-db
k8s:io.kubernetes.pod.namespace: keycloak-db
toPorts:
- ports:
- port: '5432'
protocol: TCP
---
# CiliumNetworkPolicy gateway-ingress allow (R-NET-01/02): keycloak-allow-gateway-ingress
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: keycloak-allow-gateway-ingress
namespace: keycloak
labels:
app.kubernetes.io/name: keycloak
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: keycloak
ingress:
- fromEntities:
- ingress
toPorts:
- ports:
- port: '8080'
protocol: TCP
---
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): keycloak-allow-internet-egress
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: keycloak-allow-internet-egress
namespace: keycloak
labels:
app.kubernetes.io/name: keycloak
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: keycloak
egress:
- toEntities:
- world
---
# HTTPRoute keycloak-8080 (R-TLS/R-DNS)
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: keycloak-8080
namespace: keycloak
labels:
app.kubernetes.io/name: keycloak
spec:
parentRefs:
- name: shared
namespace: gateway
sectionName: https
hostnames:
- keycloak.${BASE_DOMAIN}
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: keycloak
port: 8080
---
# ServiceMonitor (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: keycloak
namespace: keycloak
labels:
release: kube-prometheus-stack
app.kubernetes.io/name: keycloak
spec:
selector:
matchLabels:
app.kubernetes.io/name: keycloak
namespaceSelector:
matchNames:
- keycloak
endpoints:
- port: metrics
interval: 30s
---
# PrometheusRule SLI alerts (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: keycloak-slis
namespace: keycloak
labels:
release: kube-prometheus-stack
app.kubernetes.io/name: keycloak
spec:
groups:
- name: keycloak.slis
rules:
- alert: keycloak-down
expr: up{job="keycloak"} == 0
for: 5m
labels:
severity: critical
service: keycloak
annotations:
summary: keycloak is down
description: 'keycloak has been unreachable for 5m (SLI: up).'
+59 -7
View File
@@ -35,6 +35,34 @@ spec:
- protocol: TCP
port: 53
---
# PersistentVolume nextcloud-legacy-archive - static external NFS (bulk tier)
apiVersion: v1
kind: PersistentVolume
metadata:
name: nextcloud-legacy-archive
labels:
app.kubernetes.io/name: nextcloud
spec:
capacity:
storage: 270Gi
accessModes:
- ReadOnlyMany
persistentVolumeReclaimPolicy: Retain
storageClassName: ''
mountOptions:
- nfsvers=4.1
- hard
- noatime
- timeo=600
- retrans=2
nfs:
server: ${BULK_NFS_SERVER}
path: /mnt/Storage/nextcloud
readOnly: true
claimRef:
namespace: nextcloud
name: nextcloud-legacy-archive
---
# PVC nextcloud-data (R-STATE)
apiVersion: v1
kind: PersistentVolumeClaim
@@ -51,6 +79,23 @@ spec:
requests:
storage: 20Gi
---
# PVC nextcloud-legacy-archive (R-STATE)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nextcloud-legacy-archive
namespace: nextcloud
labels:
app.kubernetes.io/name: nextcloud
spec:
accessModes:
- ReadOnlyMany
storageClassName: ''
volumeName: nextcloud-legacy-archive
resources:
requests:
storage: 270Gi
---
# Deployment (R-PLAT-02)
apiVersion: apps/v1
kind: Deployment
@@ -104,13 +149,13 @@ spec:
name: nextcloud-bootstrap
key: admin-password
- name: NEXTCLOUD_TRUSTED_DOMAINS
value: nextcloud.home.keithjackson.dev
value: nextcloud.${BASE_DOMAIN}
- name: OVERWRITEHOST
value: nextcloud.home.keithjackson.dev
value: nextcloud.${BASE_DOMAIN}
- name: OVERWRITEPROTOCOL
value: https
- name: OVERWRITECLIURL
value: https://nextcloud.home.keithjackson.dev
value: https://nextcloud.${BASE_DOMAIN}
- name: NEXTCLOUD_OIDC_CLIENT_SECRET
valueFrom:
secretKeyRef:
@@ -119,19 +164,26 @@ spec:
volumeMounts:
- name: data
mountPath: /var/www/html
- name: legacy-archive
mountPath: /mnt/legacy-nextcloud
subPath: nextcloud
readOnly: true
lifecycle:
postStart:
exec:
command:
- /bin/sh
- -c
- "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n# maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n# stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n i=$((i + 1))\n [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is\n# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS\n# TRUE \u2014 so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never\n# installing/enabling user_oidc nor registering the provider. SSO kept working only because\n# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with\n# NO OIDC at all. Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator\n# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion\n# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why\n# `--clientsecret-env` and the bare uses below always worked and only the guard broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"\
[nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)\"\n exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1; break\n fi\n d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n# HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n# single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n# form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n# occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n# (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n --clientid=nextcloud \\\n --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n --scope='openid email profile' \\\n --unique-uid=0 \\\n --mapping-uid=preferred_username \\\n --mapping-email=email \\\n --mapping-display-name=name \\\n --group-provisioning=1 \\\n --mapping-groups=groups\" >/dev/null 2>&1 || true\n if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n# NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n# login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n# built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n# start: occ group:adduser is a no-op for an existing member. A user is promotable only\n# AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n# rights on their next login + the following pod start (or run `occ group:adduser admin\n# <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n# passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n current_admins=$(parse_group admin)\n missing=0\n for uid in $(parse_group \"\
$ADMIN_SRC_GROUP\"); do\n if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n continue\n fi\n missing=1\n if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n fi\n done\n [ \"$missing\" = 0 ] && break\n p=$((p + 1)); sleep 2\ndone\nif [ \"$p\" -ge 15 ]; then\n nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\nfi\n# A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:\n# the final statement used to be `[ \"$p\" -ge 15 ] && nextcloud_oidc_fail ...`, so on the\n# SUCCESS path the test returned 1 and became the script's exit status. That was latent for\n# as long as the OIDC guard above was dead code and exited 0 first; fixing the guard\n# exposed it immediately. Every failure path here logs loudly and writes a marker file and\n# deliberately does NOT crash the pod \u2014 the break-glass local admin must stay usable.\nexit 0\n"
- "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n# maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n# stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n i=$((i + 1))\n [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is\n# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS\n# TRUE \u2014 so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never\n# installing/enabling user_oidc nor registering the provider. SSO kept working only because\n# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with\n# NO OIDC at all. Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator\n# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion\n# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why\n# `--clientsecret-env` and the bare uses below always worked and only the guard broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[nextcloud-oidc]\
\ client secret unset; skipping OIDC (break-glass admin only)\"\n exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1; break\n fi\n d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n# HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n# single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n# form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n# occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n# (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n --clientid=nextcloud \\\n --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n --scope='openid email profile' \\\n --unique-uid=0 \\\n --mapping-uid=preferred_username \\\n --mapping-email=email \\\n --mapping-display-name=name \\\n --group-provisioning=1 \\\n --mapping-groups=groups\" >/dev/null 2>&1 || true\n if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n# NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n# login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n# built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n# start: occ group:adduser is a no-op for an existing member. A user is promotable only\n# AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n# rights on their next login + the following pod start (or run `occ group:adduser admin\n# <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n# passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n current_admins=$(parse_group admin)\n missing=0\n for uid in $(parse_group \"$ADMIN_SRC_GROUP\"\
); do\n if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n continue\n fi\n missing=1\n if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n fi\n done\n [ \"$missing\" = 0 ] && break\n p=$((p + 1)); sleep 2\ndone\nif [ \"$p\" -ge 15 ]; then\n nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\nfi\n# A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:\n# the final statement used to be `[ \"$p\" -ge 15 ] && nextcloud_oidc_fail ...`, so on the\n# SUCCESS path the test returned 1 and became the script's exit status. That was latent for\n# as long as the OIDC guard above was dead code and exited 0 first; fixing the guard\n# exposed it immediately. Every failure path here logs loudly and writes a marker file and\n# deliberately does NOT crash the pod \u2014 the break-glass local admin must stay usable.\nexit 0\n"
volumes:
- name: data
persistentVolumeClaim:
claimName: nextcloud-data
- name: legacy-archive
persistentVolumeClaim:
claimName: nextcloud-legacy-archive
---
# Service
apiVersion: v1
@@ -228,7 +280,7 @@ spec:
namespace: gateway
sectionName: https
hostnames:
- nextcloud.home.keithjackson.dev
- nextcloud.${BASE_DOMAIN}
rules:
- matches:
- path:
+67 -8
View File
@@ -35,6 +35,33 @@ spec:
- protocol: TCP
port: 53
---
# PersistentVolume radarr-media - static external NFS (bulk tier)
apiVersion: v1
kind: PersistentVolume
metadata:
name: radarr-media
labels:
app.kubernetes.io/name: radarr
spec:
capacity:
storage: 9Ti
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: ''
mountOptions:
- nfsvers=4.1
- hard
- noatime
- timeo=600
- retrans=2
nfs:
server: ${BULK_NFS_SERVER}
path: /mnt/Storage/plex
claimRef:
namespace: radarr
name: radarr-media
---
# PVC radarr-config (R-STATE)
apiVersion: v1
kind: PersistentVolumeClaim
@@ -51,6 +78,23 @@ spec:
requests:
storage: 5Gi
---
# PVC radarr-media (R-STATE)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: radarr-media
namespace: radarr
labels:
app.kubernetes.io/name: radarr
spec:
accessModes:
- ReadWriteMany
storageClassName: ''
volumeName: radarr-media
resources:
requests:
storage: 9Ti
---
# Deployment (R-PLAT-02)
apiVersion: apps/v1
kind: Deployment
@@ -74,16 +118,25 @@ spec:
homelab.keithjackson.dev/auth-mode: forward-auth
annotations:
homelab.keithjackson.dev/auth-mode: forward-auth
homelab.keithjackson.dev/auth-groups: media-users
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
spec:
containers:
- name: radarr
image: lscr.io/linuxserver/radarr:5.21.1
ports:
- containerPort: 7878
env:
- name: PUID
value: '1000'
- name: PGID
value: '3000'
- name: TZ
value: Etc/UTC
volumeMounts:
- name: config
mountPath: /data/config
mountPath: /config
- name: media
mountPath: /media
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
imagePullPolicy: IfNotPresent
@@ -91,16 +144,17 @@ spec:
- --http-address=0.0.0.0:4180
- --reverse-proxy=true
- --provider=oidc
- --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
- --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
- --upstream=http://127.0.0.1:7878
- --email-domain=*
- --scope=openid email profile
- --oidc-groups-claim=groups
- --allowed-group=media-users
- --allowed-group=platform-admins
- --skip-provider-button=true
- --redirect-url=https://radarr.home.keithjackson.dev/oauth2/callback
- --cookie-domain=.home.keithjackson.dev
- --whitelist-domain=.home.keithjackson.dev
- --redirect-url=https://radarr.${BASE_DOMAIN}/oauth2/callback
- --cookie-domain=.${BASE_DOMAIN}
- --whitelist-domain=.${BASE_DOMAIN}
- --cookie-secure=true
- --pass-access-token=true
- --set-xauthrequest=true
@@ -148,10 +202,15 @@ spec:
port: http
initialDelaySeconds: 5
periodSeconds: 15
securityContext:
fsGroup: 3000
volumes:
- name: config
persistentVolumeClaim:
claimName: radarr-config
- name: media
persistentVolumeClaim:
claimName: radarr-media
---
# Service
apiVersion: v1
@@ -288,14 +347,14 @@ metadata:
app.kubernetes.io/name: radarr
annotations:
homelab.keithjackson.dev/auth-mode: forward-auth
homelab.keithjackson.dev/auth-groups: media-users
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
spec:
parentRefs:
- name: shared
namespace: gateway
sectionName: https
hostnames:
- radarr.home.keithjackson.dev
- radarr.${BASE_DOMAIN}
rules:
- matches:
- path:
@@ -5,6 +5,15 @@
# credential from its own namespace (a Pod cannot read a Secret in another namespace).
# access-key-encryption: FIXED base64 32-byte AES key Semaphore uses to encrypt stored SSH keys /
# credentials; MUST stay stable across restarts or stored secrets become undecryptable.
# oidc-client-secret: the Keycloak `semaphore` CONFIDENTIAL client secret, and ONLY that. It MIRRORS
# `semaphore-client-secret` in platform/keycloak/keycloak-realm-clients.sops.yaml (the realm import
# injects that copy into the client), same cross-namespace mirror pattern as db-password above.
# It replaced the former `oidc-providers-json` key (2026-09-07): that key held the WHOLE
# SEMAPHORE_OIDC_PROVIDERS blob, which buried the environment-specific realm-issuer and redirect
# URLs inside ciphertext where Flux's ${BASE_DOMAIN} postBuild substitution could not reach them,
# so prod ran with dev's URLs. Those URLs are PUBLIC and now live in plaintext in
# catalog/semaphore.yaml (rendered to apps/wave-1/semaphore.yaml); the kubelet splices THIS value
# back into the blob via $(SEMAPHORE_OIDC_CLIENT_SECRET). Only the secret half is encrypted.
# Recover: infra/devcontainer/dev.sh run sops -d apps/semaphore-bootstrap-secret.sops.yaml
apiVersion: v1
kind: Secret
@@ -15,11 +24,11 @@ metadata:
app.kubernetes.io/name: semaphore
type: Opaque
stringData:
admin-username: ENC[AES256_GCM,data:uD/vO9A9MjerFg==,iv:i/NAqVD6rWg9oWWSGBrRkxzwcPp2IW1HLIKY0pdcPPg=,tag:r27tKLAJ+ydE7Ty3ccMNUw==,type:str]
admin-password: ENC[AES256_GCM,data:544nWY1UmBnlsdPJAQZJMXRumu4W9X7UXZvZ6vHN/ArjRgLXKTArOe5JkT/nNrNw,iv:AC1DDbZj2HfpU19EjzA7T+WpIaR4nZc3PNN61y8bAzU=,tag:r28LFi5w6ZeCCaRaX2jRdg==,type:str]
db-password: ENC[AES256_GCM,data:ySkdvaGRCW9fMQqChs5l8dY4zCG19i4HOEinRC48mgLuh9WOrjd16T6jwFgiwB55,iv:+1gztJ22HbwgQJiolZMMWOcKIsxrBJpnruJdtxdooOU=,tag:8CfJGwyPTjDGyr0CvalXMQ==,type:str]
access-key-encryption: ENC[AES256_GCM,data:1YAMLo1nVkpJWZc+KBZ7B4ia+Y8P+xMyOhHMcEGGJOXd3o+WC5FqirLBVh4=,iv:j28ozeaDhGWDjoaxFOhn2t9YbYE7L3ah/Mfi+i2DWwY=,tag:czG9Wxc/7Nsuh1W5iATOww==,type:str]
oidc-providers-json: ENC[AES256_GCM,data:90yt5mIVWLVI2KrlJWpxhL/BMDpEgRhv7+BL+mBRt79diRvU7pN9vtzhZprAKzcU1sEywlT4EwFlsvJpYyo5Qbkp+ZVknbyhAlGvHhJOBHukOPWgf4LignLerO2GzhXjOOzFqWfZ4EyUxNMWTbSe/xc0zIVHDDNQiYplGVRK+E/ukiI7sD/Tn2RrbrdvpC5GyK0NXWRsXn9ffhE3fCoiRHXagj9i/q+uUrQI5EKpPobrDUAGqvKoBXTl/wJ4phnWg9IqtCTE/c3yA7fLLz4j5dCtPZNSVPT4TKDx8Mdv8CfCgziVicjG2s+yHuWpsQn8lf6nvdJmpYPvIqTtLWdxFWuD8Vfo6Qbq2mqPbKR+KmDOGUnjl4gbZdL1SkE9JiqBUobz7UXeLXwYzxMIsinOfYZLcCk7alrD5Yndj9JDxDJZQ54sMMBfVbxDl94W1VHy03RxAR46KKAbPaxOP5wH4Q4z3SDceNwmuGCDRGChRXiHhN5X2X0GqYCbc8E6GdkIMcpmD80e26ydEmAjw7F89mCF4F/yMzJNiG82qaZIn8HeqgGA8urous9r9bYJNlitYTYQ,iv:jbfVdrrZ9fGiOxhQeoTJfFg1QwiOMkoQyaadiVfxX/w=,tag:JkepcRs34EuEgNb4buTTDQ==,type:str]
admin-username: ENC[AES256_GCM,data:ju3wVdjNzRUP/w==,iv:d/5ZxgQZ5bympu3gT2ck3DJy9iUPPBAtZYsJvD3To7I=,tag:0UtBaDVqNSK4RCJrQicchw==,type:str]
admin-password: ENC[AES256_GCM,data:pxIwpgh3BoND2TI0mPTixOTJoictzT77dqtuRdMKgY/2WaW4wM1Ra6H6n/dxnyR2,iv:eYbFQGFYVJsnYGwn6cOI3BgAdzMcin9YggAhiuL+xqM=,tag:/sPWGv4gue0vjc2nMk7wWA==,type:str]
db-password: ENC[AES256_GCM,data:hBFpWXdpk1R5a23Pntup03t56NBRPS+SZU8CrHVb/ypIaIpgf7kjYtVy6nhTF6u0,iv:eBzoQgH5ZhWWpfKZ6AQmwaSal5QaiuN9gfexQ7GsVFs=,tag:j2k8EwhKDlj/Ud6O39ioVQ==,type:str]
access-key-encryption: ENC[AES256_GCM,data:SKtAV4HxJ9pNYDqUK/1uOdI1rtMji3pTfprI2rhQxFQyhc6dFlhqI+OUEcY=,iv:rKCP+FLidcOltOwmMMcO5M8tz4qiifIMI+W5soGO2M8=,tag:HaHzkX/JI+9wmcg2SD9GAQ==,type:str]
oidc-client-secret: ENC[AES256_GCM,data:uCo7TNzbM2QKw9ie+xzyUG9MFKco1OAlwr5qPn/AGpNvo1YyXSfbZyyM0CLJ1NS821hGg5Zr/m3R/xpy84s3FQ==,iv:O8a4haskufpmorPaxOkJ1Q99Y5GTlzmDFJAsHS+irLk=,tag:VojJpA9jNMbNQmIe1Qr44w==,type:str]
sops:
kms: []
gcp_kms: []
@@ -29,14 +38,14 @@ sops:
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBuRDlqRFJ0dk14UHluSGRP
S1pESy9PWlpMakU0aTkzNVhqUGdtbWNEdTJjCm5RNG14azJHc28rdEJ4L0RzR0xO
MGRocGZ1V1JZTENWQkN3VGZzamp2eTgKLS0tIGt6QkNNaW5hdTBJVHY0SVo5d3JX
elp0SWxHcjg0RWF3aW9mWTJHZ0lhcUEKc0rU5VilAVDleCwqq4SOF+Y9sniu1Pka
w/MGDA9THSB0b+YgWRWiX/KiimFqp0ZFODOJ9cHVwFGY71+VadPwOg==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAyTzF6NTBtcFVWK1Vtc0I2
VGdySHBsRGgvc1h0VzgvQ01xYVBJV3g2Q1NRCjJyZHVYZGtuamo1MFJHWlBwWkxt
cHA4VkY1TkRpN3M4UTdMOWZTbFVGemMKLS0tIGVNdDBrWVJveU56VjVBU2JWdHF4
RGtaYStXYUpONTZjZCtmM0NyZ3Rka1UKIQpmQai8dz+uRAsXxyeuGaRlRwsFqIPQ
9AfrZJcslYwlaK3ptm1oqaCdrlxcMar2nI7thSqzbLe+e1nzA2z7vQ==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-08-27T22:12:32Z"
mac: ENC[AES256_GCM,data:nHftJTGnTr0O5h8yGXwgz3eflggXJfIXM3m8n3bzDJE/P4nvSpZzFeMb+sc2on+AL+t+SeKrIl8blzm33XsQ1lkaY85Zm+bc5cnED1g04BMDVmmK3brh/8w50Bp2wPEVzb4I01MDzhhyIbOYg7pFvZAH3/VuqvM1t6XmfF1NrOA=,iv:4KBa9PH/dfdGwLC16RenZIVB6BtSoGo6x3qt3h1R43I=,tag:KULiA1klP/RZMNigkloGDA==,type:str]
lastmodified: "2026-09-07T07:01:50Z"
mac: ENC[AES256_GCM,data:Xutm1rQt57J4RILElD8FkNOsQT1HK5jZlPtqqLyZ/VqOoDwo+IS1wp7hROprlUvQC0bUXy79tRRIl5Q0CCOD+fvAGIdmade+vB7qXlYN6Q1441dGhuSTPvckso6FaB2ddRjKhKGLc/Z4bJoz9VaRNNxJNpWXIrENTg+I+QhNhlo=,iv:klJOrEQ6QN5dTGkQfGnbMbgCWsgWwZPb5YXDMeWeE34=,tag:fuJr4t3Tt3dT9u3/0SFo8Q==,type:str]
pgp: []
encrypted_regex: ^(data|stringData)$
version: 3.9.4
+6 -4
View File
@@ -110,7 +110,7 @@ spec:
- name: SEMAPHORE_ADMIN_NAME
value: Semaphore Admin
- name: SEMAPHORE_ADMIN_EMAIL
value: admin@home.keithjackson.dev
value: admin@${BASE_DOMAIN}
- name: SEMAPHORE_ACCESS_KEY_ENCRYPTION
valueFrom:
secretKeyRef:
@@ -118,11 +118,13 @@ spec:
key: access-key-encryption
- name: SEMAPHORE_PORT
value: :3000
- name: SEMAPHORE_OIDC_PROVIDERS
- name: SEMAPHORE_OIDC_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: semaphore-bootstrap
key: oidc-providers-json
key: oidc-client-secret
- name: SEMAPHORE_OIDC_PROVIDERS
value: '{"keycloak":{"display_name":"Keycloak","provider_url":"https://keycloak.${BASE_DOMAIN}/realms/homelab","client_id":"semaphore","client_secret":"$(SEMAPHORE_OIDC_CLIENT_SECRET)","redirect_url":"https://semaphore.${BASE_DOMAIN}/api/auth/oidc/keycloak/redirect","scopes":["openid","profile","email"],"username_claim":"preferred_username","email_claim":"email","name_claim":"name","order":1}}'
volumeMounts:
- name: config
mountPath: /etc/semaphore
@@ -231,7 +233,7 @@ spec:
namespace: gateway
sectionName: https
hostnames:
- semaphore.home.keithjackson.dev
- semaphore.${BASE_DOMAIN}
rules:
- matches:
- path:
+67 -8
View File
@@ -35,6 +35,33 @@ spec:
- protocol: TCP
port: 53
---
# PersistentVolume sonarr-media - static external NFS (bulk tier)
apiVersion: v1
kind: PersistentVolume
metadata:
name: sonarr-media
labels:
app.kubernetes.io/name: sonarr
spec:
capacity:
storage: 9Ti
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: ''
mountOptions:
- nfsvers=4.1
- hard
- noatime
- timeo=600
- retrans=2
nfs:
server: ${BULK_NFS_SERVER}
path: /mnt/Storage/plex
claimRef:
namespace: sonarr
name: sonarr-media
---
# PVC sonarr-config (R-STATE)
apiVersion: v1
kind: PersistentVolumeClaim
@@ -51,6 +78,23 @@ spec:
requests:
storage: 5Gi
---
# PVC sonarr-media (R-STATE)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: sonarr-media
namespace: sonarr
labels:
app.kubernetes.io/name: sonarr
spec:
accessModes:
- ReadWriteMany
storageClassName: ''
volumeName: sonarr-media
resources:
requests:
storage: 9Ti
---
# Deployment (R-PLAT-02)
apiVersion: apps/v1
kind: Deployment
@@ -74,16 +118,25 @@ spec:
homelab.keithjackson.dev/auth-mode: forward-auth
annotations:
homelab.keithjackson.dev/auth-mode: forward-auth
homelab.keithjackson.dev/auth-groups: media-users
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
spec:
containers:
- name: sonarr
image: lscr.io/linuxserver/sonarr:4.0.15
ports:
- containerPort: 8989
env:
- name: PUID
value: '1000'
- name: PGID
value: '3000'
- name: TZ
value: Etc/UTC
volumeMounts:
- name: config
mountPath: /data/config
mountPath: /config
- name: media
mountPath: /media
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
imagePullPolicy: IfNotPresent
@@ -91,16 +144,17 @@ spec:
- --http-address=0.0.0.0:4180
- --reverse-proxy=true
- --provider=oidc
- --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
- --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
- --upstream=http://127.0.0.1:8989
- --email-domain=*
- --scope=openid email profile
- --oidc-groups-claim=groups
- --allowed-group=media-users
- --allowed-group=platform-admins
- --skip-provider-button=true
- --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
- --cookie-domain=.home.keithjackson.dev
- --whitelist-domain=.home.keithjackson.dev
- --redirect-url=https://sonarr.${BASE_DOMAIN}/oauth2/callback
- --cookie-domain=.${BASE_DOMAIN}
- --whitelist-domain=.${BASE_DOMAIN}
- --cookie-secure=true
- --pass-access-token=true
- --set-xauthrequest=true
@@ -148,10 +202,15 @@ spec:
port: http
initialDelaySeconds: 5
periodSeconds: 15
securityContext:
fsGroup: 3000
volumes:
- name: config
persistentVolumeClaim:
claimName: sonarr-config
- name: media
persistentVolumeClaim:
claimName: sonarr-media
---
# Service
apiVersion: v1
@@ -288,14 +347,14 @@ metadata:
app.kubernetes.io/name: sonarr
annotations:
homelab.keithjackson.dev/auth-mode: forward-auth
homelab.keithjackson.dev/auth-groups: media-users
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
spec:
parentRefs:
- name: shared
namespace: gateway
sectionName: https
hostnames:
- sonarr.home.keithjackson.dev
- sonarr.${BASE_DOMAIN}
rules:
- matches:
- path:
+1 -1
View File
@@ -223,7 +223,7 @@ spec:
namespace: gateway
sectionName: https
hostnames:
- seerr.home.keithjackson.dev
- seerr.${BASE_DOMAIN}
rules:
- matches:
- path:
+11 -5
View File
@@ -1,6 +1,9 @@
#!/usr/bin/env bash
#
# bootstrap.sh — the SINGLE Flux + SOPS bootstrap entrypoint for the dev cluster.
# bootstrap.sh — the SINGLE Flux + SOPS bootstrap entrypoint. ENVIRONMENT-AGNOSTIC: which overlay
# it applies is FLUX_ENV_PATH (default clusters/dev). roles/flux_bootstrap exports that from
# `flux_bootstrap_env_path`, overridden to clusters/prod in group_vars/k3s_prod/vars.yml — so the
# same script bootstraps dev and prod with no fork (R-ENV-02).
#
# Phase 1, T-1.4 · ADR-0004 (Flux) · ADR-0007 (SOPS+age) · R-GITOPS-01 · R-SEC-01.
# =============================================================================
@@ -33,7 +36,7 @@
# 2. sops-age Secret — the operator installs the age PRIVATE key out-of-band BEFORE reconcile so
# decryption works on first pass (../sops/age-key-secret.md). This script
# only VERIFIES it exists; it never handles key material.
# 3. `kubectl apply -k clusters/dev` — materialises the authored reconciliation graph:
# 3. `kubectl apply -k ${FLUX_ENV_PATH}` — materialises the authored reconciliation graph:
# GitRepository `flux-system` (source) -> Kustomization `platform` ->
# `platform-config` -> the app-wave Kustomizations `apps-wave-0/1/2`
# (chained dependsOn) + `apps-vms`, with cluster-vars overlay
@@ -52,7 +55,10 @@
set -euo pipefail
# --- config (env-overridable) ------------------------------------------------
FLUX_ENV_PATH="${FLUX_ENV_PATH:-clusters/dev}" # dev is the ONLY Phase-1 build target
# Which environment overlay to apply. Dev is the DEFAULT (it was the only Phase-1 build target);
# prod passes clusters/prod via roles/flux_bootstrap. Changing the default would silently retarget
# every existing caller — override it, never edit it.
FLUX_ENV_PATH="${FLUX_ENV_PATH:-clusters/dev}"
FLUX_NAMESPACE="${FLUX_NAMESPACE:-flux-system}"
SOPS_SECRET_NAME="${SOPS_SECRET_NAME:-sops-age}"
# The two Phase-4 image-automation controllers are installed alongside the default set so their CRDs
@@ -62,7 +68,7 @@ FLUX_COMPONENTS_EXTRA="${FLUX_COMPONENTS_EXTRA:-image-reflector-controller,image
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd "${script_dir}/../.." && pwd))"
echo "==> Flux + SOPS bootstrap (dev) — path=${FLUX_ENV_PATH}"
echo "==> Flux + SOPS bootstrap — path=${FLUX_ENV_PATH}"
# --- 0. tooling present (from infra/devcontainer; versions pinned there) ------
for bin in flux kubectl; do
@@ -134,7 +140,7 @@ if ! kubectl -n "${FLUX_NAMESPACE}" get secret "${SOPS_SECRET_NAME}" >/dev/null
exit 1
fi
# --- 3. apply the authored dev entrypoint ------------------------------------
# --- 3. apply the authored environment entrypoint ----------------------------
echo "==> [3/3] kubectl apply -k ${FLUX_ENV_PATH}"
kubectl apply -k "${repo_root}/${FLUX_ENV_PATH}"
+51 -10
View File
@@ -1,11 +1,30 @@
#!/usr/bin/env bash
# bootstrap/gitea/bootstrap-gitea.sh — run a SINGLE Gitea container on the HOST (outside the
# cluster) as the Phase-1 GitOps bootstrap source. While the cluster is built, Flux (in-cluster)
# reconciles this repo from the host Gitea over the k3s-dev NAT gateway. This is the "local
# bootstrap source" that migrates to in-cluster Gitea in Phase 6 (docs/05 "Source-of-truth",
# CONTRIBUTING.md).
# cluster) as the GitOps bootstrap source. While a cluster is built, Flux (in-cluster) reconciles
# this repo from the host Gitea. This is the "local bootstrap source" that migrates to in-cluster
# Gitea at cutover (docs/05 "Source-of-truth", CONTRIBUTING.md). Dev has already cut over; PROD is
# using it now.
#
# Run by the OPERATOR at bring-up — NOT during author-only, NOT by any agent.
# HOW THE CLUSTER REACHES THE HOST — read this before setting HOST_NAT_IP.
# It is an ADDRESS OF THIS HOST as seen from the cluster nodes, and it differs per environment:
#
# PROD (bare metal, current use): HOST_NAT_IP=192.168.2.132
# The prod nodes (192.168.2.10/.11/.12) and this workstation are on the SAME LAN. There is
# NO NAT hop of any kind, and NO firewall change was needed — verified 2026-09-04 from all
# three nodes. The firewalld note further down is a DEV-ERA libvirt concern; it does not
# apply to prod, so do not "fix" prod by opening zones.
#
# DEV (libvirt VMs, HISTORICAL): HOST_NAT_IP=10.73.0.1 was the k3s-dev libvirt NAT gateway.
# *** THAT NETWORK IS RETIRED. 10.73.0.1 is stale and must not be reused. *** Dev's own
# bootstrap Gitea was later reached on 192.168.2.103:3000 and is retained STOPPED as
# recovery (see the comment on the GitRepository patch in clusters/dev/kustomization.yaml).
#
# WHO RUNS THIS: the OPERATOR, or an agent the operator has EXPLICITLY DIRECTED to stand up a
# bootstrap source for a specific bring-up (as happened 2026-09-04 for prod). It is NOT part of
# author-only work and must never be fired off casually or "to see if it works": it starts a
# long-lived container, generates and writes admin credentials to $HOME, creates an org and repo,
# and PUSHES THE CURRENT HEAD to that repo's `main` — which is the revision a cluster pointed at
# it will then reconcile. Know which HEAD you are publishing before you run it.
set -euo pipefail
ENGINE="${ENGINE:-podman}"
GITEA_IMAGE="${GITEA_IMAGE:-docker.io/gitea/gitea:1.22}" # pinned; bump if needed
@@ -16,11 +35,24 @@ SSH_PORT="${GITEA_SSH_PORT:-2222}"
# subuid that can't write a host-owned dir, so Gitea fails to create /data/gitea/conf/app.ini.
DATA_VOLUME="${GITEA_DATA_VOLUME:-gitea-bootstrap-data}"
DATA_DIR="podman volume ${DATA_VOLUME}" # display label only (data lives in the named volume)
# The cluster VMs reach the host on the k3s-dev NAT gateway (10.73.0.1). ROOT_URL must match so
# The address the CLUSTER uses to reach this host (see the header). ROOT_URL must match it so
# clone/redirect URLs are correct from inside the cluster.
# NOTE: firewalld's `libvirt` zone (owns virbrN) blocks VM->host:3000 by default. Open it once:
#
# The DEFAULT IS DELIBERATELY UNSET — there is no correct default any more. The old default
# (10.73.0.1) was the retired k3s-dev libvirt NAT gateway; leaving it in place would silently
# publish a ROOT_URL no cluster can resolve, and Flux would report what looks like a network fault.
# Pass it explicitly, e.g. for prod: HOST_NAT_IP=192.168.2.132 bash bootstrap-gitea.sh
#
# NOTE (DEV/libvirt ONLY, historical): firewalld's `libvirt` zone (owns virbrN) blocked
# VM->host:3000 by default and had to be opened once with
# sudo firewall-cmd --zone=libvirt --add-port=3000/tcp --permanent && sudo firewall-cmd --reload
HOST_NAT_IP="${HOST_NAT_IP:-10.73.0.1}"
# PROD needs NO firewall change: same-LAN, no libvirt zone involved (verified 2026-09-04).
if [ -z "${HOST_NAT_IP:-}" ]; then
echo "ERROR: HOST_NAT_IP is not set — it is this host's address AS SEEN FROM THE CLUSTER." >&2
echo " prod: HOST_NAT_IP=192.168.2.132 (same LAN as the nodes, no NAT, no firewall change)" >&2
echo " There is no safe default: the old one (10.73.0.1) is the RETIRED libvirt NAT." >&2
exit 1
fi
ORG="${GITEA_ORG:-Home-Network}"
REPO="${GITEA_REPO:-home-network-k3s-impl}"
@@ -75,8 +107,17 @@ cat <<EOF
Gitea bootstrap source ready (${DATA_DIR}).
UI (host): ${BASE}/${ORG}/${REPO}
Flux GitRepository (cluster): http://${HOST_NAT_IP}:${HTTP_PORT}/${ORG}/${REPO}.git (clusters/dev)
Flux GitRepository (cluster): http://${HOST_NAT_IP}:${HTTP_PORT}/${ORG}/${REPO}.git
(patch this url into the overlay being bootstrapped:
clusters/prod today; clusters/dev has already cut over)
Admin creds (host, 0600): ${ADMIN_CRED_FILE}
Prerequisite: firewalld must allow VM->host:${HTTP_PORT} —
Reachability: prod nodes reach ${HOST_NAT_IP}:${HTTP_PORT} directly on the LAN — no NAT, no
firewall change (verified 2026-09-04). For libvirt-based clusters ONLY, firewalld's libvirt zone
must allow VM->host:${HTTP_PORT} —
ansible-playbook infra/ansible/playbooks/host-prereqs.yml --ask-become-pass
REMINDER: the push above published the CURRENT HEAD as this repo's \`main\`. Whatever cluster points
at this source will reconcile THAT revision.
TEMPORARY BY DESIGN: this container is outside every cluster and is a single point of failure for
any cluster reconciling from it. Cut over to in-cluster Gitea (docs/05); for prod the cutover
procedure is written on the GitRepository patch in clusters/prod/kustomization.yaml.
EOF
+33 -5
View File
@@ -24,9 +24,9 @@ env:
name: gitea-bootstrap
key: secret-key
- name: GITEA__server__DOMAIN # correct public clone/redirect URLs for the TLS-terminated host
value: gitea.home.keithjackson.dev
value: gitea.${BASE_DOMAIN}
- name: GITEA__server__ROOT_URL
value: https://gitea.home.keithjackson.dev/
value: https://gitea.${BASE_DOMAIN}/
- name: GITEA__actions__ENABLED # Phase 6 CI/CD: declared (not implicit upstream default) so the
value: "true" # in-cluster act_runner (ns gitea-actions) registers against it
- name: GITEA__actions__DEFAULT_ACTIONS_URL # resolve `uses: <org>/<action>@ref` from github.com (cluster
@@ -104,7 +104,7 @@ lifecycle:
# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker
# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays
# usable). The verify-then-retry loop is the real guarantee against "exit 0 but nothing stuck".
GITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
GITEA_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
gitea_oidc_fail() {
echo "[gitea-oidc] FAILED to register 'keycloak' login source: $1" >&2
{ echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true
@@ -168,7 +168,7 @@ lifecycle:
# be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,
# the moment its OIDC guard was fixed and the block actually ran for the first time.
exit 0
exposure: public # edge-forwarded git host at gitea.home.keithjackson.dev (docs/09; R-DNS/R-NET)
exposure: public # edge-forwarded git host at gitea.${BASE_DOMAIN} (docs/09; R-DNS/R-NET)
egress: internet # normal WAN egress (docs/03; R-NET-05)
storage:
- name: data
@@ -178,8 +178,36 @@ storage:
# the default /data/<name> would strand real state on the ephemeral rootfs (R-STATE-01)
dependencies: [] # default-deny: reaches only DNS; git clients reach it north-south via the Gateway (R-NET-01/02)
ingressRoutes:
- host: gitea.home.keithjackson.dev
- host: gitea.${BASE_DOMAIN}
port: 3000 # Gitea HTTP web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
# THIS IS WHAT KEEPS PROD'S GITOPS OUT OF A DEADLOCK — it is not an incidental allowance. Remove it
# and prod cannot reconcile AT ALL, in a way that is very hard to diagnose from cold.
#
# Prod's Flux source is THIS SERVICE, not a public hostname:
# clusters/prod/kustomization.yaml -> GitRepository.spec.url =
# http://gitea.gitea.svc.cluster.local:3000/Home-Network/home-network-k3s-impl.git
# chosen deliberately so prod's GitOps depends on NEITHER public DNS, Cloudflare, prod's own
# Gateway, NOR ${BASE_DOMAIN} — a future domain change cannot sever prod from its own source. That
# file documents the other half of this pair at length; the two must be read together.
#
# The only other ingress this entry renders is gitea-allow-gateway-ingress, which admits
# `fromEntities: [ingress]` — the shared Gateway's Envoy proxy — and nothing else. Flux's
# source-controller dials the ClusterIP DIRECTLY, so under the namespace default-deny (R-NET-01) its
# clone was DROPPED (`wget` from source-controller timed out; srcReady=False, "unable to list
# remote"). The deadlock that made this urgent: the HTTPRoute that would have served the new public
# name after the 2026-09-13 apex flip lives in a DOWNSTREAM tier, which cannot apply while the
# source is down — source down -> routes never rename -> source stays down. It was broken on
# 2026-09-13 with a live `kubectl apply` of a hand-written gitea-allow-flux-ingress, i.e. with a
# cluster object that existed OUTSIDE git. This declaration renders that object, so one
# `kubectl delete` or one cluster rebuild can no longer put prod back in the deadlock.
inClusterClients:
# `name` is pinned to `flux` (rather than defaulting to the namespace) so the rendered policy name
# equals the live object's, `gitea-allow-flux-ingress`, and Flux ADOPTS it in place: no
# near-duplicate policy beside it, and no window in which prod's own source has no allow.
- name: flux
namespace: flux-system # matched by NAMESPACE alone — the flux controllers' pod labels belong
# to upstream's manifests, not to this platform's contract
port: 3000 # ONLY the Gitea HTTP port — the same single port the Gateway allow opens
auth:
mode: oidc # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC" set; R-IDP-02)
groups:
+22 -2
View File
@@ -17,20 +17,40 @@ name: jackett
kind: container
source:
image: lscr.io/linuxserver/jackett:latest
exposure: internal # LAN-only web UI at jackett.home.keithjackson.dev (docs/13 row 4; R-DNS/R-NET)
exposure: internal # LAN-only web UI at jackett.${BASE_DOMAIN} (docs/13 row 4; R-DNS/R-NET)
egress: vpn # indexer scraping via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
storage:
- name: config
size: 2Gi
class: longhorn # indexer definitions/API keys on 3-replica Longhorn (docs/04; R-STATE-02)
mountPath: /config # THE APP WRITES /config, NOT the renderer default /data/config.
# LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
# /config), so without this the Longhorn PVC mounted at /data/config sat
# holding nothing but lost+found while the app wrote its entire
# configuration to the container's EPHEMERAL writable layer — destroyed on
# every pod recreation: restart, reschedule, image bump. Measured in-cluster
# 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
# dev=2053 on the node's own disk with [Sentry asp config.xml logs].
# This is the SAME failure catalog/plex.yaml:54 already records fixing.
# The live /config contents were copied onto each PVC before this landed,
# so API keys and credentials other apps reference were preserved.
dependencies: [] # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
ingressRoutes:
- host: jackett.home.keithjackson.dev
- host: jackett.${BASE_DOMAIN}
port: 9117 # Jackett WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
auth:
mode: forward-auth # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)
groups:
- media-users # authenticated AND in an authorized group (R-IDP-06)
# platform-admins is a SUPERSET, not a widening. --allowed-group is repeatable and OR-ed, so
# listing both means "media consumers OR platform operators". Without it the media UIs are
# reachable by NOBODY the moment media-users is empty: on prod the realm's only user is the
# operator, who is in platform-admins only, so jackett/qbittorrent/radarr/sonarr answered an
# immediate 403 at the oauth2-proxy callback (no login prompt, because the Keycloak SSO session
# already existed) while pihole/crafty — which already gate on platform-admins — worked.
# This grants no real privilege: platform-admins already maps to cluster-admin via
# k8s-oidc-rbac.yaml (oidc:platform-admins), so a member can already exec into these pods.
- platform-admins # operators of these services must not be locked out of them
update:
channel: stable # R-UPD-01
range: ">=0.22.0" # stay on the Jackett 0.22.x stable line
+1 -1
View File
@@ -18,7 +18,7 @@ dependencies:
port: 5432
protocol: tcp
ingressRoutes:
- host: keycloak.home.keithjackson.dev
- host: keycloak.${BASE_DOMAIN}
port: 8080 # HTTP listener; TLS terminates at the Gateway (docs/08, docs/09)
auth:
mode: native # it IS the IdP; own admin login is local + MFA (docs/15; ADR-0014). Documented exception.
+43 -5
View File
@@ -50,13 +50,13 @@ env:
# the vhost answer for the public FQDN; OVERWRITE* make Nextcloud emit https:// self-URLs even
# though it speaks plain HTTP behind the proxy. ---
- name: NEXTCLOUD_TRUSTED_DOMAINS
value: nextcloud.home.keithjackson.dev
value: nextcloud.${BASE_DOMAIN}
- name: OVERWRITEHOST
value: nextcloud.home.keithjackson.dev
value: nextcloud.${BASE_DOMAIN}
- name: OVERWRITEPROTOCOL
value: https
- name: OVERWRITECLIURL
value: https://nextcloud.home.keithjackson.dev
value: https://nextcloud.${BASE_DOMAIN}
# --- Keycloak native OIDC SSO (auth.mode oidc; docs/15 "Native OIDC"; R-IDP-02/06). The confidential
# `nextcloud` realm client's secret is mirrored into this namespace's nextcloud-bootstrap Secret
# (oidc-client-secret; same value SOPS-held in keycloak-realm-clients). The postStart hook reads it
@@ -88,7 +88,7 @@ lifecycle:
# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as
# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.
OCC() { su -s /bin/sh www-data -c "php /var/www/html/occ $*"; }
NEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
NEXTCLOUD_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
nextcloud_oidc_fail() {
echo "[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1" >&2
{ echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true
@@ -223,12 +223,50 @@ storage:
mountPath: /var/www/html # 3-replica tier; irreplaceable, DR-critical (docs/04; R-STATE-02).
# The whole html dir is persisted so config.php + user data survive a
# rebuild (the default /data/<name> would strand them on ephemeral rootfs).
# THE LEGACY NEXTCLOUD DATASET (Storage/nextcloud, 270 G) — MOUNTED READ-ONLY, DELIBERATELY, AND
# NOT AS THIS INSTANCE'S DATA DIRECTORY. Read this before changing either decision.
#
# WHAT IS ACTUALLY IN THERE (measured 2026-09-07, not assumed):
# /mnt/Storage/nextcloud/nextcloud/config 103 M
# /mnt/Storage/nextcloud/nextcloud/config_old 3.4 G
# /mnt/Storage/nextcloud/nextcloud/config_backup 267 G <- 99% of the dataset
# There is NO `data/` directory. This is the OLD, non-k3s Nextcloud's configuration plus a backup
# blob — an ARCHIVE, not a live user-data tree.
#
# WHY READ-ONLY. Two independent reasons, either sufficient:
# 1. The old instance is STILL LIVE on it. The export is currently shared rw to 192.168.1.6 and
# 192.168.1.5, and the operator has said the existing non-k3s prod must remain active. Two
# Nextcloud instances writing one instance's config directory, each with its own database and
# therefore its own oc_filecache, is a corruption scenario with no clean recovery.
# 2. Nothing here needs writing. An archive is read.
# "Do not grant more than needed" therefore lands on `ro`. If the operator's intent is in fact a
# MIGRATION (read this data in, then decommission the old instance), that is a different and
# larger change than a mount, and it should be a deliberate one — flip `readOnly` and update the
# export at that point, not before. Enforced by the PV's nfs.readOnly, so it is a real `ro` mount.
#
# WHY NOT /var/www/html/data. That path is the LONGHORN PVC above and is this instance's own
# state; mounting a foreign tree into it would have Nextcloud scan and index an archive it does
# not own. A neutral path keeps the two completely separate. To expose it to users, add it as a
# Nextcloud External Storage (Local) mount pointing at this path — an in-app, per-user decision.
#
# subPath skips the redundant single-child directory: the dataset root holds exactly one entry,
# `nextcloud/`, so mounting that subtree puts config/ config_old/ config_backup/ at the top.
#
# UID: nothing to do. The tree is 0755 owned 1000:3000 and the mount is read-only, so www-data
# (uid 33) can read and traverse it. (fsGroup would be a no-op on an NFS volume in any case.)
- name: legacy-archive
class: bulk
export: /mnt/Storage/nextcloud # the ZFS dataset Storage/nextcloud, on ${BULK_NFS_SERVER}
subPath: nextcloud
mountPath: /mnt/legacy-nextcloud
readOnly: true
size: 270Gi # ADVISORY ONLY on a static NFS PV — the dataset's used size.
dependencies:
- to: nextcloud-db # explicit allow: Nextcloud -> its PostgreSQL (docs/13 graph; R-NET-02).
port: 5432 # Renders the egress half; the DB namespace mirrors keycloak-db (no
protocol: tcp # default-deny), so Cilium's default-allow accepts the reciprocal ingress.
ingressRoutes:
- host: nextcloud.home.keithjackson.dev
- host: nextcloud.${BASE_DOMAIN}
port: 80 # apache HTTP; wildcard TLS terminates at the shared Gateway (docs/08/09)
auth:
mode: oidc # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC"; R-IDP-02)
+1 -1
View File
@@ -14,7 +14,7 @@ storage:
class: longhorn # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03)
dependencies: [] # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02)
ingressRoutes:
- host: pihole.home.keithjackson.dev
- host: pihole.${BASE_DOMAIN}
port: 80 # admin web UI, protected at the Gateway by forward-auth (docs/09)
auth:
mode: forward-auth # oauth2-proxy -> Keycloak; bare UI, no native OIDC (docs/15; R-IDP-02)
+31 -16
View File
@@ -4,21 +4,11 @@
# auth: native (Plex accounts, NOT Keycloak — docs/13 + docs/15 documented exception).
# Validates against schemas/servicedefinition.schema.json.
#
# BULK MEDIA (spec bulk-tier requirement, docs/04) — OPEN PREREQUISITE, intentionally NOT declared
# here yet (the requirement ID is cited in the WS-APPS report, not as a token here: this change does
# NOT exercise the bulk tier, so it must not be selected as a "touched" requirement whose in-cluster
# conformance test is still a Phase-1 gap).
# The Plex library belongs on the `bulk` tier (docs/04 / docs/13 cross-cutting note: metadata DB
# on replicated Longhorn, large library on bulk). The `bulk` StorageClass exists
# (platform/longhorn/storageclasses.yaml) and the schema+renderer both support `class: bulk`, but
# that class is a documented STUB: OQ-2 (external NFS/SMB backing) is UNRESOLVED and the NFS CSI
# driver is not installed, so a `bulk` PVC will not bind and would leave the Plex pod Pending.
# To onboard the running Plex shell now (compute + config + public exposure), the media volume is
# omitted. Once OQ-2 is resolved and `bulk` provisions, attach the library with a one-line add:
# storage:
# - name: media
# class: bulk # external mount; size omitted (sized outside the cluster)
# then re-render. See the WS-APPS report for the full bulk-media finding.
# BULK MEDIA — NOW ATTACHED (docs/04 bulk tier; the long-standing OQ-2 prerequisite, resolved for
# the STATIC case). The library is 8.33 T of ZFS on the Proxmox host at /mnt/Storage/plex, and it
# already exists: nothing is provisioned, an existing dataset is mounted. See
# docs/BULK-NFS-STORAGE.md for the layout, the export change and the uid analysis. (The bulk-tier
# requirement ID is deliberately still not typed here — see that document's "gate" note.)
name: plex
kind: container
source:
@@ -70,9 +60,34 @@ storage:
# UI sign-in, the operator adds a PLEX_CLAIM env from a fresh https://plex.tv/claim token
# (4-min TTL) — deliberately NOT wired here: the env secretKeyRef pattern has no `optional`,
# so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
# THE LIBRARY (docs/04 bulk tier). One READ-ONLY mount of the export ROOT, giving /media/movies,
# /media/tv and /media/photos — the whole tree the layout actually has (`ls /mnt/Storage/plex`:
# downloads/ movies/ photos/ tv/).
#
# WHY THE ROOT AND NOT THREE subPath MOUNTS. Nothing is gained by narrowing: the mount is `ro`,
# so no subPath could remove a privilege that matters, while three mounts would triple the NFS
# mounts per pod and mean a manifest change every time a top-level library is added.
#
# WHY READ-ONLY, when the export is `rw` and the other consumers write. Plex is a SERVER: it
# indexes and streams. Every write on that tree belongs to Sonarr and Radarr. Marking it here
# costs nothing and removes Plex from the set of things that can damage 8.33 T. Enforced by the
# PV's nfs.readOnly (a real `ro` mount), not only by the volumeMount flag.
#
# UID: no change needed. The tree is 0755 owned 1000:3000 throughout, so it is world-readable and
# world-traversable, and PLEX_UID/PLEX_GID above already make the process uid 1000 in any case.
# (fsGroup would have been a no-op here regardless — the in-tree NFS plugin does no ownership
# management. That is the trap the arrs' PUID/PGID exists to avoid; Plex never hits it.)
- name: media
class: bulk
export: /mnt/Storage/plex # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER}
mountPath: /media
readOnly: true
size: 9Ti # ADVISORY ONLY (nothing enforces a static NFS PV's capacity): the
# dataset's 8.33 T used, rounded up, so `kubectl get pv` reports
# something true. The real limit is the ZFS pool.
dependencies: [] # default-deny: reaches only DNS; clients/Seerr reach IT (docs/03; R-NET-01/02)
ingressRoutes:
- host: plex.home.keithjackson.dev
- host: plex.${BASE_DOMAIN}
port: 32400 # Plex web UI + API served on 32400 (wildcard TLS at the Gateway; docs/08/09)
auth:
mode: native # Plex account auth — documented exception, NOT Keycloak (docs/13/15; R-IDP-02)
+129 -16
View File
@@ -4,19 +4,18 @@
# internal download client | egress VPN + kill-switch | config on Longhorn | auth forward-auth
# (bare web UI, no native OIDC).
#
# BULK DOWNLOADS TIER — OPEN PREREQUISITE, intentionally NOT declared here (mirrors plex + sonarr/radarr):
# docs/13 row 13 places qBittorrent's downloads/staging volume on the `bulk` tier (large, non-3x-
# replicated media handoff to the arr stack). The `bulk` StorageClass exists
# (platform/longhorn/storageclasses.yaml) and both the schema and renderer support `class: bulk`, BUT
# it is a NON-PROVISIONING STUB pending OQ-2 (its NFS/SMB backing is REPLACE-ME and the NFS CSI driver
# is not installed) — a `bulk` PVC would never bind and would leave the pod Pending. Declaring it now
# would also pull the bulk-tier requirement into the change's "touched" set, whose in-cluster
# conformance test is not yet scaffolded, blocking the merge gate. So the downloads volume is DEFERRED
# and this onboarding ships config-only. Once OQ-2 is resolved and `bulk` provisions, attach downloads
# with a one-line add under storage:
# - name: downloads
# class: bulk # external mount; size omitted (sized outside the cluster)
# then re-render. See the WS-APPS report for the full bulk-downloads finding.
# BULK DOWNLOADS TIER — DELIVERED 2026-09-12 (the deferral below is CLOSED; kept for the record).
# This entry used to ship config-only, with a comment promising to attach the downloads volume "once
# OQ-2 is resolved". OQ-2 was resolved for STATIC pre-existing datasets (schema 1.5.0 / renderer
# `class: bulk` + `export` -> a static, pre-bound NFS PV), and plex/sonarr/radarr/nextcloud all took
# their media mounts — but qBittorrent's was never added, so the DOWNLOAD CLIENT was the one member
# of the chain with no media volume at all. Measured on prod 2026-09-12:
# Session\DefaultSavePath=/downloads/ Session\TempPath=/downloads/incomplete/
# Downloads\SavePath=/downloads/ `ls /downloads` -> No such file or directory
# i.e. the app was configured to write a path that DID NOT EXIST, in a container with no volume there.
# Anything it did manage to create would have been invisible to sonarr/radarr, lost on every pod
# recreation, and filling the NODE's ephemeral disk in the meantime. The `downloads` entry under
# storage: below closes that. See docs/BULK-NFS-STORAGE.md and the bulk-tier L0 conformance test.
#
# VPN-class (docs/13 row 13; docs/03 "VPN-class set"): ALL of this pod's public-internet egress
# (peer/tracker/DHT traffic) MUST leave via the node PIA tunnel, never the WAN. `egress: vpn` renders
@@ -32,20 +31,134 @@ name: qbittorrent
kind: container
source:
image: lscr.io/linuxserver/qbittorrent:5.1.2 # pinned stable tag (R-UPD-01); update.range governs advancement
exposure: internal # LAN-only web UI at qbittorrent.home.keithjackson.dev (docs/13 row 13; R-DNS/R-NET)
# LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
# LOAD-BEARING FOR THE BULK MOUNT, not cosmetic — exactly as on sonarr/radarr. Measured on prod
# 2026-09-12 BEFORE this change: `id` inside the container reported the qbittorrent-nox process as
# uid=911 gid=911, and /mnt/Storage/plex/downloads is 0755 owned 1000:1000. 0755 grants write to the
# OWNER ONLY, so uid 911 would have mounted the export cleanly and then failed every write with
# EACCES — the "mounts fine, cannot write" failure the sonarr entry warns about at length.
env:
- name: PUID
value: "1000" # owner uid of /mnt/Storage/plex/downloads (and of every dir under the export)
- name: PGID
value: "3000" # the export's dominant group, matching sonarr/radarr: files this writes stay
# readable to the EXISTING non-k3s consumers of the same dataset, and land
# group-owned the same way the arrs' own writes do
podSecurityContext:
# For the LONGHORN config PVC only, and for the same reason radarr/sonarr set it: fsGroup is a NO-OP
# on the NFS volume (the in-tree plugin does no ownership management), but on a Longhorn filesystem
# the kubelet applies it, so /config becomes group-writable by the now-uid-1000 process without
# depending on the image chowning it. (This image DOES chown: init-qbittorrent-config runs
# `lsiown -R abc:abc /config`. Belt and braces — the live /config was 911:911 before this change,
# and losing write access to it would lose the WebUI password hash and the session state.)
fsGroup: 3000
exposure: internal # LAN-only web UI at qbittorrent.${BASE_DOMAIN} (docs/13 row 13; R-DNS/R-NET)
egress: vpn # ALL public egress via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
storage:
- name: config
size: 5Gi
class: longhorn # settings/categories/session on 3-replica Longhorn; DR-critical (docs/04; R-STATE-02)
dependencies: [] # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
mountPath: /config # THE APP WRITES /config, NOT the renderer default /data/config.
# LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
# /config), so without this the Longhorn PVC mounted at /data/config sat
# holding nothing but lost+found while the app wrote its entire
# configuration to the container's EPHEMERAL writable layer — destroyed on
# every pod recreation: restart, reschedule, image bump. Measured in-cluster
# 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
# dev=2053 on the node's own disk with [Sentry asp config.xml logs].
# This is the SAME failure catalog/plex.yaml:54 already records fixing.
# The live /config contents were copied onto each PVC before this landed,
# so API keys and credentials other apps reference were preserved.
# THE DOWNLOADS AREA — ONE mount of the SAME export sonarr/radarr/plex already consume
# (192.168.2.102:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
# /downloads, which is precisely where this app is ALREADY configured to write.
#
# WHY subPath: downloads AND NOT THE EXPORT ROOT AT /media (the shape sonarr/radarr use):
# * LEAST PRIVILEGE, AND IT COSTS NOTHING HERE. A download client has no business writing the
# LIBRARY. The arrs need the root mount because THEY hardlink downloads/ -> tv/ or movies/ and
# link() is a same-MOUNT operation (EXDEV otherwise) — their smallest common ancestor genuinely
# IS the export root. qBittorrent performs no such cross-directory link: it writes into the
# downloads tree and nothing else, so narrowing to that subtree removes movies/, tv/ and photos/
# from this pod's mount namespace without removing any operation it actually performs.
# * HARDLINKS STILL WORK, because they are not done here. The arrs still see ONE mount containing
# both /media/downloads and /media/tv, so their link() stays intra-mount. Proven live 2026-09-12:
# a file created from this pod under /downloads was hardlinked from the sonarr pod between
# /media/downloads/... and /media/tv/... with st_nlink 1 -> 2 and no EXDEV.
# * IT IS ONE MOUNT, so the L0 test `bulk-tier-distinct-from-replicated` assertion (6) — no
# workload mounts one export twice — holds. That assertion is about TWO mounts inside ONE pod;
# a single subPath mount is not the EXDEV trap, two are.
# * The narrowing is a CLIENT/kubelet guarantee, stated honestly: the NFS server still exports the
# whole dataset to this node, so the containment is the mount namespace, not the server's ACLs —
# the same class of caveat docs/BULK-NFS-STORAGE.md §5 records for plex's readOnly.
#
# NOT readOnly — this is the one workload whose whole job is WRITING here.
#
# KNOWN, DELIBERATE SIDE EFFECT ON SHARED STATE: the image's init-qbittorrent-config runs
# `lsiown abc:abc /downloads` (non-recursive) whenever /downloads is a mount, so starting this pod
# chgrps /mnt/Storage/plex/downloads from 1000:1000 to 1000:3000 on the SERVER. Benign and verified:
# the directory is mode 0755, whose group bits (r-x) are identical to its other bits, so no consumer
# — k3s or legacy — gains or loses any access. Recorded because it is a write to state shared with
# non-cluster consumers, which should never be a surprise.
#
# PATH AGREEMENT WITH THE ARRS IS STILL *NOT* CLOSED — see DECISIONS-LOG 2026-09-12 "Remote Path
# Mappings". This pod calls the finished file /downloads/<name>; sonarr/radarr call the same bytes
# /media/downloads/<name>. Same inode, different absolute path, so Completed Download Handling needs
# ONE Remote Path Mapping per arr (remote host `qbittorrent.qbittorrent.svc.cluster.local`, remote
# path /downloads/, local path /media/downloads/).
#
# THOSE MAPPINGS ARE NOW CONFIGURED AND VERIFIED ON PROD (2026-09-12), together with the qBittorrent
# download client itself — which did not exist in EITHER arr until then (both
# `GET /api/v3/downloadclient` returned `[]`, so completed-download handling had never been wired up
# at all). `downloadclient/testall` reports isValid=true in both, and Sonarr's queue was observed
# reporting outputPath=/downloads/... with the mapping deleted and /media/downloads/... with it
# present — same torrent, same client.
#
# BUT THEY LIVE IN THE ARRS' OWN DATABASES (app state on their config PVCs), NOT IN GIT: a rebuild
# from git alone does NOT restore them and no gate here notices their absence. The alternative —
# repointing qBittorrent's SavePath at /media/downloads and mounting this subPath there instead —
# buys path agreement at the cost of editing THIS app's config PVC, which is also app state; it was
# not chosen because it changes a running download client's save path, which the deferral above
# already shows is easy to get wrong. It is nonetheless the RECOMMENDED end state (it makes the
# mappings unnecessary and puts the mountPath half of the fix in git) and is awaiting an operator
# decision, not a technical blocker.
#
# RELATED OPEN BLOCKER: this app has no durable WebUI credential (no WebUI\Username /
# WebUI\Password_PBKDF2 in qBittorrent.conf), so v5 issues a NEW temporary `admin` password on every
# start. The arrs' saved download-client password therefore goes stale on every pod restart. See the
# same DECISIONS-LOG entry for the two fixes, neither of which was taken unilaterally.
- name: downloads
class: bulk
export: /mnt/Storage/plex # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER} — the SAME export
# plex (ro) and sonarr/radarr (rw) mount; one dataset, one set of inodes
subPath: downloads # the handoff subtree ONLY: no movies/, no tv/, no photos/
mountPath: /downloads # matches the app's EXISTING Session\DefaultSavePath / TempPath, so no
# change to config-PVC app state is needed to make this take effect
size: 9Ti # ADVISORY ONLY on a static NFS PV (the ZFS pool is the real limit): the
# dataset's 8.4 T used rounded up, stated the same way plex/sonarr/radarr
# state it so `kubectl get pv` reports something true
dependencies: [] # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02).
# STILL EMPTY WITH THE NFS MOUNT ABOVE, AND THAT IS CORRECT, NOT AN
# OVERSIGHT. An NFS PersistentVolume is mounted by the NODE KERNEL in the
# HOST netns, not by this pod: there is no pod-side socket to 2049, so no
# CiliumNetworkPolicy can permit or deny it and none is needed. Verified by
# construction — sonarr and radarr have mounted this same export read-write
# for weeks and neither has any CNP naming the NFS server or port 2049
# (grep apps/wave-1/sonarr.yaml: none). The SAME fact is what keeps the VPN
# egress gateway off this traffic (its selector is a podSelector), and
# 192.168.2.102 is inside the 192.168.0.0/16 excludedCIDR besides — see
# platform/vpn-egress/egress-steering.yaml, "BULK NFS".
ingressRoutes:
- host: qbittorrent.home.keithjackson.dev
- host: qbittorrent.${BASE_DOMAIN}
port: 8080 # qBittorrent WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
auth:
mode: forward-auth # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)
groups:
- media-users # authenticated AND in an authorized group (R-IDP-06)
# See catalog/jackett.yaml for the full rationale: --allowed-group is repeatable and OR-ed, so
# platform-admins here is a superset, not a widening (that group already carries cluster-admin
# via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
# immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
- platform-admins # operators of these services must not be locked out of them
update:
channel: stable # R-UPD-01
range: ">=5.0.0 <6.0.0" # stay on the qBittorrent 5.x stable line
+84 -7
View File
@@ -9,18 +9,90 @@ name: radarr
kind: container
source:
image: lscr.io/linuxserver/radarr:5.21.1 # pinned stable v5 tag (R-UPD-01); update.range governs advancement
exposure: internal # LAN-only web UI at radarr.home.keithjackson.dev (docs/13 row 3; R-DNS/R-NET)
# LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
# LOAD-BEARING for the bulk mount, not cosmetic: see the storage note below. The default is 911,
# which cannot write a 1000:3000-owned 0755 tree.
env:
- name: PUID
value: "1000" # owner uid of every directory under /mnt/Storage/plex
- name: PGID
value: "3000" # dominant group there; keeps files this writes readable to the
# EXISTING non-k3s consumers of the same export, which stay live
- name: TZ
value: Etc/UTC # same safe default plex uses; the arrs schedule by local time
podSecurityContext:
# For the LONGHORN config PVC only. fsGroup does nothing on the NFS volume (the in-tree plugin
# does no ownership management), but on a Longhorn filesystem the kubelet applies it, which is
# what makes /config writable by the now-uid-1000 process WITHOUT relying on the image chowning
# /config for itself when PUID changes. Matches PGID so the primary group owns the config too.
fsGroup: 3000
exposure: internal # LAN-only web UI at radarr.${BASE_DOMAIN} (docs/13 row 3; R-DNS/R-NET)
egress: vpn # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
# kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
storage:
- name: config
size: 5Gi
class: longhorn # app DB / config on the 3-replica tier; DR-critical (docs/04; R-STATE-02)
# NOTE (bulk-media PREREQUISITE): docs/13 row 3 places the media library / downloads volume on the
# `bulk` tier (docs/04 bulk storage). The `bulk` StorageClass exists (platform/longhorn/storageclasses.yaml)
# but is a NON-provisioning STUB pending OQ-2 (NFS/SMB backing = REPLACE-ME). Adding a `class: bulk`
# volume now would leave the PVC (and therefore the pod) Pending. The media/downloads volume is
# therefore DEFERRED until OQ-2 is resolved; config-only on longhorn keeps Radarr deployable today.
mountPath: /config # THE APP WRITES /config, NOT the renderer default /data/config.
# LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
# /config), so without this the Longhorn PVC mounted at /data/config sat
# holding nothing but lost+found while the app wrote its entire
# configuration to the container's EPHEMERAL writable layer — destroyed on
# every pod recreation: restart, reschedule, image bump. Measured in-cluster
# 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
# dev=2053 on the node's own disk with [Sentry asp config.xml logs].
# This is the SAME failure catalog/plex.yaml:54 already records fixing.
# The live /config contents were copied onto each PVC before this landed,
# so API keys and credentials other apps reference were preserved.
# THE MEDIA TREE (docs/04 bulk tier) — ONE read-WRITE mount of the export ROOT at /media, giving
# /media/movies (the library) and /media/downloads (the import source) UNDER THE SAME MOUNT.
#
# *** THE ROOT MOUNT IS THE POINT, NOT A CONVENIENCE. *** Radarr imports by HARDLINKING or
# atomically renaming a finished download into the library. Both are same-FILESYSTEM operations,
# and to the kernel "same filesystem" means SAME MOUNT. The layout on the host is:
# /mnt/Storage/plex/movies/ the library
# /mnt/Storage/plex/downloads/complete/radarr/ where finished releases land
# — siblings under one dataset, so a hardlink between them is legal on the server. But mounting
# them as two subPath volumes (subPath: movies at /media/movies, subPath: downloads elsewhere) creates
# TWO mounts, and link() across mounts fails EXDEV no matter what the server thinks. Radarr then
# silently degrades to copy-then-delete: every import rewrites the whole file over NFS and the
# seeding copy is destroyed. Their smallest common ancestor IS the export root, so the root mount
# is the MINIMAL mount that preserves hardlinks — least privilege bounded below by a hard
# requirement, not a shortcut. (The cost is stated plainly: Radarr can also write /media/tv,
# which is sonarr's. Narrowing that would cost hardlinks, which is a worse trade.)
#
# *** /media, NOT /data — AND THAT IS NOT A STYLE CHOICE. *** The config PVC above renders at the
# renderer default /data/config. A media mount at /data would be mounted OVER it and MASK the
# config volume entirely; the pod would come up with its app database shadowed by an 8 T media
# tree. /media collides with nothing in a LinuxServer image and matches the path plex uses for
# the same export, so when qbittorrent is eventually given this mount too, the arrs and the
# download client will already agree on absolute paths — which is the other precondition for
# hardlinked imports (otherwise every import needs a Remote Path Mapping).
#
# SEPARATE LATENT BUG, NOT FIXED HERE (deliberately — it changes live on-disk state): /data/config
# is almost certainly the wrong place for this PVC in the first place. LinuxServer's arr images run
# with `-data=/config`, so the app writes /config — the container's EPHEMERAL layer — while the
# Longhorn PVC at /data/config sits unused. That is the EXACT failure plex's own mountPath comment
# documents having already been bitten by. Fixing it means starting from an empty config volume, so
# it belongs in its own change with the operator present.
#
# UID/GID — THE FAILURE THIS AVOIDS IS "MOUNTS FINE, CANNOT WRITE". The export is no_root_squash
# + sec=sys, so the client's numeric uid is passed through verbatim and the server's ownership is
# the whole story. The tree is 0755 owned 1000:3000 (some dirs 1000:1000), so ONLY uid 1000 can
# write it. LinuxServer images default to PUID/PGID 911 — which would mount cleanly and then fail
# every import with EACCES. The PUID/PGID set above make the process uid 1000, matching the owner.
#
# AND NOTE fsGroup CANNOT SUBSTITUTE FOR THAT. The in-tree NFS volume plugin performs no
# ownership management, so podSecurityContext.fsGroup is a NO-OP on this volume — it is not the
# lever here, and reaching for it is how this failure normally gets misdiagnosed. The fsGroup
# that IS set above exists for the LONGHORN config PVC, so that changing PUID does not depend on
# the image re-chowning /config for itself.
- name: media
class: bulk
export: /mnt/Storage/plex # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER}
mountPath: /media # => /media/movies (Root Folder) and /media/downloads (import source)
size: 9Ti # ADVISORY ONLY on a static NFS PV: the dataset's 8.33 T used,
# rounded up, so `kubectl get pv` reports something true.
dependencies:
- to: qbittorrent # arr → download client (docs/13 dependency graph; R-NET-02 explicit allow)
port: 8080
@@ -29,12 +101,17 @@ dependencies:
port: 9117
protocol: tcp
ingressRoutes:
- host: radarr.home.keithjackson.dev
- host: radarr.${BASE_DOMAIN}
port: 7878 # Radarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
auth:
mode: forward-auth # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)
groups:
- media-users # authenticated AND in the media group (R-IDP-06)
# See catalog/jackett.yaml for the full rationale: --allowed-group is repeatable and OR-ed, so
# platform-admins here is a superset, not a widening (that group already carries cluster-admin
# via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
# immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
- platform-admins # operators of these services must not be locked out of them
update:
channel: stable # R-UPD-01
range: ">=5.0.0 <6.0.0" # stay on the Radarr v5 stable line
+383 -10
View File
@@ -4,7 +4,8 @@
Consumes ONE ServiceDefinition (schemas/servicedefinition.schema.json, catalog/*.yaml)
and emits the concrete manifest set the renderer contract specifies
(schemas/renderer-contract.md): Namespace + default-deny baseline, Deployment/Service or
KubeVirt VirtualMachine + CDI DataVolume, PVCs, per-dependency CiliumNetworkPolicy allows,
KubeVirt VirtualMachine + CDI DataVolume, PVCs, per-dependency and per-in-cluster-client
CiliumNetworkPolicy allows,
egress-class steering label, HTTPRoute(s), ServiceMonitor/alerts, Flux image-automation,
and the auth-mode / backup wiring hints.
@@ -44,7 +45,39 @@ except ImportError: # pragma: no cover - explicit, actionable failure
# --- Label / naming conventions (match platform/kyverno/policies + platform/*/) --------
LBL = "homelab.keithjackson.dev"
ZONE = "home.keithjackson.dev"
# The platform hostname suffix, emitted as the Flux post-build TOKEN rather than a literal.
# Every rendered manifest lands under a Flux path whose Kustomization declares
# postBuild.substituteFrom cluster-vars, so `${BASE_DOMAIN}` is resolved per environment at
# reconcile time: dev renders home.keithjackson.dev, prod renders prod.home.keithjackson.dev.
# It is a DECLARED cluster-vars knob (clusters/base/cluster-vars.yaml), so it is written BARE —
# the escape contract below reserves `$${...}` for expansions meant for a container's shell.
# Hardcoding the literal here is what let prod be authored to serve dev's public hostnames,
# and prod's external-dns then deleted dev's records (infra/PROD-PROVISIONING.md §10).
ZONE = "${BASE_DOMAIN}"
# The bulk (external NFS) tier's SERVER, emitted as a Flux post-build TOKEN for exactly the same
# reason as ZONE: dev and prod reach the same Proxmox ZFS host by DIFFERENT addresses (dev's nodes
# are on the host's own LAN and dial it directly; prod's nodes are on the far side of pfSense and
# must be given whichever address the still-pending routing decision produces). Hardcoding either
# one authors a manifest that is silently wrong in the other environment. Declared in
# clusters/base/cluster-vars.yaml and patched by BOTH overlays; written BARE (it is a real knob,
# not a shell expansion), so StrictPostBuildSubstitutions fails the tier if an overlay forgets it.
BULK_NFS_SERVER = "${BULK_NFS_SERVER}"
# Mount options for every bulk NFS volume. These are a PLATFORM decision, not a per-service one,
# so they live here rather than on the ServiceDefinition surface:
# nfsvers=4.1 NFSv4 carries mount+lock IN the protocol, so it needs ONE port (tcp/2049) and no
# rpcbind/111, mountd or statd. That is what makes the prod path a single pfSense
# rule instead of four, and it is why v4 is chosen over v3. The server advertises
# +3 +4 +4.1 +4.2; 4.1 is the conservative floor every node kernel already speaks.
# hard A `soft` mount turns a server hiccup into a short read / EIO handed to the
# application, which for a media library and an archive is silent corruption.
# `hard` blocks instead. The cost is explicit: an NFS outage HANGS I/O rather than
# failing it, so an outage shows up as stuck pods, not as damaged files. Chosen
# deliberately -- hung is recoverable, truncated is not.
# noatime The datasets are relatime, so every read would otherwise generate a metadata
# WRITE back to the pool. Plex walking an 8 T library must not do that.
# timeo=600/retrans=2 the TCP defaults, stated explicitly so a future reader knows the retry
# budget was considered rather than inherited.
BULK_NFS_MOUNT_OPTIONS = ["nfsvers=4.1", "hard", "noatime", "timeo=600", "retrans=2"]
GATEWAY_NAME = "shared"
GATEWAY_NS = "gateway"
KP_RELEASE = "kube-prometheus-stack" # ServiceMonitor/PrometheusRule discovery label
@@ -84,6 +117,7 @@ def apply_defaults(sd: dict) -> dict:
s.setdefault("storage", [])
s.setdefault("dependencies", [])
s.setdefault("ingressRoutes", [])
s.setdefault("inClusterClients", [])
# auth.mode default posture: forward-auth for any exposed UI, else none (internal, no
# UI). An explicit auth block always wins (R-IDP-02).
@@ -103,10 +137,19 @@ def apply_defaults(sd: dict) -> dict:
obs.setdefault("slis", [])
s["observability"] = obs
# storage entry class default is longhorn (replicated tier).
s["storage"] = [{**{"class": "longhorn"}, **v} for v in s["storage"]]
# storage entry class default is longhorn (replicated tier); readOnly default is False
# (grant-nothing-extra is expressed by SETTING it, so the default stays back-compatible and
# every pre-existing golden renders byte-identical).
s["storage"] = [{**{"class": "longhorn", "readOnly": False}, **v} for v in s["storage"]]
# dependency protocol default is tcp.
s["dependencies"] = [{**{"protocol": "tcp"}, **d} for d in s["dependencies"]]
# in-cluster client protocol default is tcp (as for a dependency); its `name` — which only
# ever appears in the rendered POLICY name — defaults to the namespace it admits, so the
# common case declares just `namespace` + `port` and reads like a dependency edge.
s["inClusterClients"] = [
{**{"protocol": "tcp", "name": c.get("namespace", "")}, **c}
for c in s["inClusterClients"]
]
return s
@@ -192,18 +235,139 @@ def _service_ports(s: dict) -> list:
return [{"name": f"port-{p}", "port": p, "targetPort": p} for p in ports]
def _bulk_access_mode(vol: dict) -> str:
"""A bulk volume's access mode follows its read/write posture.
Every bulk volume is `...Many`: the SAME export is consumed by several namespaces at once
(plex, sonarr and radarr all sit on /mnt/Storage/plex) and by several nodes at once, which is
precisely what ReadWriteOnce cannot express. WHICH `Many` it is says whether this consumer
writes -- so `kubectl get pv` reports the intent, not merely the topology.
"""
return "ReadOnlyMany" if vol.get("readOnly") else "ReadWriteMany"
def r_bulk_pvs(s: dict) -> list:
"""One STATIC PersistentVolume per `class: bulk` storage entry (docs/04 bulk tier).
WHY A PV AT ALL, RATHER THAN AN INLINE `nfs:` POD VOLUME
-------------------------------------------------------
An inline volume is fewer objects and it would work. It is rejected for four reasons, in
descending order of what they would cost to discover later:
1. RECLAIM POLICY. `persistentVolumeReclaimPolicy: Retain` is a declared, reviewable
property of a PV. There is no equivalent for an inline volume -- and the data behind
these mounts is 8.33 TiB of media and a 267 GiB archive that ALSO has live non-k3s
consumers. A statement in git that nothing in this cluster may reclaim it is worth an
object.
2. READ-ONLY LIVES IN ONE PLACE. `spec.nfs.readOnly: true` becomes a real `ro` mount option
for every pod that ever binds this volume. The same intent expressed on an inline volume
is one Deployment edit away from being dropped silently, with no diff anywhere near the
storage. (It is still a CLIENT-side guarantee -- the export cannot distinguish plex from
sonarr, since both may run on any node -- but it is one that lives in one reviewable
place.)
3. VISIBILITY. `kubectl get pv,pvc` is where an operator looks for "what external storage is
attached, and who holds it". An inline volume is invisible there, and invisible to
anything that reasons over PVCs.
4. IT IS THE CONTRACT THE REPO ALREADY HAS. Every other storage entry is a PVC; the
renderer, the Kyverno ServiceDefinition policy and the backup wiring all key on that. An
inline volume would be a second, parallel storage model inside the same renderer.
CROSS-NAMESPACE. A PV binds 1:1 to a PVC, so "plex, sonarr and radarr share one export" is NOT
one shared PV -- it is three PVs, one per namespace, each naming the same server+path. That is
the correct model rather than a workaround: the sharing happens on the NFS server, which is
built for concurrent clients, and each namespace keeps its own independently reviewable access
mode (plex ReadOnlyMany, the arrs ReadWriteMany). One shared PV would have forced a single
posture on all three.
BINDING IS PINNED FROM BOTH ENDS -- `claimRef` here and `volumeName` on the PVC. Either alone
leaves a race in which some other PVC of matching size and mode grabs the volume first;
together the binding is deterministic. `storageClassName: ""` (explicitly empty, NOT the `bulk`
StorageClass) is what tells the binder this volume is statically provisioned: it suppresses the
default-StorageClass annotation and guarantees no provisioner is ever asked to create anything.
It also sidesteps a trap -- a StorageClass's `provisioner` is IMMUTABLE, so re-pointing the
existing `bulk` class (whose stub names an NFS CSI driver that is not installed) at this data
would make Flux fail forever on that one object.
"""
out = []
for vol in s["storage"]:
if vol.get("class") != "bulk":
continue
name = f"{s['name']}-{vol['name']}"
# `export` and `size` are both structurally required of a bulk entry, and the JSON
# Schema only enforces the first. A PersistentVolume with no `capacity` is rejected by
# the API server, so a missing size must fail HERE with a sentence a human can act on
# rather than as a bare KeyError three frames down.
for field, why in (
("export", "the absolute NFS export path this volume attaches"),
("size", "the PV capacity (advisory, but the API requires the field)"),
):
if not vol.get(field):
raise ValueError(
f"{s['name']}: storage entry {vol['name']!r} has class 'bulk' but no "
f"'{field}' — {why}. A bulk volume attaches storage that already exists "
f"outside the cluster, so it cannot be rendered without it."
)
nfs = {"server": BULK_NFS_SERVER, "path": vol["export"]}
if vol.get("readOnly"):
nfs["readOnly"] = True
spec = {
"capacity": {"storage": vol["size"]},
"accessModes": [_bulk_access_mode(vol)],
# Retain: deleting the PVC must NEVER be able to reach the data. There is nothing to
# "reclaim" here in any case -- the volume was not provisioned by the cluster.
"persistentVolumeReclaimPolicy": "Retain",
"storageClassName": "",
"mountOptions": list(BULK_NFS_MOUNT_OPTIONS),
"nfs": nfs,
"claimRef": {"namespace": s["name"], "name": name},
}
out.append(
{
"apiVersion": "v1",
"kind": "PersistentVolume",
"metadata": {
"name": name,
"labels": {"app.kubernetes.io/name": s["name"]},
},
"spec": spec,
}
)
return out
def r_pvcs(s: dict) -> list:
"""One PVC per storage entry (R-STATE-01/02/04).
longhorn -> 3-replica replicated tier; bulk -> external media tier (not 3x replicated).
longhorn -> 3-replica replicated tier, dynamically provisioned, ReadWriteOnce.
bulk -> the STATIC external NFS PV rendered by r_bulk_pvs, pre-bound by volumeName.
"""
out = []
for vol in s["storage"]:
cls = vol.get("class", "longhorn")
storage_class = "longhorn" if cls == "longhorn" else "bulk"
if cls == "bulk":
# Mirror image of the PV: the same explicitly-empty class (so nothing dynamically
# provisions), the same access mode, and volumeName completing the two-sided binding.
out.append(
{
"apiVersion": "v1",
"kind": "PersistentVolumeClaim",
"metadata": {
"name": f"{s['name']}-{vol['name']}",
"namespace": s["name"],
"labels": {"app.kubernetes.io/name": s["name"]},
},
"spec": {
"accessModes": [_bulk_access_mode(vol)],
"storageClassName": "",
"volumeName": f"{s['name']}-{vol['name']}",
"resources": {"requests": {"storage": vol["size"]}},
},
}
)
continue
spec = {
"accessModes": ["ReadWriteOnce"],
"storageClassName": storage_class,
"storageClassName": "longhorn",
}
if vol.get("size"):
spec["resources"] = {"requests": {"storage": vol["size"]}}
@@ -249,7 +413,17 @@ def r_deployment(s: dict) -> dict:
# it verbatim for apps that need a fixed on-disk layout (e.g. Gitea keeps all state
# directly under /data). Omitting it preserves the /data/<name> default (back-compat).
mount_path = vol.get("mountPath", f"/data/{vol['name']}")
volume_mounts.append({"name": vol["name"], "mountPath": mount_path})
mount = {"name": vol["name"], "mountPath": mount_path}
# subPath mounts a subtree of the volume instead of its root. Emitted only when declared,
# so every pre-existing golden stays byte-identical.
if vol.get("subPath"):
mount["subPath"] = vol["subPath"]
# readOnly on the mount as well as on the PV's nfs source. Belt and braces on purpose: the
# PV field is what makes the kernel mount `ro`; this one is what a reviewer reading the
# Deployment sees. An edit that drops one still leaves the other.
if vol.get("readOnly"):
mount["readOnly"] = True
volume_mounts.append(mount)
volumes.append(
{
"name": vol["name"],
@@ -535,6 +709,82 @@ def r_gateway_ingress_policy(s: dict):
}
def r_in_cluster_client_policies(s: dict) -> list:
"""East-west allow: let a declared IN-CLUSTER client reach this service DIRECTLY (R-NET-02).
`ingressRoutes` opens the NORTH-SOUTH path only — r_gateway_ingress_policy admits the
reserved `ingress` identity (the Gateway proxy) and nothing else — and `dependencies`
compiles edges only BETWEEN CATALOG SERVICES (both halves are rendered from a catalog
entry's own declaration). A PLATFORM CONTROLLER that dials this service's ClusterIP
directly therefore had no authorable form at all, and under default-deny (R-NET-01) it is
DROPPED.
That gap is not theoretical: prod's `flux-system` source-controller clones the platform's
own repo from the in-cluster Gitea Service (clusters/prod/kustomization.yaml pins
`url: http://gitea.gitea.svc.cluster.local:3000/...` so prod's GitOps depends on neither
public DNS, Cloudflare, its own Gateway nor ${BASE_DOMAIN}). gitea-allow-gateway-ingress
admits `fromEntities: [ingress]` only, so the clone was denied, the source could not
reconcile — and the HTTPRoute that would have served a public name instead lives in a
DOWNSTREAM tier that cannot apply while the source is down. Source down -> routes never
rename -> source stays down. It had to be broken with a live `kubectl apply`, i.e. with a
cluster object outside git, which one `kubectl delete` or one rebuild would take away.
Each entry renders ONE least-privilege ingress CiliumNetworkPolicy, shaped exactly like the
reciprocal-dependency half (r_reverse_dependency_policies) but selecting the client by
NAMESPACE alone, because a platform controller's pod labels are the upstream chart's to
change and are not part of this platform's contract: `fromEndpoints` carrying ONLY the
reserved `k8s:io.kubernetes.pod.namespace` label, `toPorts` carrying ONLY the declared
port/protocol. Never `fromEntities: all`, and never the whole service. Additive: the
namespace default-deny baseline keeps denying every undeclared source and port.
The policy NAME is `<service>-allow-<name>-ingress`, `name` defaulting to the namespace —
it is operator-facing (`kubectl get cnp`, a Hubble drop), and letting an author pin it is
what allows an allowance that already had to be applied live to be authored under its
EXISTING name, so Flux adopts that object in place rather than creating a near-duplicate
beside it and leaving the live one unmanaged.
"""
out = []
for client in s["inClusterClients"]:
proto = client.get("protocol", "tcp").upper()
out.append(
{
"apiVersion": "cilium.io/v2",
"kind": "CiliumNetworkPolicy",
"metadata": {
"name": f"{s['name']}-allow-{client['name']}-ingress",
"namespace": s["name"],
"labels": {"app.kubernetes.io/name": s["name"]},
},
"spec": {
"endpointSelector": {
"matchLabels": {"app.kubernetes.io/name": s["name"]}
},
"ingress": [
{
"fromEndpoints": [
{
"matchLabels": {
"k8s:io.kubernetes.pod.namespace": client[
"namespace"
]
}
}
],
"toPorts": [
{
"ports": [
{"port": str(client["port"]), "protocol": proto}
]
}
],
}
],
},
}
)
return out
def _route_meta(s: dict, route: dict) -> dict:
meta = {
"name": f"{s['name']}-{route['port']}",
@@ -1087,6 +1337,14 @@ def render(sd: dict, image_automation: bool = True, inbound: list | None = None)
if s["kind"] == "container":
# Container PVCs. (For a vm, the CDI DataVolume owns/creates the disk PVC, so we
# never render a plain PVC for the VM root disk — it would collide with CDI.)
# The static bulk PV is rendered BEFORE the PVC that names it. Document order inside one
# kustomize build is not an apply-ordering guarantee, but a PVC whose PV does not exist
# yet merely stays Pending and binds on the next reconcile, so this is a readability
# convention rather than a correctness requirement.
for pv in r_bulk_pvs(s):
docs.append(
(f"PersistentVolume {pv['metadata']['name']} - static external NFS (bulk tier)", pv)
)
for pvc in r_pvcs(s):
docs.append((f"PVC {pvc['metadata']['name']} (R-STATE)", pvc))
docs.append(("Deployment (R-PLAT-02)", r_deployment(s)))
@@ -1119,6 +1377,15 @@ def render(sd: dict, image_automation: bool = True, inbound: list | None = None)
)
)
for cnp in r_in_cluster_client_policies(s):
docs.append(
(
"CiliumNetworkPolicy in-cluster client ingress allow (R-NET-02): "
f"{cnp['metadata']['name']}",
cnp,
)
)
ie = r_internet_egress(s)
docs.append(
(
@@ -1313,6 +1580,11 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
- `<svc>-db.yaml` -> wave(svc-db) (a strictly-lower wave; CNPG DB)
- `<svc>-db-app.sops.yaml` -> wave(svc-db) (CNPG initdb owner secret)
- `<svc>-db-backup-credentials.sops.yaml` -> wave(svc-db) (CNPG Barman S3 backup secret)
- `<svc>-db-backup-credentials-prod.sops.yaml` -> wave(svc-db) (the PROD half of that
secret: the two environments carry DIFFERENT object-store credentials, selected in
the Cluster CR by `${BACKUP_CREDENTIALS_SUFFIX}`. Both objects ship to both clusters,
so both must be placed in the DB wave — the Secret has to exist before the Cluster
that names it.)
Non-catalog units (EXTRA_UNITS, e.g. gitea-actions) place by their declared anchor+offset.
Only files that actually exist in `present_files` are placed, so the plan is robust to a
service having no bootstrap secret or no DB. Deterministic (sorted).
@@ -1340,6 +1612,9 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
add(wdb, f"{db}.yaml")
add(wdb, f"{db}-app.sops.yaml")
add(wdb, f"{db}-backup-credentials.sops.yaml") # CNPG Barman S3 creds (R-STATE-03)
# PROD half of the same credential (2026-09-07 per-env split). Placed only when
# the file exists, so a DB that has not been split is unaffected.
add(wdb, f"{db}-backup-credentials-prod.sops.yaml")
for unit, cfg in EXTRA_UNITS.items():
anchor = cfg["anchor"]
@@ -1627,6 +1902,101 @@ def _selftest() -> int:
"a non-exposed service must not render a gateway-ingress allow (least privilege)",
)
# 4c) inClusterClients -> one ADDITIVE, least-privilege ingress allow per entry that admits a
# PLATFORM controller's NAMESPACE (not the reserved `ingress` identity, not `all`) to ONE
# declared port. This is what keeps a cluster whose GitOps source is an in-cluster Service
# reconcilable: fromEntities[ingress] does not cover a controller dialling the ClusterIP.
icc_docs = render(
{
"name": "g",
"source": {"image": "a:1"},
"ingressRoutes": [{"host": "g.home.keithjackson.dev", "port": 3000}],
"auth": {"mode": "none"},
"inClusterClients": [
{"name": "flux", "namespace": "flux-system", "port": 3000}
],
}
)
icc = [
d[1]
for d in icc_docs
if d[1]["kind"] == "CiliumNetworkPolicy"
and d[1]["metadata"]["name"] == "g-allow-flux-ingress"
]
check(
len(icc) == 1,
"an inClusterClients entry must render one ingress CNP named "
"<service>-allow-<name>-ingress",
)
if icc:
p = icc[0]
check(
p["metadata"]["namespace"] == "g"
and p["spec"]["endpointSelector"]["matchLabels"]
== {"app.kubernetes.io/name": "g"},
"in-cluster client allow must live in the service's namespace and select the service",
)
ing = p["spec"]["ingress"]
check(
ing[0]["fromEndpoints"]
== [{"matchLabels": {"k8s:io.kubernetes.pod.namespace": "flux-system"}}],
"in-cluster client allow must match the client NAMESPACE only (reserved k8s: label)",
)
check(
"fromEntities" not in ing[0],
"in-cluster client allow must not widen to an entity (never fromEntities: all)",
)
icc_ports = {
(pt["port"], pt["protocol"])
for r in ing
for tp in r["toPorts"]
for pt in tp["ports"]
}
check(
icc_ports == {("3000", "TCP")},
"in-cluster client allow must be least-privilege (only the declared port/protocol)",
)
# ADDITIVE: default-deny ingress stays [] and the gateway allow is untouched.
icc_dd = next(d[1] for d in icc_docs if d[1]["kind"] == "NetworkPolicy")
check(
icc_dd["spec"]["ingress"] == [],
"in-cluster client allow must be ADDITIVE — default-deny ingress stays [] (R-NET-01)",
)
check(
any(
d[1]["metadata"].get("name") == "g-allow-gateway-ingress"
for d in icc_docs
if d[1]["kind"] == "CiliumNetworkPolicy"
),
"in-cluster client allow must not displace the north-south gateway allow",
)
# `name` is OPTIONAL and defaults to the namespace, so the common case is namespace + port.
icc_default = render(
{
"name": "g",
"source": {"image": "a:1"},
"inClusterClients": [{"namespace": "monitoring", "port": 9090}],
}
)
check(
any(
d[1]["metadata"].get("name") == "g-allow-monitoring-ingress"
for d in icc_default
if d[1]["kind"] == "CiliumNetworkPolicy"
),
"inClusterClients.name must default to the admitted namespace",
)
# A service that declares none must render NO such policy (grant-nothing-extra).
check(
not any(
d[1]["kind"] == "CiliumNetworkPolicy"
and d[1]["metadata"]["name"].endswith("-ingress")
and "fromEndpoints" in d[1]["spec"].get("ingress", [{}])[0]
for d in render({"name": "q", "source": {"image": "a:1"}})
),
"a service declaring no inClusterClients must render no in-cluster client allow",
)
# 5) vm kind -> VirtualMachine + standalone DataVolume.
vm_docs = render(
{
@@ -1822,6 +2192,7 @@ def _selftest() -> int:
present = {
"a.yaml", "svc.yaml", "svc-bootstrap-secret.sops.yaml",
"svc-db.yaml", "svc-db-app.sops.yaml", "svc-db-backup-credentials.sops.yaml",
"svc-db-backup-credentials-prod.sops.yaml",
"gitea.yaml",
"gitea-actions.yaml", "gitea-actions-runner-secret.sops.yaml",
}
@@ -1831,8 +2202,10 @@ def _selftest() -> int:
flat = {f: wv for wv, fs in plan.items() for f in fs}
check(flat.get("a.yaml") == 0, "leaf app must land in wave 0")
check(flat.get("svc-db.yaml") == 0 and flat.get("svc-db-app.sops.yaml") == 0
and flat.get("svc-db-backup-credentials.sops.yaml") == 0,
"CNPG DB manifest + its owner secret + its Barman backup secret must be in the DB's (lower) wave")
and flat.get("svc-db-backup-credentials.sops.yaml") == 0
and flat.get("svc-db-backup-credentials-prod.sops.yaml") == 0,
"CNPG DB manifest + its owner secret + BOTH halves of its per-env Barman backup secret "
"must be in the DB's (lower) wave")
check(flat.get("svc.yaml") == 1 and flat.get("svc-bootstrap-secret.sops.yaml") == 1,
"a DB-backed app + its bootstrap secret must be in the consumer wave, above the DB")
check(flat.get("gitea.yaml") == 0, "gitea (leaf) must be wave 0")
+1 -1
View File
@@ -57,7 +57,7 @@ dependencies:
port: 32400
protocol: tcp
ingressRoutes:
- host: seerr.home.keithjackson.dev
- host: seerr.${BASE_DOMAIN}
port: 5055 # Overseerr web UI + API; wildcard TLS terminates at the shared Gateway (docs/08/09)
auth:
mode: native # Plex account auth — documented exception, NOT Keycloak (docs/13/15; R-IDP-02)
+33 -13
View File
@@ -58,7 +58,7 @@ env:
- name: SEMAPHORE_ADMIN_NAME
value: Semaphore Admin
- name: SEMAPHORE_ADMIN_EMAIL
value: admin@home.keithjackson.dev
value: admin@${BASE_DOMAIN}
# --- Persistent access-key encryption key (see header): FIXED base64 32-byte key from SOPS so the
# SSH keys / credentials Semaphore stores in the DB stay decryptable across restarts. ---
- name: SEMAPHORE_ACCESS_KEY_ENCRYPTION
@@ -78,16 +78,34 @@ env:
# (util ConfigType `oidc_providers`, env-tagged SEMAPHORE_OIDC_PROVIDERS). The value is a single
# JSON blob for the `keycloak` provider — provider_url is the homelab realm issuer (auto-discovery),
# client_id=semaphore, redirect_url=/api/auth/oidc/keycloak/redirect, claims map to the realm's
# token. It CONTAINS the confidential client secret, so the whole blob is held SOPS-encrypted in the
# semaphore-bootstrap Secret (key oidc-providers-json; same secret value SOPS-held in
# keycloak-realm-clients) and injected here by reference — never in cleartext. The name is NOT a
# kubelet service-link var (those are SEMAPHORE_SERVICE_HOST / SEMAPHORE_PORT_*), so no collision.
# The LOCAL SEMAPHORE_ADMIN above stays as the break-glass path (docs/15). ---
- name: SEMAPHORE_OIDC_PROVIDERS
# token. The name is NOT a kubelet service-link var (those are SEMAPHORE_SERVICE_HOST /
# SEMAPHORE_PORT_*), so no collision. The LOCAL SEMAPHORE_ADMIN above stays break-glass (docs/15).
#
# PER-ENVIRONMENT URLS (2026-09-07 fix). The blob USED to live whole inside the SOPS-encrypted
# semaphore-bootstrap Secret (key `oidc-providers-json`) because it CONTAINS the confidential
# client secret. That hid two environment-specific, NON-SECRET URLs — the realm issuer and the
# redirect URI — inside ciphertext, where Flux's ${BASE_DOMAIN} postBuild substitution CANNOT
# reach them. Both were frozen at the apex `home.keithjackson.dev`, so PROD (BASE_DOMAIN =
# prod.home.keithjackson.dev) pointed its OIDC at DEV's Keycloak — unroutable from prod's
# 192.168.2.0/24 (the identical root cause as the Headlamp login timeout; see
# platform/headlamp/README.md). Dev was only ever "right" because dev IS the apex.
#
# THE SPLIT: the URLs are public, so the blob is now a PLAINTEXT literal here and gets
# ${BASE_DOMAIN}-substituted per environment like every other hostname. Only the confidential
# client secret stays SOPS-encrypted, in its own key (`oidc-client-secret`), and is spliced back
# in by the KUBELET's dependent-env-var expansion: an env `value` may reference an EARLIER env
# var of the same container as $(NAME), and secretKeyRef-sourced vars are eligible. So
# SEMAPHORE_OIDC_CLIENT_SECRET MUST STAY IMMEDIATELY ABOVE — the kubelet resolves env in list
# order, and a reference to a not-yet-defined name is left as the LITERAL text $(...). It is
# fail-loud in the way that matters: if the Secret key is missing the Pod never starts
# (CreateContainerConfigError), rather than silently authenticating against nothing. ---
- name: SEMAPHORE_OIDC_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: semaphore-bootstrap
key: oidc-providers-json
key: oidc-client-secret
- name: SEMAPHORE_OIDC_PROVIDERS
value: '{"keycloak":{"display_name":"Keycloak","provider_url":"https://keycloak.${BASE_DOMAIN}/realms/homelab","client_id":"semaphore","client_secret":"$(SEMAPHORE_OIDC_CLIENT_SECRET)","redirect_url":"https://semaphore.${BASE_DOMAIN}/api/auth/oidc/keycloak/redirect","scopes":["openid","profile","email"],"username_claim":"preferred_username","email_claim":"email","name_claim":"name","order":1}}'
# The semaphoreui/semaphore image runs as the non-root user UID 1001 / GID 0 (tini -> server-wrapper),
# and on boot the wrapper writes its generated config.json into the config dir. A mounted Longhorn PVC
# is root:root, so UID 1001 cannot write it -> the pod panics ("permission denied"). fsGroup=1001 makes
@@ -110,17 +128,19 @@ dependencies:
port: 5432 # Renders the egress half; the DB namespace mirrors keycloak-db (no
protocol: tcp # default-deny), so Cilium's default-allow accepts the reciprocal ingress.
ingressRoutes:
- host: semaphore.home.keithjackson.dev
- host: semaphore.${BASE_DOMAIN}
port: 3000 # Semaphore HTTP web/API; wildcard TLS terminates at the shared Gateway (docs/08/09)
auth:
mode: oidc # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC"; R-IDP-02)
groups:
- platform-admins # access scoped to an authorized group (R-IDP-06)
# WIRED (native OIDC): the Keycloak `semaphore` confidential client (platform/keycloak/keycloak-realm-
# import.yaml + live kcadm) is consumed via the SEMAPHORE_OIDC_PROVIDERS env (JSON blob held encrypted
# in the semaphore-bootstrap Secret, oidc-providers-json). The `groups` claim is exposed for
# authorization (R-IDP-06). Semaphore KEEPS its LOCAL admin as the break-glass path so an IdP outage
# never locks the operator out (docs/15 "Availability caveat"). No secret is embedded here.
# import.yaml + live kcadm) is consumed via the SEMAPHORE_OIDC_PROVIDERS env — a PLAINTEXT JSON blob
# (see `env` above) whose per-environment URLs are ${BASE_DOMAIN}-substituted, with the confidential
# client secret spliced in at runtime from the semaphore-bootstrap Secret key `oidc-client-secret`.
# The `groups` claim is exposed for authorization (R-IDP-06). Semaphore KEEPS its LOCAL admin as the
# break-glass path so an IdP outage never locks the operator out (docs/15 "Availability caveat").
# No secret is embedded here.
update:
channel: stable # R-UPD-01
range: ">=2.19.0 <3.0.0" # stay on the 2.x stable line
+84 -7
View File
@@ -9,18 +9,90 @@ name: sonarr
kind: container
source:
image: lscr.io/linuxserver/sonarr:4.0.15 # pinned stable v4 tag (R-UPD-01); update.range governs advancement
exposure: internal # LAN-only web UI at sonarr.home.keithjackson.dev (docs/13 row 2; R-DNS/R-NET)
# LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
# LOAD-BEARING for the bulk mount, not cosmetic: see the storage note below. The default is 911,
# which cannot write a 1000:3000-owned 0755 tree.
env:
- name: PUID
value: "1000" # owner uid of every directory under /mnt/Storage/plex
- name: PGID
value: "3000" # dominant group there; keeps files this writes readable to the
# EXISTING non-k3s consumers of the same export, which stay live
- name: TZ
value: Etc/UTC # same safe default plex uses; the arrs schedule by local time
podSecurityContext:
# For the LONGHORN config PVC only. fsGroup does nothing on the NFS volume (the in-tree plugin
# does no ownership management), but on a Longhorn filesystem the kubelet applies it, which is
# what makes /config writable by the now-uid-1000 process WITHOUT relying on the image chowning
# /config for itself when PUID changes. Matches PGID so the primary group owns the config too.
fsGroup: 3000
exposure: internal # LAN-only web UI at sonarr.${BASE_DOMAIN} (docs/13 row 2; R-DNS/R-NET)
egress: vpn # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
# kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
storage:
- name: config
size: 5Gi
class: longhorn # app DB / config on the 3-replica tier; DR-critical (docs/04; R-STATE-02)
# NOTE (bulk-media PREREQUISITE): docs/13 row 2 places the media library / downloads volume on the
# `bulk` tier (docs/04 bulk storage). The `bulk` StorageClass exists (platform/longhorn/storageclasses.yaml)
# but is a NON-provisioning STUB pending OQ-2 (NFS/SMB backing = REPLACE-ME). Adding a `class: bulk`
# volume now would leave the PVC (and therefore the pod) Pending. The media/downloads volume is
# therefore DEFERRED until OQ-2 is resolved; config-only on longhorn keeps Sonarr deployable today.
mountPath: /config # THE APP WRITES /config, NOT the renderer default /data/config.
# LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
# /config), so without this the Longhorn PVC mounted at /data/config sat
# holding nothing but lost+found while the app wrote its entire
# configuration to the container's EPHEMERAL writable layer — destroyed on
# every pod recreation: restart, reschedule, image bump. Measured in-cluster
# 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
# dev=2053 on the node's own disk with [Sentry asp config.xml logs].
# This is the SAME failure catalog/plex.yaml:54 already records fixing.
# The live /config contents were copied onto each PVC before this landed,
# so API keys and credentials other apps reference were preserved.
# THE MEDIA TREE (docs/04 bulk tier) — ONE read-WRITE mount of the export ROOT at /media, giving
# /media/tv (the library) and /media/downloads (the import source) UNDER THE SAME MOUNT.
#
# *** THE ROOT MOUNT IS THE POINT, NOT A CONVENIENCE. *** Sonarr imports by HARDLINKING or
# atomically renaming a finished download into the library. Both are same-FILESYSTEM operations,
# and to the kernel "same filesystem" means SAME MOUNT. The layout on the host is:
# /mnt/Storage/plex/tv/ the library
# /mnt/Storage/plex/downloads/complete/tv-sonarr/ where finished releases land
# — siblings under one dataset, so a hardlink between them is legal on the server. But mounting
# them as two subPath volumes (subPath: tv at /media/tv, subPath: downloads elsewhere) creates
# TWO mounts, and link() across mounts fails EXDEV no matter what the server thinks. Sonarr then
# silently degrades to copy-then-delete: every import rewrites the whole file over NFS and the
# seeding copy is destroyed. Their smallest common ancestor IS the export root, so the root mount
# is the MINIMAL mount that preserves hardlinks — least privilege bounded below by a hard
# requirement, not a shortcut. (The cost is stated plainly: Sonarr can also write /media/movies,
# which is radarr's. Narrowing that would cost hardlinks, which is a worse trade.)
#
# *** /media, NOT /data — AND THAT IS NOT A STYLE CHOICE. *** The config PVC above renders at the
# renderer default /data/config. A media mount at /data would be mounted OVER it and MASK the
# config volume entirely; the pod would come up with its app database shadowed by an 8 T media
# tree. /media collides with nothing in a LinuxServer image and matches the path plex uses for
# the same export, so when qbittorrent is eventually given this mount too, the arrs and the
# download client will already agree on absolute paths — which is the other precondition for
# hardlinked imports (otherwise every import needs a Remote Path Mapping).
#
# SEPARATE LATENT BUG, NOT FIXED HERE (deliberately — it changes live on-disk state): /data/config
# is almost certainly the wrong place for this PVC in the first place. LinuxServer's arr images run
# with `-data=/config`, so the app writes /config — the container's EPHEMERAL layer — while the
# Longhorn PVC at /data/config sits unused. That is the EXACT failure plex's own mountPath comment
# documents having already been bitten by. Fixing it means starting from an empty config volume, so
# it belongs in its own change with the operator present.
#
# UID/GID — THE FAILURE THIS AVOIDS IS "MOUNTS FINE, CANNOT WRITE". The export is no_root_squash
# + sec=sys, so the client's numeric uid is passed through verbatim and the server's ownership is
# the whole story. The tree is 0755 owned 1000:3000 (some dirs 1000:1000), so ONLY uid 1000 can
# write it. LinuxServer images default to PUID/PGID 911 — which would mount cleanly and then fail
# every import with EACCES. The PUID/PGID set above make the process uid 1000, matching the owner.
#
# AND NOTE fsGroup CANNOT SUBSTITUTE FOR THAT. The in-tree NFS volume plugin performs no
# ownership management, so podSecurityContext.fsGroup is a NO-OP on this volume — it is not the
# lever here, and reaching for it is how this failure normally gets misdiagnosed. The fsGroup
# that IS set above exists for the LONGHORN config PVC, so that changing PUID does not depend on
# the image re-chowning /config for itself.
- name: media
class: bulk
export: /mnt/Storage/plex # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER}
mountPath: /media # => /media/tv (Root Folder) and /media/downloads (import source)
size: 9Ti # ADVISORY ONLY on a static NFS PV: the dataset's 8.33 T used,
# rounded up, so `kubectl get pv` reports something true.
dependencies:
- to: qbittorrent # arr → download client (docs/13 dependency graph; R-NET-02 explicit allow)
port: 8080
@@ -29,12 +101,17 @@ dependencies:
port: 9117
protocol: tcp
ingressRoutes:
- host: sonarr.home.keithjackson.dev
- host: sonarr.${BASE_DOMAIN}
port: 8989 # Sonarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
auth:
mode: forward-auth # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)
groups:
- media-users # authenticated AND in the media group (R-IDP-06)
# See catalog/jackett.yaml for the full rationale: --allowed-group is repeatable and OR-ed, so
# platform-admins here is a superset, not a widening (that group already carries cluster-admin
# via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
# immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
- platform-admins # operators of these services must not be locked out of them
update:
channel: stable # R-UPD-01
range: ">=4.0.0 <5.0.0" # stay on the Sonarr v4 stable line
+80 -3
View File
@@ -27,9 +27,31 @@ make -C ci test # unit tests for these scripts themselves (ci/tests/*.tes
```
The `l0` stage has three sub-stages: ServiceDefinition schema validation, the Kyverno policy gate
(`kyverno test`), and **`conformance/run.sh --layer l0`** — the whole-repo L0 invariants a
per-resource policy cannot express (secret hygiene across every tracked file, least-privilege RBAC,
spec-immutability enforcement). See `conformance/tests/README.md`.
(`kyverno test`), and **`conformance/run.sh --layer l0 --include-scope whole-repo`** — the whole-repo
L0 invariants a per-resource policy cannot express (secret hygiene across every tracked file,
least-privilege RBAC, spec-immutability enforcement). See `conformance/tests/README.md`.
### `--include-scope whole-repo`: why the L0 sub-stage is not purely requirement-scoped
R-TEST-03 runs the conformance subset a change **touches**. That has one blind spot, and it is the
worst-shaped kind: an invariant that quantifies over the whole tracked tree, or over subtrees the
coarse path→requirement map keys to *nothing* (`apps/`, `infra/`, most of `platform/`), cannot be
selected *by* the requirements a change touches — because **the change most likely to break it
attributes no requirement at all**. Requirement-scoped selection then filters the check out of
exactly that change.
This used to be handled by a hand-written "UNCONDITIONAL" stage per such check (3c subst-safety,
3d keycloak-client-secrets, 3e bulk-tier): three near-identical rationales, three hardcoded
`check.sh` paths, three copies of the same `rc 0/3/*` case block — and, worse, a mechanism that
failed **silently**, because whoever added the next whole-repo invariant had to *remember* to write
a stage. Those stages are gone. A test now **declares** `conformance.home.k3s/scope: "whole-repo"`
and the runner unions every declaring test into the run. Adding the annotation is the only step.
`rc 3` is preserved exactly as those stages had it: a **SKIP** of a scope-declared check means the
tree it audits has vanished, so the runner exits `3`, `ci/l0.sh` sets the gate's own `rc=3`, and the
gate reports SKIPPED — **never a pass**. The mechanism is itself proven, end-to-end and falsifiably,
by `conformance/tests/l0-static/whole-repo-invariants-wired`; `conformance/lib/common.sh` carries the
full rationale and the incidents behind it.
## Exit-code contract
@@ -37,6 +59,47 @@ Every stage script uses: **`0` = pass · `1` = fail (blocks merge) · `3` = SKIP
with no cluster, or an unwired seam / missing tool). A `3` is **never** counted as green — that is how
the gate stays honest in Phase 0 (no fake green).
### The gate's OWN verdict (changed 2026-09-13 — read this if you consume the exit code)
**`MERGE GATE: PASS` + exit 0 means every stage that was supposed to run RAN and was GREEN.** It
used to mean less than that: a FAILED stage set the block flag but a SKIPPED one only printed a
warning, so the gate printed `l0: skipped (unwired seam / missing tool) — not green` and then
`MERGE GATE: PASS`, exit 0. That happened for real twice — a missing `promtool` (the alert-rule
unit tests) and a missing pinned `kyverno` CLI — and exit 0 is what automation reads as
merge-approved. Fail-open in the one component whose whole job is to fail closed.
| verdict | exit | meaning |
|---|---|---|
| `MERGE GATE: PASS` | `0` | every stage that was supposed to run ran and was green; the R-TEST-03 subset was checked. Any capability the environment **declared** out of scope is **named in the headline** (see below), so a PASS never implies more than it verified. |
| `MERGE GATE: BLOCKED` | `1` | a stage FAILED, or a stage did not run **where it must**, or the decision input is unusable. |
| `MERGE GATE: UNDETERMINED` | `3` | nothing failed, but a stage did not run, so greenness was **not established**. Never a pass; non-zero so nothing can read it as approval. `3` is this repo's existing "not run" code, now spoken by the gate as a whole and not only by its stages. |
| `MERGE GATE: NOT APPLICABLE` | `0` | there is no branch under gate at all (`HEAD` is on base — a push-to-`main` run). Stages ran; no subset was checked, and the headline says so. |
Precedence: BLOCKED > UNDETERMINED > NOT APPLICABLE > PASS.
**`make` masks the difference.** GNU make reports any recipe failure as its own exit `2`, so
`make -C ci gate` collapses `1` and `3` into `2` (the headline still tells a human which it was). A
caller that must distinguish them invokes `./ci/gate.sh` directly.
### Which skips block, and why — a skip is not one thing
| stage | a skip there means | verdict |
|---|---|---|
| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is installed for us by `.gitea/workflows/scripts/install-tools.sh`, so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
| `spec-immutability` (R-TEST-06) | the sibling spec repo is not checked out, so the spec **could not be verified** | **UNDETERMINED** — unverified is unverified. Deliberately *not* upgraded to BLOCKED by strict skips: the live runner does not check the spec repo out at all, so blocking would leave the only enforced status check permanently red for a reason no PR could clear. It is therefore the **one** skip whose non-answer a caller may knowingly tolerate, and `.gitea/workflows/gate.yml` does so in one visible line. `REQUIRE_SPEC_VERIFICATION=true` makes it a hard block once that checkout is wired. Locally: check the spec repo out beside this one, or set `SPEC_REPO` (`dev.sh` mounts a sibling automatically). |
| `conformance` (L2/L3) | `CLUSTER_AVAILABLE!=true` — the environment **declared** the capability absent (documented Phase-0 behaviour) | **does not deny a PASS**: a declared-inapplicable stage was never *supposed* to run. It is NAMED in the PASS headline and step summary, the touched requirements it would have covered are reported UNVERIFIED, and `REQUIRE_CLUSTER_TESTS=true` (Phase 1+) turns that into a block. A skip **despite** `CLUSTER_AVAILABLE=true` is a defect and BLOCKS. |
That asymmetry is the whole rule: **declared-not-applicable may pass; could-not-run may not.**
`GATE_STRICT_SKIPS=true|false` decides strictness and is **declared, never sniffed** — `gate.yml`
sets it on the gate step, because the workflow is the thing that knows it is CI. Unset, it falls
back to `GITHUB_ACTIONS` / `CI` (set by the runner itself). A wrong answer in either direction
cannot fail open: both branches are non-zero and neither is a PASS. Set it locally to rehearse CI
strictness.
Locked down by `ci/tests/gate-decision.test.sh` (47 assertions over the decision table: stubbed
stage outcomes against the real `gate.sh`; 32 of them fail against the pre-2026-09-13 gate).
### The touched-requirement subset can also be *undeterminable* (exit 3)
`touched-requirements.sh` is the input to the R-TEST-03 decision. It resolves a diff base in this
@@ -87,6 +150,8 @@ ci/
lint.sh Stage: yaml/lint
l0.sh Stage: schema validation + Kyverno L0 gate + L0 static conformance
l1.sh Stage: L1 rendering checks (Chainsaw L1 via the conformance runner)
(harness: platform/monitoring/tests/). Catches the two alerting bugs no
other stage can see — permanently-firing and structurally INERT rules.
conformance.sh Stage: L2/L3 in-cluster Chainsaw Job (trigger + collect)
touched-requirements.sh compute the R-* subset a change touches (drives the R-TEST-03 filter);
exits 3 LOUDLY rather than guessing when the diff base is unreliable
@@ -122,3 +187,15 @@ its owner delivers a runnable entrypoint.
`CLUSTER_AVAILABLE=true` (env locally, `vars.CLUSTER_AVAILABLE` in Gitea) makes the L2/L3 stage run and
be **required**; anything else makes it **SKIP** (Phase 0). `REQUIRE_CLUSTER_TESTS=true` (Phase 1+)
turns any unverified cluster-needing touched requirement into a hard block.
All four switches are read through one `truthy()` in `lib.sh`, so they spell "on" identically
(`true|1|yes|y|on`, any case) and — the direction that matters — **anything else is OFF**: a
misspelled `CLUSTER_AVAILABLE=ture` leaves the capability undeclared instead of silently claiming
it, which can only make the gate stricter about what it says it verified.
| switch | default | effect |
|---|---|---|
| `CLUSTER_AVAILABLE` | `false` | declares a cluster; L2/L3 then run and are required |
| `REQUIRE_CLUSTER_TESTS` | `false` | Phase 1+: an unverified cluster-needing touched requirement BLOCKS |
| `REQUIRE_SPEC_VERIFICATION` | `false` | an unverifiable spec repo BLOCKS instead of UNDETERMINED |
| `GATE_STRICT_SKIPS` | `auto` | `true` = a missing tool / unwired seam BLOCKS (CI); `auto` follows `GITHUB_ACTIONS`/`CI` |
+5 -1
View File
@@ -47,7 +47,11 @@ fi
# (conformance/lib/common.sh) now unions a comma/space list of layers, so this is a single
# invocation with one aggregated report (no per-layer looping needed). Artifacts are then
# COLLECTED from the fixed location below.
"$CONFORMANCE_CMD" --layer l2,l3 "${req_args[@]}"
# --include-scope: union the touched subset with every L2/L3 test DECLARING
# `conformance.home.k3s/scope: whole-repo` (none today — wired once, here, so a future
# whole-repo behavioural invariant needs only the annotation; see conformance/lib/common.sh).
: "${WHOLE_REPO_SCOPE:=whole-repo}"
"$CONFORMANCE_CMD" --layer l2,l3 --include-scope "$WHOLE_REPO_SCOPE" "${req_args[@]}"
CONF_ART="$(dirname "$CONFORMANCE_CMD")/_artifacts"
if [ -f "${CONF_ART}/report.junit.xml" ]; then
+163 -10
View File
@@ -7,6 +7,75 @@
# skip (3) not run (cluster-gated stage w/ no cluster, or unwired seam) —
# NOT counted as green (HARD RULE 4: don't fake green)
#
# ...AND THREE OUTCOMES FOR THE GATE ITSELF (2026-09-13). Until today a stage that
# SKIPPED only printed a warning ("— not green") and the gate then printed
# `MERGE GATE: PASS` and exited 0 anyway. That is fail-open in the one component whose
# entire job is to fail closed: a pinned tool missing from the runner, or a seam silently
# becoming unwired, produced exit 0 and read as merge-approved. Observed live:
# `prom-rules: skipped (missing tool) — not green` followed by `MERGE GATE: PASS`, and
# `l0: skipped (kyverno CLI not installed)` likewise. The gate's own vocabulary already
# had the answer — the 2026-08-31 "local false green" introduced UNDECIDED for exactly
# this shape of non-answer — the offline stages just did not use it.
#
# MERGE GATE: PASS exit 0 every stage that was SUPPOSED TO RUN ran and was
# green, and the R-TEST-03 subset was checked.
# MERGE GATE: BLOCKED exit 1 a stage FAILED, or a stage did not run WHERE IT
# MUST (see the per-stage rules below), or the
# decision input is unusable.
# MERGE GATE: UNDETERMINED exit 3 nothing failed, but a stage did not run, so
# greenness was NOT ESTABLISHED. Never a pass;
# non-zero so no automation can read it as approval.
# `3` is this repo's existing "not run / not
# established" code (ci/lib.sh), now spoken by the
# gate as a whole and not only by its stages.
# MERGE GATE: NOT APPLICABLE exit 0 there is no branch under gate at all (HEAD is on
# base) — a push-to-main CI run. Stages were green;
# no subset was checked, and the headline says so.
#
# NOTE ON `make`: GNU make reports ANY recipe failure as its own exit 2, so
# `make -C ci gate` collapses 1 and 3 into 2 — a human sees the headline and that is enough.
# A caller that must tell BLOCKED from UNDETERMINED invokes ./ci/gate.sh directly; to tolerate
# an UNDETERMINED run it writes `./ci/gate.sh || [ $? -eq 3 ]`, the same explicit idiom
# .gitea/workflows/gate.yml already used for its spec-immutability pre-step and now uses for
# the gate itself (with GATE_STRICT_SKIPS=true, so that tolerance can ONLY ever cover the
# declared-environmental skip below — a missing tool on the runner is exit 1, untolerated).
# TOLERATING A NON-ANSWER IS A VISIBLE, REVIEWED CHOICE IN THE CALLER. There is deliberately no
# knob inside this script that turns UNDETERMINED into exit 0: that would be the fail-open this
# change exists to remove, just relocated.
#
# WHICH SKIPS BLOCK, AND WHY (per stage — a skip is not one thing):
#
# lint / l0 / render-golden / l1 A skip here means a PINNED tool is absent or a wired
# seam went unwired. On a CI runner the pinned toolchain is installed for us by
# .gitea/workflows/scripts/install-tools.sh, so a skip there is a DEFECT, not an
# environment: it BLOCKS. On a workstation a missing tool is a real local condition, so
# it degrades to UNDETERMINED (still non-zero, still never a pass). See strict_skips()
# in ci/lib.sh for how CI is detected and why misdetecting it cannot fail open.
#
# spec-immutability (R-TEST-06) A skip means the sibling spec repo is not checked out, so
# the spec COULD NOT BE VERIFIED. Never a pass — unverified is unverified — so it denies a
# PASS and yields UNDETERMINED. It is deliberately NOT upgraded to BLOCKED by CI strictness,
# because on the live runner it is not a defect anyone can fix from inside this repo: the
# workflow does not check the spec repo out at all (SPEC_REPO_SLUG is unwired — the blueprint
# lives in merge-gate.yaml), and blocking would leave the only enforced status check
# permanently red for a reason no PR could clear. So THIS is the one skip whose non-answer the
# CALLER may knowingly tolerate, and gate.yml does so in one visible line. Locally the
# remedy exists and the message names it: check the spec repo out beside this one, or point
# SPEC_REPO at it (infra/devcontainer/dev.sh mounts it automatically when it is a sibling).
# Set REQUIRE_SPEC_VERIFICATION=true (the counterpart of REQUIRE_CLUSTER_TESTS) to make it a
# hard block once that checkout is wired.
#
# conformance (L2/L3) A skip when CLUSTER_AVAILABLE!=true is NOT a defect and
# NOT an accident: it is the environment DECLARING that the capability is absent, which
# is the documented Phase-0 behaviour. A declared-inapplicable stage was never "supposed
# to run", so it does not deny a PASS — the touched requirements it would have covered
# are still reported UNVERIFIED, and REQUIRE_CLUSTER_TESTS=true (Phase 1+) turns that
# into a block. A skip DESPITE CLUSTER_AVAILABLE=true is a defect and blocks, exactly as
# it already did. That asymmetry is the whole distinction this file now draws:
# DECLARED-not-applicable may pass; COULD-NOT-RUN may not. A declared gap is still NAMED in
# the PASS headline (see DECLARED_GAPS below), so a PASS never reads as "everything was
# verified" when a layer was declared out of scope.
#
# Phase 0 policy (no cluster): L0/L1 stages (lint, schema, kyverno, L1 render)
# MUST be green. L2/L3 SKIP. If the touched subset needs L2/L3 and no cluster is
# available, the gate reports those requirements UNVERIFIED and — unless
@@ -20,6 +89,7 @@ source "${CI_DIR}/lib.sh"
set +e # lib.sh enables -e; the gate must run ALL stages and aggregate, not abort.
: "${REQUIRE_CLUSTER_TESTS:=false}"
: "${REQUIRE_SPEC_VERIFICATION:=false}"
declare -A OUTCOME
run_stage() {
@@ -47,6 +117,13 @@ run_stage conformance "${CI_DIR}/conformance.sh"
echo >&2
log "==== gate decision ===="
BLOCK=0
# Stages that did NOT run and are tolerated here — but never as green. A non-empty list
# means the gate could not establish greenness -> MERGE GATE: UNDETERMINED, exit 3.
NOT_ESTABLISHED=()
# Stages the environment DECLARED inapplicable (today: L2/L3 with no cluster declared). These
# do not deny a PASS — nothing was supposed to run — but they are named IN the PASS headline, so
# "PASS" is never mistaken for "everything was verified".
DECLARED_GAPS=()
# --- R-TEST-03 input: the touched-requirement subset --------------------------
# Computed ONCE here. ci/touched-requirements.sh exits 3 when it cannot establish a
@@ -64,37 +141,80 @@ rm -f "$_reason_file"
TOUCHED=()
[ -n "$TOUCHED_OUT" ] && mapfile -t TOUCHED <<< "$TOUCHED_OUT"
UNDECIDED=0
SUBSET_BROKEN=0
if [ "$TOUCHED_RC" -eq 3 ]; then
UNDECIDED=1
echo >&2
err "R-TEST-03 DECISION INPUT MISSING: the touched-requirement subset could not be"
err " determined (${TOUCHED_REASON:-unknown}). The scoped coverage gate did NOT run."
summary "- **R-TEST-03 subset UNDETERMINED** (\`${TOUCHED_REASON:-unknown}\`) — the scoped coverage gate did not run."
elif [ "$TOUCHED_RC" -ne 0 ]; then
# Any OTHER non-zero exit is a hard defect in the subset computation itself — an
# incomplete path->requirement map (a changed path matches no row) or an invalid row
# would report exactly this. The reasoning is taken from the unlanded
# fix/touched-requirements-fail-closed (47068ee): gate.sh special-cased ONLY exit 3, so
# any other non-zero exit left TOUCHED empty and UNDECIDED=0, and the decision below
# printed "MERGE GATE: PASS" having applied no coverage gate at all. Unreachable on
# `main` today (touched-requirements.sh exits only 0 or 3) and wired here so it is
# already closed the moment a fail-closed path map makes exit 1 reachable. A subset that
# cannot be computed is NOT an empty subset.
SUBSET_BROKEN=1
echo >&2
err "R-TEST-03 DECISION INPUT BROKEN: touched-requirements.sh exited ${TOUCHED_RC}"
err " (${TOUCHED_REASON:-unknown}). Its output above names the paths/rows to fix."
summary "- **R-TEST-03 subset UNUSABLE** (\`${TOUCHED_REASON:-unknown}\`, exit ${TOUCHED_RC}) — the coverage gate could not be applied."
elif [ "${#TOUCHED[@]}" -gt 0 ]; then
log "touched requirements: ${TOUCHED[*]}"
fi
# spec-immutability (R-TEST-06): a FAIL always blocks (the spec was modified/drifted).
# A skip means the spec repo wasn't available to verify — surfaced, not green.
# A skip means the spec repo wasn't available to verify — UNVERIFIED, never green: it
# denies a PASS (exit 3) and, with REQUIRE_SPEC_VERIFICATION=true, blocks outright. See the
# file header for why this one is not upgraded to a block by CI detection alone.
case "${OUTCOME[spec-immutability]}" in
fail) err "spec-immutability: FAILED -> blocks merge (R-TEST-06)"; BLOCK=1 ;;
skip) warn "spec-immutability: skipped (spec repo unavailable) — not verified" ;;
skip)
if truthy "$REQUIRE_SPEC_VERIFICATION"; then
err "spec-immutability: skipped (spec repo unavailable) and REQUIRE_SPEC_VERIFICATION=true -> blocks merge"
summary "- spec-immutability: **SKIPPED** (spec repo unavailable) — blocks merge (REQUIRE_SPEC_VERIFICATION=true)"
BLOCK=1
else
warn "spec-immutability: skipped (spec repo unavailable) — R-TEST-06 NOT VERIFIED, so"
warn " this run cannot be a PASS. Check the spec repo out beside this one (or set"
warn " SPEC_REPO) to verify it."
summary "- spec-immutability: **SKIPPED** (spec repo unavailable — R-TEST-06 not verified; not a pass)"
NOT_ESTABLISHED+=("spec-immutability")
fi ;;
pass) ok "spec-immutability: green" ;;
esac
# L0/L1 stages: any failure blocks. A SKIP here means a seam is unwired — in
# Phase 0 that is tolerated (placeholders), but surfaced. render-golden is a
# L0/L1 stages: any failure blocks. A SKIP here means a pinned tool is missing or a seam is
# unwired — NEVER green. In CI (strict) that is a defect and blocks; on a workstation it
# degrades to UNDETERMINED, which is still non-zero and still not a pass. render-golden is a
# no-cluster reproducibility gate (canonical render == committed apps/ goldens):
# a FAIL means a golden drifted from its ServiceDefinition source — always blocks.
for s in lint l0 render-golden l1; do
case "${OUTCOME[$s]}" in
fail) err "${s}: FAILED -> blocks merge"; BLOCK=1 ;;
skip) warn "${s}: skipped (unwired seam / missing tool) — not green" ;;
skip)
if strict_skips; then
err "${s}: skipped (unwired seam / missing tool) under STRICT skips, where the pinned"
err " toolchain is installed for us (.gitea/workflows/scripts/install-tools.sh) and every"
err " seam is on main — so this is a DEFECT, not an environment -> blocks merge"
summary "- ${s}: **SKIPPED under strict skips** (unwired seam / missing tool) — blocks merge"
BLOCK=1
else
warn "${s}: skipped (unwired seam / missing tool) — not green, so this run cannot be a PASS"
summary "- ${s}: **SKIPPED** (unwired seam / missing tool) — not green"
NOT_ESTABLISHED+=("$s")
fi ;;
pass) ok "${s}: green" ;;
esac
done
# L2/L3 conformance: cluster-gated.
# L2/L3 conformance: cluster-gated. The ONLY stage whose non-execution can still be a PASS,
# because CLUSTER_AVAILABLE!=true is the environment DECLARING the capability absent rather
# than the stage failing to run where it could have — see the file header.
case "${OUTCOME[conformance]}" in
fail)
err "conformance (L2/L3): FAILED -> blocks merge"; BLOCK=1 ;;
@@ -105,18 +225,22 @@ case "${OUTCOME[conformance]}" in
err "conformance: skipped despite CLUSTER_AVAILABLE=true -> blocks merge"; BLOCK=1
elif [ "$UNDECIDED" -eq 1 ]; then
warn "conformance L2/L3 UNVERIFIED, and the touched subset is unknown — nothing certified."
DECLARED_GAPS+=("conformance-l2-l3(no cluster declared)")
else
if [ "${#TOUCHED[@]}" -gt 0 ]; then
reqs=("${TOUCHED[@]}")
warn "conformance L2/L3 UNVERIFIED for touched requirements: ${reqs[*]}"
warn " (no cluster — Phase 0). These are NOT certified green."
summary "- **UNVERIFIED (no cluster):** ${reqs[*]}"
if [ "$REQUIRE_CLUSTER_TESTS" = "true" ]; then
if truthy "$REQUIRE_CLUSTER_TESTS"; then
err "REQUIRE_CLUSTER_TESTS=true and cluster-needing requirements are unverified -> blocks merge"
BLOCK=1
else
DECLARED_GAPS+=("conformance-l2-l3(no cluster declared; ${#TOUCHED[@]} touched reqs unverified)")
fi
else
skip "conformance L2/L3 skipped; no cluster-needing requirements touched"
DECLARED_GAPS+=("conformance-l2-l3(no cluster declared)")
fi
fi ;;
esac
@@ -132,13 +256,34 @@ if [ "$UNDECIDED" -eq 1 ] && [ "$TOUCHED_REASON" != "on-base" ]; then
err "cannot compute the R-TEST-03 subset (${TOUCHED_REASON:-unknown}) -> blocks merge"
BLOCK=1
fi
# An incomplete/invalid path map is not an environment problem to skip past: the change under
# gate contains paths whose requirements are UNKNOWN, so no subset can be certified.
if [ "$SUBSET_BROKEN" -eq 1 ]; then
err "R-TEST-03 subset unusable (${TOUCHED_REASON:-unknown}) -> blocks merge"
BLOCK=1
fi
echo >&2
# Precedence: BLOCKED (something is wrong) > UNDETERMINED (a stage did not run, so
# greenness was not established) > NOT APPLICABLE (nothing to gate) > PASS. UNDETERMINED
# outranks NOT APPLICABLE because "no branch under gate" AND "a stage did not run" is two
# reasons this is not a green run, and the stronger, non-zero one must be the headline.
if [ "$BLOCK" -ne 0 ]; then
err "MERGE GATE: BLOCKED"
summary ""
summary "**Result: BLOCKED** — see failures above."
exit 1
elif [ "${#NOT_ESTABLISHED[@]}" -gt 0 ]; then
err "MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED."
err " Stages that did not run: ${NOT_ESTABLISHED[*]}"
err " Nothing FAILED, and nothing is certified either: a stage that did not run is not"
err " green (HARD RULE 4). This is exit 3, never 0 — no automation may read it as"
err " merge-approved. Fix the environment (install the pinned tool / wire the seam /"
err " check out the spec repo) and run again, or accept it KNOWINGLY and explicitly:"
err " ./ci/gate.sh || [ \$? -eq 3 ]"
summary ""
summary "**Result: UNDETERMINED** — these stages did not run, so nothing is certified: \`${NOT_ESTABLISHED[*]}\`. Not a pass (exit 3)."
exit 3
elif [ "$UNDECIDED" -eq 1 ]; then
warn "MERGE GATE: NOT APPLICABLE — no branch under gate (${TOUCHED_REASON})."
warn " The stages above ran and did not fail, but NO touched-requirement subset was"
@@ -148,8 +293,16 @@ elif [ "$UNDECIDED" -eq 1 ]; then
summary "**Result: NOT APPLICABLE** — no branch under gate (\`${TOUCHED_REASON}\`); the R-TEST-03 subset was not checked."
exit 0
else
ok "MERGE GATE: PASS (touched-requirement subset green for the layers that can run)"
summary ""
summary "**Result: PASS** — L0/L1 green. L2/L3 status per cluster availability above."
ok "MERGE GATE: PASS — every stage that was supposed to run RAN and was GREEN, and the"
ok " R-TEST-03 touched-requirement subset was checked."
if [ "${#DECLARED_GAPS[@]}" -gt 0 ]; then
ok " NOT verified here, because the environment DECLARED it out of scope (never a silent"
ok " gap — Phase 0 by policy, see the header): ${DECLARED_GAPS[*]}"
summary ""
summary "**Result: PASS** — every stage that was supposed to run ran green. Declared out of scope (NOT verified): \`${DECLARED_GAPS[*]}\`."
else
summary ""
summary "**Result: PASS** — every stage that was supposed to run ran green; nothing was declared out of scope."
fi
exit 0
fi
+43 -44
View File
@@ -10,6 +10,18 @@
# Stage 3b L0 : conformance/run.sh --layer l0 — the WHOLE-REPO L0 invariants that
# a per-resource policy cannot express (secret hygiene across every
# tracked file, least-privilege RBAC, spec-immutability enforcement).
# Selection = the TOUCHED requirement subset (R-TEST-03) UNION every L0
# test DECLARING `conformance.home.k3s/scope: whole-repo`. That union
# replaces what used to be a hand-written "UNCONDITIONAL" stage per
# whole-repo check (3c subst-safety / 3d keycloak-client-secrets / 3e
# bulk-tier): three near-identical 12-line rationales, three hardcoded
# check.sh paths and three copies of the same rc 0/3/* case block, which
# did not scale and — worse — failed SILENTLY, because whoever added the
# next whole-repo invariant had to REMEMBER to hand-write a stage. Now the
# test declares its own scope and the runner enforces it. The reasoning and
# the incidents those stages recorded were moved INTO the tests they
# describe (see each chainsaw-test.yaml's `scope` annotation) and the
# general mechanism is documented in conformance/lib/common.sh.
#
# Integration seams are env-overridable (see ci/lib.sh + README). When an owning
# subtree is still a placeholder, this reports SKIPPED (exit 3) for that sub-stage
@@ -83,58 +95,45 @@ fi
# binding, the spec-immutability gate's own behaviour) live in the conformance
# harness's tests/l0-static subtree and run with no cluster. This stage executes them
# through the same single-command runner CI and a laptop both use (R-TEST-02).
log "Stage: L0 static conformance checks (${CONFORMANCE_CMD} --layer l0)"
#
# THE UNION, AND WHY IT IS NOT OPTIONAL (R-TEST-03's blind spot)
# --------------------------------------------------------------
# 3b scopes the conformance run to the TOUCHED requirements. That is right for an invariant
# whose inputs live under the subtree its requirements map to. It is WRONG for one that
# quantifies over the WHOLE tracked tree, or over subtrees (apps/, infra/, most of platform/)
# that ci/touched-requirements.sh's coarse path map keys to NOTHING: for those, the change
# most likely to break the invariant attributes no requirement at all, so requirement-scoped
# selection filters the check out of EXACTLY that change. `--include-scope whole-repo` unions
# in every L0 test that DECLARES that scope, so those run on every gate invocation.
#
# rc 3 IS NOT A PASS. The runner returns 3 when a scope-declared check SKIPPED (its subject
# subtree is absent, or it has no check script): UNWIRED, not green, exactly as the deleted
# per-check stages had it ("check SKIPPED (clusters/ absent) — not a pass"). It is mapped to
# the gate's own rc 3 below, never to rc 0.
: "${WHOLE_REPO_SCOPE:=whole-repo}"
log "Stage: L0 static conformance checks (${CONFORMANCE_CMD} --layer l0 --include-scope ${WHOLE_REPO_SCOPE})"
if ! entrypoint_ready "$CONFORMANCE_CMD"; then
skip "conformance runner not present at '${CONFORMANCE_CMD}' — UNWIRED, not green"
summary "- l0-static: **SKIPPED** (conformance runner missing)"
if [ "$rc" -eq 0 ]; then rc=3; fi
else
load_touched_requirements
if "$CONFORMANCE_CMD" --layer l0 ${REQ_ARGS[@]+"${REQ_ARGS[@]}"}; then
ok "L0 static conformance checks passed"
summary "- l0-static: **passed**"
else
err "L0 static conformance checks failed"
summary "- l0-static: **FAILED**"
rc=1
fi
fi
# ---- Stage 3c: Flux post-build substitution safety (UNCONDITIONAL) ----------
# R-ENV-02's promote-once mechanism is `postBuild.substituteFrom cluster-vars`, and it
# FAILS OPEN: an undefined ${TOKEN} becomes the empty string and the Kustomization still
# reports Ready. That emptied Cilium's k8sServiceHost and broke the CNI on 2/3 dev nodes
# (DECISIONS-LOG 2026-09-01, "substitution race that fails OPEN"). The check asserts every
# ${...} in every substituted path is DEFINED in cluster-vars or deliberately ESCAPED
# `$${...}` — i.e. that enabling the StrictPostBuildSubstitutions feature gate cannot fail
# a legitimate manifest, and that no expansion silently empties today.
#
# WHY THIS IS ITS OWN STAGE and not left to Stage 3b: 3b scopes the conformance run to the
# TOUCHED requirements (R-TEST-03), and ci/touched-requirements.sh's coarse path map keys
# only clusters/* to R-ENV-02 — an edit to a SUBSTITUTED manifest under apps/ or platform/
# attributes no requirement, so the requirement-scoped selection would filter this check
# out of exactly the change that introduces the bug. The invariant is whole-repo (same
# reasoning as spec-immutability's own gate stage), so it runs on EVERY gate invocation.
# When 3b's touched set does include R-ENV-02/R-GITOPS-01 the check runs twice; it is a
# sub-second offline scan with no side effects, so the duplication is not worth avoiding.
SUBST_CHECK="${SUBST_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/flux-postbuild-substitution/check.sh}"
log "Stage: Flux postBuild substitution safety (${SUBST_CHECK})"
if [ ! -f "$SUBST_CHECK" ]; then
skip "substitution check not present at '${SUBST_CHECK}' — UNWIRED, not green"
summary "- subst-safety: **SKIPPED** (check missing)"
if [ "$rc" -eq 0 ]; then rc=3; fi
else
subst_rc=0
bash "$SUBST_CHECK" || subst_rc=$?
case "$subst_rc" in
0) ok "Flux postBuild substitution safety passed"
summary "- subst-safety: **passed**" ;;
3) skip "substitution check SKIPPED (clusters/ absent) — not a pass"
summary "- subst-safety: **SKIPPED** (nothing to audit)"
l0_static_rc=0
"$CONFORMANCE_CMD" --layer l0 --include-scope "$WHOLE_REPO_SCOPE" \
${REQ_ARGS[@]+"${REQ_ARGS[@]}"} || l0_static_rc=$?
case "$l0_static_rc" in
0) ok "L0 static conformance checks passed"
summary "- l0-static: **passed**" ;;
3) skip "an L0 check declaring scope='${WHOLE_REPO_SCOPE}' SKIPPED — UNWIRED, not a pass"
summary "- l0-static: **SKIPPED** (a whole-repo check is UNWIRED — not a pass)"
if [ "$rc" -eq 0 ]; then rc=3; fi ;;
*) err "Flux postBuild substitution safety FAILED"
summary "- subst-safety: **FAILED**"; rc=1 ;;
*) err "L0 static conformance checks failed"
summary "- l0-static: **FAILED**"
rc=1 ;;
esac
# Per-check step-summary lines, so the gate output still NAMES what ran — the generated
# replacement for the one hardcoded `summary` line each deleted stage carried.
emit_offline_summary
fi
case "$rc" in
+23 -3
View File
@@ -29,7 +29,16 @@ req_args=(${REQ_ARGS[@]+"${REQ_ARGS[@]}"})
# [--requirements R-A,R-B]; it ALWAYS writes artifacts to a FIXED location
# (conformance/_artifacts/) and does NOT accept --junit/--coverage. Invoke by layer +
# touched subset, then COLLECT its artifacts into the CI artifact dirs.
"$CONFORMANCE_CMD" --layer l1 ${req_args[@]+"${req_args[@]}"}
# --include-scope: union the touched subset with every L1 test DECLARING
# `conformance.home.k3s/scope: whole-repo`. No L1 test declares it today, so this is a no-op
# now — it is wired HERE, once, so the NEXT whole-repo invariant (at any layer) is picked up by
# declaring the annotation, instead of needing someone to remember to hand-write a stage. That
# "someone must remember" is the failure mode this whole change removes; see
# conformance/lib/common.sh. rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass.
: "${WHOLE_REPO_SCOPE:=whole-repo}"
l1_rc=0
"$CONFORMANCE_CMD" --layer l1 --include-scope "$WHOLE_REPO_SCOPE" \
${req_args[@]+"${req_args[@]}"} || l1_rc=$?
CONF_ART="$(dirname "$CONFORMANCE_CMD")/_artifacts"
if [ -f "${CONF_ART}/report.junit.xml" ]; then
@@ -39,5 +48,16 @@ if [ -d "${CONF_ART}/coverage" ]; then
cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
fi
ok "L1 rendering checks passed"
summary "- l1-render: **passed**"
case "$l1_rc" in
0) ok "L1 rendering checks passed"
summary "- l1-render: **passed**"
emit_offline_summary ;;
3) skip "an L1 check declaring scope='${WHOLE_REPO_SCOPE}' SKIPPED — UNWIRED, not a pass"
summary "- l1-render: **SKIPPED** (a whole-repo check is UNWIRED — not a pass)"
emit_offline_summary
exit 3 ;;
*) err "L1 rendering checks FAILED (rc=${l1_rc})"
summary "- l1-render: **FAILED**"
emit_offline_summary
exit "$l1_rc" ;;
esac
+81 -9
View File
@@ -72,20 +72,83 @@ entrypoint_ready() {
return 1
}
# --- Cluster detection (drives SKIP vs run for L2/L3) -------------------------
# A cluster is 'available' only if explicitly declared. We NEVER auto-assume a
# cluster in Phase 0 — silence means SKIP, not pass (HARD RULE 4).
# CLUSTER_AVAILABLE=true -> L2/L3 run and are REQUIRED green
# CLUSTER_AVAILABLE=false/unset -> L2/L3 SKIP (reported, never faked green)
cluster_available() {
case "${CLUSTER_AVAILABLE:-false}" in
true|1|yes) return 0 ;;
# truthy <value> — the ONE place the repo's "is this switch on" spelling lives, so
# CLUSTER_AVAILABLE / REQUIRE_CLUSTER_TESTS / REQUIRE_SPEC_VERIFICATION / GATE_STRICT_SKIPS
# all agree. Anything not listed is OFF, and that direction matters: a misspelled value
# (CLUSTER_AVAILABLE=ture, REQUIRE_SPEC_VERIFICATION=yeah) leaves the guard OFF rather than
# silently declaring a capability "available", so a typo can only make the gate stricter about
# what it claims to have verified, never looser.
truthy() {
case "${1:-}" in
true|TRUE|True|1|yes|YES|y|on|ON) return 0 ;;
*) return 1 ;;
esac
}
# --- Cluster detection (drives SKIP vs run for L2/L3) -------------------------
# A cluster is 'available' only if explicitly DECLARED. We NEVER auto-assume a
# cluster in Phase 0 — silence means SKIP, not pass (HARD RULE 4).
# CLUSTER_AVAILABLE=true -> L2/L3 run and are REQUIRED green
# CLUSTER_AVAILABLE=false/unset -> L2/L3 SKIP (reported, never faked green)
# That DECLARATION is what makes an L2/L3 skip different IN KIND from every other skip —
# see "WHICH SKIPS BLOCK, AND WHY" in ci/gate.sh.
cluster_available() { truthy "${CLUSTER_AVAILABLE:-false}"; }
# --- Skip strictness: is this a CI runner, or a workstation? -------------------
# A SKIPPED stage is never green anywhere (HARD RULE 4), but its SEVERITY differs:
#
# CI the pinned toolchain is installed by .gitea/workflows/scripts/install-tools.sh
# before the gate runs, and every seam is on `main`. A stage that skips here is a
# DEFECT (a tool dropped from the installer, a seam that went unwired) -> BLOCK.
# local a missing pinned tool is a real, ordinary local condition -> the gate reports
# UNDETERMINED (exit 3): not green, not a pass, but distinguishable from a red.
#
# HOW STRICTNESS IS DECIDED — DECLARED first, detected only as a fallback:
#
# 1. GATE_STRICT_SKIPS=true|false is the AUTHORITY. .gitea/workflows/gate.yml sets `true` on
# the gate step, so on the one CI runner that matters strictness is a DECLARATION IN THE
# WORKFLOW, not an inference: the environment that KNOWS it is CI says so, and no
# env-var sniffing can get it wrong.
# 2. `auto` (the default — anywhere the caller said nothing) falls back to GITHUB_ACTIONS /
# CI, which are set by the runner ITSELF (Gitea's act_runner, as well as GitHub). We do
# NOT sniff hostnames, paths or tool availability: a heuristic that guesses "this looks
# like CI" is exactly the unreliable inference this gate must not contain.
#
# WHY A WRONG ANSWER CANNOT FAIL OPEN: both branches are non-zero and NEITHER is a pass. A CI
# run misread as local yields UNDETERMINED (exit 3) instead of BLOCKED (exit 1) — that still
# fails the job unless its caller explicitly tolerates 3, which gate.yml does only knowingly
# and only for the one declared-environmental skip (see its comment). A workstation misread as
# CI yields BLOCKED instead of UNDETERMINED — harsher, never looser. Set GATE_STRICT_SKIPS=true
# locally to rehearse CI strictness.
: "${GATE_STRICT_SKIPS:=auto}"
strict_skips() {
case "$GATE_STRICT_SKIPS" in
auto) truthy "${GITHUB_ACTIONS:-}" || truthy "${CI:-}" ;;
*) truthy "$GATE_STRICT_SKIPS" ;;
esac
}
mkartifacts() { mkdir -p "$JUNIT_DIR" "$COVERAGE_DIR"; }
# --- Offline (L0/L1) per-check step summary -----------------------------------
# conformance/lib/run_l1_offline.py writes one Markdown line per check it executed to
# conformance/_artifacts/offline-summary.md. Splice those into $GITHUB_STEP_SUMMARY so the
# gate's output still NAMES every check that ran and its verdict.
#
# WHY THIS EXISTS: ci/l0.sh used to carry a hand-written stage per whole-repo check, each with
# its own `summary "- subst-safety: **passed**"` line. Those stages are gone (the tests declare
# `conformance.home.k3s/scope: whole-repo` instead and the runner unions them in), so the
# naming is generated for EVERY offline check rather than for the three that had a stage.
: "${OFFLINE_SUMMARY_FILE:=${CONFORMANCE_DIR}/_artifacts/offline-summary.md}"
emit_offline_summary() {
local f="${1:-$OFFLINE_SUMMARY_FILE}" line
[ -f "$f" ] || return 0
while IFS= read -r line; do
[ -n "$line" ] && summary " $line"
done < "$f"
return 0
}
# --- Touched-requirement subset (R-TEST-03 input) ----------------------------
# Sets: TOUCHED_RC (0 ok / 3 not determinable), TOUCHED_REQS[] and REQ_ARGS[]
# (ready to splice into a conformance/run.sh invocation).
@@ -111,7 +174,16 @@ load_touched_requirements() {
warn " Running the FULL layer as a superset sanity net; the SCOPED R-TEST-03"
warn " coverage check is NOT performed here — ci/gate.sh reports that as UNDECIDED."
elif [ "$rc" -ne 0 ]; then
warn "touched-requirements.sh exited ${rc}; proceeding with the full layer."
# Any OTHER non-zero exit means the subset is UNUSABLE, not empty (today: a
# path->requirement map that cannot classify a changed path, once that map fails closed).
# The FULL layer still runs here as a superset — nothing goes untested — but this stage must
# not claim the SCOPED check happened, and ci/gate.sh turns the same exit into a BLOCKED
# merge decision. Wording carried over from the unlanded fix/touched-requirements-fail-closed
# (47068ee), whose gate-side half this branch lands.
warn "touched-requirements.sh exited ${rc} (subset UNUSABLE — see its output above)."
warn " Running the FULL layer as a superset; the SCOPED R-TEST-03 check did NOT happen."
warn " ci/gate.sh BLOCKS the merge on this: fix the path->requirement map in"
warn " ci/touched-requirements.sh so every changed path is classified."
elif [ "${#TOUCHED_REQS[@]}" -gt 0 ]; then
log "touched requirements: ${TOUCHED_REQS[*]}"
REQ_ARGS=(--requirements "$(IFS=,; echo "${TOUCHED_REQS[*]}")")
+214
View File
@@ -0,0 +1,214 @@
#!/usr/bin/env python3
# =============================================================================
# ci/render-keycloak-client-urls.py
#
# Render platform/keycloak/client-urls.yaml FROM platform/keycloak/keycloak-realm-import.yaml.
#
# WHY THIS EXISTS
# ---------------
# The Keycloak realm import is a ONE-SHOT that fails OPEN: once it rests at
# Done=True / Started=False it never re-runs, so a redirect URI changed in git is
# never applied to the realm. The client-secret half of that failure class is
# already healed by a level-triggered CronJob (platform/keycloak/client-secret-
# reconciler.yaml). This renderer supplies the DESIRED STATE that the same CronJob
# needs to heal the URL half.
#
# THE CONSTRAINT THAT SHAPES IT: the reconciler pod runs with
# automountServiceAccountToken: false. It has NO Kubernetes API access at all, by
# design (R-SEC-02) — so it cannot read the KeycloakRealmImport CR to learn what
# the realm is supposed to look like. Its inputs must be MOUNTED. Hence a
# ConfigMap.
#
# A hand-authored ConfigMap would be a SECOND SOURCE OF TRUTH for every public
# hostname in the realm, free to disagree with the realm import — and "two
# declarations that can disagree" is the same shape as the bug this whole effort
# is closing. So the ConfigMap is GENERATED from the realm import and the
# generation is GATED: conformance/tests/l0-static/keycloak-client-secret-
# reconciler/check.sh runs this script with --check on every merge, and the merge
# fails if the committed file is not what this renderer produces. That is the same
# render==golden idiom catalog/render.py + ci/render-golden.sh already use for
# apps/.
#
# STRINGS ARE COPIED VERBATIM, NEVER RE-DERIVED. If a URL in the realm import is
# written as a Flux substitution token (https://grafana.${BASE_DOMAIN}/...), the
# token is carried through into the ConfigMap CHARACTER FOR CHARACTER. Both files
# are rendered by the SAME Flux Kustomization (./platform-config, whose
# postBuild.substituteFrom reads cluster-vars), so both are substituted in the
# same pass with the same values and cannot disagree about what BASE_DOMAIN is.
# This renderer never parses, splits, or reconstructs a hostname, and never reads
# cluster-vars — so there is no second place for the domain to be decided.
#
# USAGE
# python3 ci/render-keycloak-client-urls.py # print the rendered file
# python3 ci/render-keycloak-client-urls.py --write # write it (regenerate)
# python3 ci/render-keycloak-client-urls.py --check # exit 1 if it has drifted
# =============================================================================
from __future__ import annotations
import argparse
import difflib
import json
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("FATAL: PyYAML is required (same dependency as catalog/render.py)",
file=sys.stderr)
raise SystemExit(1)
ROOT = Path(__file__).resolve().parent.parent
IMPORT_FILE = "platform/keycloak/keycloak-realm-import.yaml"
OUT_FILE = "platform/keycloak/client-urls.yaml"
DATA_KEY = "client-urls.json"
CONFIGMAP_NAME = "keycloak-client-urls"
NAMESPACE = "keycloak"
# The client fields this renderer exports, and therefore the ONLY fields the
# reconciler may write. Every one of them is an environment-derived URL: it changes
# when BASE_DOMAIN changes, and a stale value breaks OIDC login (a wrong
# redirectUri) or silently widens trust (a stale wildcard left behind).
#
# DELIBERATELY ABSENT: `secret` (healed by the secret pass from the SOPS Secret,
# and it must never appear in a ConfigMap), and every non-URL field —
# protocolMappers, defaultClientScopes, attributes, flags. Those are not
# environment-derived, so nothing re-renders them on a domain flip; keeping them
# out keeps the reconciler's write surface as narrow as the problem.
LIST_FIELDS = ("redirectUris", "webOrigins")
STR_FIELDS = ("rootUrl", "baseUrl", "adminUrl")
def render(root: Path) -> str:
"""Return the full text of client-urls.yaml as it should be committed."""
docs = list(yaml.safe_load_all((root / IMPORT_FILE).read_text()))
imp = next((d for d in docs if d and d.get("kind") == "KeycloakRealmImport"), None)
if imp is None:
raise SystemExit("FATAL: no KeycloakRealmImport document in %s" % IMPORT_FILE)
clients = (imp.get("spec") or {}).get("realm", {}).get("clients") or []
if not clients:
# Fail closed. Emitting an empty desired state would make the reconciler
# believe every client is unmanaged and silently stop reconciling URLs.
raise SystemExit("FATAL: %s declares zero clients — refusing to render an "
"empty desired state" % IMPORT_FILE)
want: dict[str, dict] = {}
for c in clients:
cid = c.get("clientId")
if not isinstance(cid, str) or not cid:
raise SystemExit("FATAL: %s has a client with no clientId" % IMPORT_FILE)
entry: dict[str, object] = {}
for f in LIST_FIELDS:
if f not in c:
continue # not declared => NOT MANAGED (never forced empty)
v = c[f]
if not isinstance(v, list) or not all(isinstance(x, str) for x in v):
raise SystemExit("FATAL: %s client %r field %s is not a list of strings"
% (IMPORT_FILE, cid, f))
entry[f] = list(v) # verbatim, order as authored
for f in STR_FIELDS:
if f not in c:
continue
v = c[f]
if not isinstance(v, str):
raise SystemExit("FATAL: %s client %r field %s is not a string"
% (IMPORT_FILE, cid, f))
entry[f] = v
if entry:
want[cid] = entry
if not want:
raise SystemExit("FATAL: no client in %s declares any of %s — refusing to render "
"an empty desired state"
% (IMPORT_FILE, ", ".join(LIST_FIELDS + STR_FIELDS)))
body = json.dumps(want, indent=2, sort_keys=True, ensure_ascii=False)
indented = "\n".join((" " + ln) if ln else "" for ln in body.splitlines())
return HEADER + indented + "\n"
HEADER = """---
# =============================================================================
# GENERATED FILE — DO NOT EDIT BY HAND.
#
# source: platform/keycloak/keycloak-realm-import.yaml (spec.realm.clients)
# regenerate: python3 ci/render-keycloak-client-urls.py --write
# gated by: conformance/tests/l0-static/keycloak-client-secret-reconciler
# (runs this renderer with --check on every merge)
#
# WHAT THIS IS: the DESIRED STATE for every environment-derived client URL in the
# `homelab` realm — redirectUris, webOrigins, rootUrl, baseUrl, adminUrl. The
# client-secret reconciler CronJob mounts it and makes the live realm match it.
#
# WHY IT IS A SEPARATE OBJECT: the realm import is a ONE-SHOT that never re-runs
# once Done=True, so a redirect URI edited in git never reaches the realm. Prod's
# stored redirect URIs were measured still pointing at the pre-migration hostname
# while git declared the new one, and every OIDC login would have failed with a
# redirect_uri mismatch. The reconciler is the level-triggered loop that closes
# that; this file is what it reconciles TOWARDS.
#
# WHY IT IS GENERATED AND NOT AUTHORED: a hand-written copy of every public
# hostname would be a second source of truth, free to disagree with the realm
# import. It is rendered from the import and the render is gated, so the two
# cannot drift. URL strings are copied VERBATIM — if the source uses a
# ${BASE_DOMAIN} token it appears here unchanged, and both files are substituted
# by the SAME Flux Kustomization (./platform-config), so the domain is decided in
# exactly one place.
#
# A field that is NOT declared on a client in the realm import is simply ABSENT
# here, and the reconciler leaves it alone. Absence means UNMANAGED, never
# "force to empty".
# =============================================================================
apiVersion: v1
kind: ConfigMap
metadata:
name: %s
namespace: %s
labels:
app.kubernetes.io/name: %s
app.kubernetes.io/part-of: keycloak
data:
%s: |
""" % (CONFIGMAP_NAME, NAMESPACE, CONFIGMAP_NAME, DATA_KEY)
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__)
g = ap.add_mutually_exclusive_group()
g.add_argument("--write", action="store_true", help="write %s" % OUT_FILE)
g.add_argument("--check", action="store_true",
help="exit 1 if the committed file has drifted from the source")
args = ap.parse_args()
want = render(ROOT)
target = ROOT / OUT_FILE
if args.write:
target.write_text(want, encoding="utf-8")
print("wrote %s" % OUT_FILE)
return 0
if args.check:
if not target.is_file():
print("CHECK FAIL: %s does not exist. Run: python3 "
"ci/render-keycloak-client-urls.py --write" % OUT_FILE, file=sys.stderr)
return 1
have = target.read_text(encoding="utf-8")
if have == want:
print("client-urls.yaml is in sync with %s" % IMPORT_FILE)
return 0
print("CHECK FAIL: %s has DRIFTED from %s — the reconciler would converge the "
"realm on a desired state the realm import does not declare.\n"
"Regenerate with: python3 ci/render-keycloak-client-urls.py --write\n"
% (OUT_FILE, IMPORT_FILE), file=sys.stderr)
for line in difflib.unified_diff(have.splitlines(True), want.splitlines(True),
fromfile="committed " + OUT_FILE,
tofile="rendered from " + IMPORT_FILE):
sys.stderr.write(line)
return 1
sys.stdout.write(want)
return 0
if __name__ == "__main__":
raise SystemExit(main())
+189
View File
@@ -0,0 +1,189 @@
#!/usr/bin/env bash
# ci/tests/gate-decision.test.sh — the merge gate's OWN decision table.
#
# WHY THIS SUITE EXISTS (2026-09-13). ci/gate.sh used to set BLOCK=1 for a FAILED stage but
# only `warn` for a SKIPPED one, so it printed the words "— not green" and then concluded
# `MERGE GATE: PASS` and exited 0. Observed live twice: `prom-rules` skipped for a missing
# promtool, and `l0` skipped for a missing pinned kyverno CLI — both read by automation as
# merge-approved. That is fail-open in the one component whose entire job is to fail closed,
# and nothing tested it: every other ci/tests suite covers the INPUT (touched-requirements),
# none covered the DECISION.
#
# WHAT IS UNDER TEST: the real ci/gate.sh and ci/lib.sh, copied into a throwaway CI_DIR beside
# STUB stage scripts whose exit codes this suite dictates (STAGE_RC_<stage>) and a stub
# touched-requirements.sh whose rc/reason it dictates (TR_RC / TR_REASON / TR_OUT). Stubbing the
# stages is the point: this asserts how the gate REASONS about pass/fail/skip, independently of
# what any stage happens to do on this machine — so it runs anywhere, with no tools, no cluster,
# no network, in milliseconds.
#
# THE INVARIANT: `MERGE GATE: PASS` + exit 0 means every stage that was supposed to run RAN and
# was GREEN. A stage that could not run is never a pass, and the exit code says so too.
#
# Run: ci/tests/gate-decision.test.sh (or `make -C ci test`)
set -uo pipefail
REAL_CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
PASS=0 FAIL=0
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
bad() { printf ' FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
TMP="$(mktemp -d)"
trap 'rm -rf "$TMP"' EXIT
CI="${TMP}/ci"; mkdir -p "$CI"
REPO="${TMP}/repo"; mkdir -p "$REPO"
cp "${REAL_CI_DIR}/gate.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env" "$CI/"
# One stub per stage. It exits with STAGE_RC_<stage-with-dashes-as-underscores>, default 0.
for stage in spec-immutability lint l0 render-golden l1 conformance; do
cat > "${CI}/${stage}.sh" <<'STUB'
#!/usr/bin/env bash
n="$(basename "$0" .sh)"; v="STAGE_RC_${n//-/_}"
exit "${!v:-0}"
STUB
chmod +x "${CI}/${stage}.sh"
done
# Stub decision input. TR_RC (default 0), TR_REASON (written to $TOUCHED_REASON_FILE like the
# real script does), TR_OUT (the touched requirement list on stdout).
cat > "${CI}/touched-requirements.sh" <<'STUB'
#!/usr/bin/env bash
if [ -n "${TOUCHED_REASON_FILE:-}" ]; then printf '%s\n' "${TR_REASON:-}" > "$TOUCHED_REASON_FILE"; fi
[ -n "${TR_OUT:-}" ] && printf '%s\n' "$TR_OUT"
exit "${TR_RC:-0}"
STUB
chmod +x "${CI}/touched-requirements.sh"
# run_gate <VAR=VAL>... — run the gate with a clean, explicit environment. Sets OUT and RC.
# GATE_STRICT_SKIPS is pinned false unless a case says otherwise, so the suite never depends on
# whether the machine running it happens to look like CI.
OUT="" RC=0
run_gate() {
local summary_file="${TMP}/summary.md"; : > "$summary_file"
OUT="$(env -i PATH="$PATH" HOME="$HOME" \
REPO_ROOT="$REPO" GITHUB_STEP_SUMMARY="$summary_file" \
GATE_STRICT_SKIPS=false TR_OUT="R-TEST-03" \
"$@" bash "${CI}/gate.sh" 2>&1)"
RC=$?
SUMMARY="$(cat "$summary_file")"
}
# assert_verdict <headline> <expected-rc> <label>
assert_verdict() {
local want="$1" want_rc="$2" label="$3"
if ! grep -qF "MERGE GATE: ${want}" <<<"$OUT"; then
bad "${label}: headline should be '${want}' — got: $(grep -oE 'MERGE GATE: [A-Z ]+' <<<"$OUT" | tail -1)"
elif [ "$RC" -ne "$want_rc" ]; then
bad "${label}: headline '${want}' correct but exit code ${RC} != ${want_rc}"
else
ok "${label} -> ${want}, exit ${want_rc}"
fi
}
# The load-bearing negative: nothing may say PASS, and nothing may exit 0.
assert_not_pass() {
local label="$1"
if grep -qF "MERGE GATE: PASS" <<<"$OUT"; then bad "${label}: printed 'MERGE GATE: PASS'"
elif [ "$RC" -eq 0 ]; then bad "${label}: exited 0"
else ok "${label}: not a PASS, non-zero exit (${RC})"; fi
}
echo "== 1. the happy path: every stage green"
run_gate
assert_verdict PASS 0 "all stages green + a touched subset"
grep -qF "nothing was declared out of scope" <<<"$SUMMARY" \
&& ok "PASS summary states nothing was declared out of scope" \
|| bad "PASS summary does not state the absence of declared gaps"
echo
echo "== 2. a FAILED stage blocks (unchanged behaviour, locked down)"
for s in lint l0 render-golden l1 conformance spec-immutability; do
run_gate "STAGE_RC_${s//-/_}=1"
assert_verdict BLOCKED 1 "${s} FAILED"
done
echo
echo "== 3. THE REGRESSION: a SKIPPED stage is not a PASS and not exit 0"
for s in lint l0 render-golden l1; do
run_gate "STAGE_RC_${s//-/_}=3"
assert_not_pass "${s} SKIPPED (local)"
assert_verdict UNDETERMINED 3 "${s} SKIPPED (local)"
grep -qF "$s" <<<"$SUMMARY" && ok "${s}: the step summary names the stage that did not run" \
|| bad "${s}: step summary does not name the skipped stage"
done
echo
echo "== 4. under STRICT skips (CI declares it) a could-not-run skip BLOCKS, never merely 3"
for s in lint l0 render-golden l1; do
run_gate GATE_STRICT_SKIPS=true "STAGE_RC_${s//-/_}=3"
assert_verdict BLOCKED 1 "${s} SKIPPED with GATE_STRICT_SKIPS=true"
done
# ...which is what makes gate.yml's `|| [ $? -eq 3 ]` safe: in CI, exit 3 cannot be a missing
# tool or an unwired seam. Only the declared-environmental skip below can produce it.
run_gate GATE_STRICT_SKIPS=true STAGE_RC_l0=3 STAGE_RC_lint=3
assert_verdict BLOCKED 1 "several stages SKIPPED under strict skips"
echo
echo "== 5. spec-immutability: unverified is never green, but is not upgraded to a block by CI"
run_gate STAGE_RC_spec_immutability=3
assert_not_pass "spec-immutability SKIPPED"
assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED (local)"
run_gate GATE_STRICT_SKIPS=true STAGE_RC_spec_immutability=3
assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED under strict skips (deliberate exception)"
run_gate REQUIRE_SPEC_VERIFICATION=true STAGE_RC_spec_immutability=3
assert_verdict BLOCKED 1 "spec-immutability SKIPPED + REQUIRE_SPEC_VERIFICATION=true"
echo
echo "== 6. conformance L2/L3: a DECLARED-absent capability may still pass — and is NAMED"
run_gate STAGE_RC_conformance=3
assert_verdict PASS 0 "conformance SKIPPED, no cluster declared"
grep -qF "conformance-l2-l3" <<<"$OUT" \
&& ok "the PASS headline names the declared L2/L3 gap" \
|| bad "the PASS headline hides the declared L2/L3 gap"
grep -qF "conformance-l2-l3" <<<"$SUMMARY" \
&& ok "the step summary names the declared L2/L3 gap" \
|| bad "the step summary hides the declared L2/L3 gap"
run_gate CLUSTER_AVAILABLE=true STAGE_RC_conformance=3
assert_verdict BLOCKED 1 "conformance SKIPPED despite CLUSTER_AVAILABLE=true"
run_gate REQUIRE_CLUSTER_TESTS=true STAGE_RC_conformance=3
assert_verdict BLOCKED 1 "conformance SKIPPED + REQUIRE_CLUSTER_TESTS=true with a touched subset"
# A misspelled switch must leave the guard OFF (truthy()), never half-on.
run_gate CLUSTER_AVAILABLE=ture STAGE_RC_conformance=3
assert_verdict PASS 0 "CLUSTER_AVAILABLE=ture is NOT 'available' (no cluster was declared)"
run_gate CLUSTER_AVAILABLE=TRUE STAGE_RC_conformance=3
assert_verdict BLOCKED 1 "CLUSTER_AVAILABLE=TRUE IS a declaration (skip then blocks)"
echo
echo "== 7. the undetermined touched subset: unchanged (2026-08-31 false-green paths)"
run_gate TR_RC=3 TR_REASON=on-base TR_OUT=
assert_verdict "NOT APPLICABLE" 0 "subset undeterminable, reason on-base, all stages green"
run_gate TR_RC=3 TR_REASON=no-base TR_OUT=
assert_verdict BLOCKED 1 "subset undeterminable, reason no-base"
run_gate TR_RC=3 TR_REASON=diff-failed TR_OUT=
assert_verdict BLOCKED 1 "subset undeterminable, reason diff-failed"
# Precedence: a stage that did not run outranks "nothing to gate" — the stronger, non-zero
# headline must win, or a push-to-main run could hide a missing tool behind NOT APPLICABLE.
run_gate TR_RC=3 TR_REASON=on-base TR_OUT= STAGE_RC_l1=3
assert_not_pass "on-base AND l1 SKIPPED"
assert_verdict UNDETERMINED 3 "on-base AND l1 SKIPPED (UNDETERMINED outranks NOT APPLICABLE)"
echo
echo "== 8. an UNUSABLE subset blocks (carried over from 47068ee): any other non-zero exit"
for rc in 1 2 4; do
run_gate "TR_RC=${rc}" TR_REASON=unclassified-path TR_OUT=
assert_verdict BLOCKED 1 "touched-requirements.sh exited ${rc} (subset unusable, not empty)"
done
run_gate TR_RC=1 TR_REASON=unclassified-path TR_OUT=
grep -qF "UNUSABLE" <<<"$SUMMARY" && ok "the step summary calls the subset UNUSABLE" \
|| bad "the step summary does not report an unusable subset"
echo
echo "== 9. combinations: a FAILURE outranks a non-run, and the headline is the worst news"
run_gate STAGE_RC_l0=1 STAGE_RC_l1=3
assert_verdict BLOCKED 1 "l0 FAILED + l1 SKIPPED"
run_gate STAGE_RC_spec_immutability=3 STAGE_RC_conformance=3
assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED + conformance declared-absent"
echo
if [ "$FAIL" -eq 0 ]; then echo "gate-decision tests: PASS ($PASS assertions)"; exit 0
else echo "gate-decision tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi
+164
View File
@@ -0,0 +1,164 @@
#!/usr/bin/env bash
# ci/tests/touched-requirements-test-attribution.test.sh — regression tests for DETECTION
# RULE 3 in ci/touched-requirements.sh: a change anywhere under `conformance/tests/<test>/`
# attributes THAT TEST'S OWN `conformance.home.k3s/R-*` labels.
#
# THE GAP THESE LOCK DOWN (2026-09-12). The coarse path map keys `conformance/*` to
# R-TEST-01..04 — the requirements about the HARNESS — and says nothing about what a given
# test PROVES. So weakening `conformance/tests/.../check.sh` attributed R-TEST-01..04 and
# nothing else: the merge gate ran the harness's self-tests and NOT the weakened test's own
# requirement subset. Weakening (or deleting) a test is the one change that most needs the
# requirement it certifies to be re-verified, and it was the change least likely to select it.
# The content scan cannot close this: it reads only ADDED/REMOVED lines, so it sees a change
# to the LABEL BLOCK but not one to the test BODY — exactly the case that matters.
#
# Also covers the two path-map entries added in the same change (infra/* -> R-ENV-01, and
# R-TEST-06 on .gitea/*|ci/*), which is how the two L0 checks that are DELIBERATELY not
# whole-repo-scoped get selected by the paths that can break them.
#
# Same harness as the sibling suites: the real script, pointed at a THROWAWAY git repo via
# the REPO_ROOT override. No network, no cluster, the working repo is never touched.
#
# Run: ci/tests/touched-requirements-test-attribution.test.sh (or `make -C ci test`)
set -euo pipefail
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
SCRIPT="${CI_DIR}/touched-requirements.sh"
PASS=0 FAIL=0
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
bad() { printf ' FAIL %s\n' "$*" >&2; FAIL=$((FAIL+1)); }
assert_present() {
if grep -qxF "$1" <<<"$2"; then ok "$3"
else bad "$3 (missing '$1' from: $(tr '\n' ' ' <<<"$2"))"; fi
}
assert_absent() {
if grep -qxF "$1" <<<"$2"; then bad "$3 (found unexpected '$1' in: $(tr '\n' ' ' <<<"$2"))"
else ok "$3"; fi
}
TMP="$(mktemp -d)"
trap 'rm -rf "$TMP"' EXIT
git -C "$TMP" init -q -b main
git -C "$TMP" config user.email [email protected]
git -C "$TMP" config user.name t
git -C "$TMP" config commit.gpgsign false
# --- Baseline: two conformance tests, each declaring its own requirement labels ----------
# TEST A: a flat l0-static test. TEST B: a test with a NESTED fixture dir, to prove the
# walk-up from an arbitrary changed file to the enclosing test dir.
mkdir -p "$TMP/conformance/tests/l0-static/alpha" \
"$TMP/conformance/tests/l2-behavioral/beta/fixtures" \
"$TMP/conformance/lib" "$TMP/infra" "$TMP/ci"
cat > "$TMP/conformance/tests/l0-static/alpha/chainsaw-test.yaml" <<'YAML'
# HEADER PROSE. This test used to also cover conformance.home.k3s/R-STALE-09: "true",
# which is a REFERENCE in a comment, not a declaration, and must NOT be attributed.
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: alpha
labels:
conformance.home.k3s/R-ALPHA-01: "true"
conformance.home.k3s/R-ALPHA-02: "true"
annotations:
conformance.home.k3s/layer: "L0"
spec: {}
YAML
# The check BODY deliberately contains NO R-ID text, so anything the scan finds for a
# body-only edit came from rule 3 and not from the content scan.
cat > "$TMP/conformance/tests/l0-static/alpha/check.sh" <<'SH'
#!/usr/bin/env bash
echo "assert something"
SH
cat > "$TMP/conformance/tests/l2-behavioral/beta/chainsaw-test.yaml" <<'YAML'
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: beta
labels:
conformance.home.k3s/R-BETA-07: "true"
spec: {}
YAML
echo "seed: 1" > "$TMP/conformance/tests/l2-behavioral/beta/fixtures/data.yaml"
echo "harness helper" > "$TMP/conformance/lib/helper.sh"
git -C "$TMP" add -A && git -C "$TMP" commit -qm baseline
# ========================================================================================
echo "== case 1: a BODY-ONLY edit to a test attributes that test's own labels"
git -C "$TMP" checkout -q -b body-edit
cat > "$TMP/conformance/tests/l0-static/alpha/check.sh" <<'SH'
#!/usr/bin/env bash
echo "assert something weaker"
SH
git -C "$TMP" add -A && git -C "$TMP" commit -qm weaken
OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
printf 'touched set:\n%s\n---\n' "$OUT"
assert_present "R-ALPHA-01" "$OUT" "body-only test edit attributes the test's own R-ALPHA-01"
assert_present "R-ALPHA-02" "$OUT" "body-only test edit attributes the test's own R-ALPHA-02"
assert_present "R-TEST-03" "$OUT" "the conformance/* path map still applies (R-TEST-03)"
assert_absent "R-STALE-09" "$OUT" "an R-ID cited only in the test's HEADER COMMENT is NOT attributed"
assert_absent "R-BETA-07" "$OUT" "an UNRELATED test's labels are NOT attributed"
# ========================================================================================
echo "== case 2: an edit to a NESTED file walks up to the enclosing test"
git -C "$TMP" checkout -q -b nested-edit main
echo "seed: 2" > "$TMP/conformance/tests/l2-behavioral/beta/fixtures/data.yaml"
git -C "$TMP" add -A && git -C "$TMP" commit -qm fixture
OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
printf 'touched set:\n%s\n---\n' "$OUT"
assert_present "R-BETA-07" "$OUT" "nested fixture edit attributes the enclosing test's R-BETA-07"
assert_absent "R-ALPHA-01" "$OUT" "nested fixture edit does NOT attribute a sibling test's labels"
# ========================================================================================
echo "== case 3: DELETING a test still attributes what it used to certify"
# Removing the only verifier of a requirement must not be the cheapest way to make the gate
# stop checking it — so the labels are read from the BASE when HEAD no longer has them.
git -C "$TMP" checkout -q -b delete-test main
git -C "$TMP" rm -rq "$TMP/conformance/tests/l0-static/alpha"
git -C "$TMP" commit -qm "delete the alpha test"
OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
printf 'touched set:\n%s\n---\n' "$OUT"
assert_present "R-ALPHA-01" "$OUT" "deleting a test STILL attributes its R-ALPHA-01 (read from base)"
assert_present "R-ALPHA-02" "$OUT" "deleting a test STILL attributes its R-ALPHA-02 (read from base)"
# ========================================================================================
echo "== case 4: a change OUTSIDE conformance/tests/ gains no test labels"
git -C "$TMP" checkout -q -b lib-edit main
echo "harness helper v2" > "$TMP/conformance/lib/helper.sh"
git -C "$TMP" add -A && git -C "$TMP" commit -qm lib
OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
printf 'touched set:\n%s\n---\n' "$OUT"
assert_present "R-TEST-02" "$OUT" "a conformance/lib edit still attributes the harness requirements"
assert_absent "R-ALPHA-01" "$OUT" "a conformance/lib edit attributes NO individual test's labels"
assert_absent "R-BETA-07" "$OUT" "a conformance/lib edit attributes NO individual test's labels (B)"
# ========================================================================================
echo "== case 5: path-map entries for the two DELIBERATELY path-scoped L0 checks"
# infra/* -> R-ENV-01 (dev-nodes-provisioned-as-code reads nothing but infra/).
git -C "$TMP" checkout -q -b infra-edit main
printf 'resource "x" "y" {}\n' > "$TMP/infra/main.tf"
git -C "$TMP" add -A && git -C "$TMP" commit -qm infra
OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
printf 'touched set:\n%s\n---\n' "$OUT"
assert_present "R-ENV-01" "$OUT" "an infra/ edit attributes R-ENV-01 (was UNMAPPED: attributed nothing)"
# ci/* -> ... R-TEST-06 (spec-immutability-enforced reads nothing but ci/ + the spec repo).
git -C "$TMP" checkout -q -b ci-edit main
printf 'echo hello\n' > "$TMP/ci/thing.sh"
git -C "$TMP" add -A && git -C "$TMP" commit -qm ci
OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
printf 'touched set:\n%s\n---\n' "$OUT"
assert_present "R-TEST-06" "$OUT" "a ci/ edit attributes R-TEST-06 (the spec-immutability test's own ID)"
assert_present "R-GITOPS-01" "$OUT" "a ci/ edit still attributes R-GITOPS-01"
echo
if [ "$FAIL" -eq 0 ]; then
echo "touched-requirements test-attribution tests: PASS (${PASS} assertions)"; exit 0
fi
echo "touched-requirements test-attribution tests: FAIL (${FAIL} failed, ${PASS} passed)" >&2
exit 1
+43
View File
@@ -93,6 +93,49 @@ printf 'touched set (removal):\n%s\n---\n' "$OUT2"
assert_absent "R-DELDOC-01" "$OUT2" "REMOVED whole-line comment R-ID is NOT attributed"
assert_present "R-DELREAL-01" "$OUT2" "REMOVED real-line R-ID IS attributed"
# ---------------------------------------------------------------------------------------
# PATH MAP — bootstrap/* (added 2026-09-13)
#
# WHY THIS IS LOCKED DOWN. path_requirements() keys `bootstrap/*` to R-STATE-03 +
# R-GITOPS-01, and until 2026-09-13 the comment justifying it described
# `bootstrap/standup.sh` as a file that carries the placeholder guards. That file has never
# been on `main` (it landed only on unmerged prod-standup branches), so the comment was
# stale and the entry looked like dead weight someone would tidy away. It is not dead: it
# is what SELECTS conformance/tests/l0-static/placeholder-guard-coupling for the change
# that eventually lands a placeholder guard under bootstrap/ — and that test discovers its
# guard by content rather than by path, so it activates on exactly that change. Removing
# the entry would let the change introducing a guard be gated by nothing, which is the
# 2026-09-05 class. These assertions make the entry's deletion a test failure rather than a
# silent loss of selection.
#
# NOTE the R-ID content scan must not be what produces these: `notes/` is outside the path
# map precisely so the other cases isolate the scan, and here the file content cites no
# R-ID at all, so any R-ID in the output came from the PATH.
git -C "$TMP" checkout -q -b feature3 main
mkdir -p "$TMP/bootstrap/flux"
cat > "$TMP/bootstrap/flux/bootstrap.sh" <<'SH'
#!/usr/bin/env bash
# a bootstrap-path change that cites no requirement id of its own
echo "flux bootstrap"
SH
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch bootstrap/"
OUT3="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
printf 'touched set (bootstrap/ path map):\n%s\n---\n' "$OUT3"
assert_present "R-STATE-03" "$OUT3" "bootstrap/* attributes R-STATE-03 (selects placeholder-guard-coupling)"
assert_present "R-GITOPS-01" "$OUT3" "bootstrap/* attributes R-GITOPS-01 (it IS the GitOps bootstrap path)"
# And the neighbouring entry the same test leans on for its other half: clusters/* must
# attribute R-ENV-02, which is what selects placeholder-guard-coupling's HALF A (the
# authored-values sweep over clusters/<env> effective cluster-vars).
git -C "$TMP" checkout -q -b feature4 main
mkdir -p "$TMP/clusters/base"
printf 'kind: ConfigMap\nmetadata:\n name: cluster-vars\n' > "$TMP/clusters/base/cluster-vars.yaml"
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch clusters/"
OUT4="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
printf 'touched set (clusters/ path map):\n%s\n---\n' "$OUT4"
assert_present "R-ENV-02" "$OUT4" "clusters/* attributes R-ENV-02 (selects placeholder-guard-coupling HALF A)"
assert_present "R-GITOPS-01" "$OUT4" "clusters/* attributes R-GITOPS-01"
echo
if [ "$FAIL" -eq 0 ]; then echo "touched-requirements tests: PASS ($PASS assertions)"; exit 0;
else echo "touched-requirements tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi
+123 -2
View File
@@ -21,8 +21,12 @@
# the R-IDs that policies/ServiceDefinitions self-declare (docs/14 + ADR-0011).
# Untouched lines and prose are NOT scanned; see changed_diff() for why.
# 2. A coarse path->requirement fallback for changes that don't self-declare IDs.
# 3. For a change anywhere under `conformance/tests/<test>/`: THAT TEST'S OWN requirement
# labels, read from its chainsaw-test.yaml. See test_requirements_for_path().
# The coarse map (2) NEVER overrides an explicit tag; it only widens coverage for
# untagged changes so nothing silently escapes the gate.
# untagged changes so nothing silently escapes the gate. (3) is the same principle applied
# to the tests themselves: editing or weakening a test must be gated by the requirements
# that test CERTIFIES, not merely by R-TEST-0x.
set -euo pipefail
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck disable=SC1091
@@ -180,14 +184,128 @@ path_requirements() {
platform/externaldns/*) echo "R-DNS-02" ;;
platform/velero/*) echo "R-STATE-03" ;;
platform/monitoring/*) echo "R-OBS-01 R-OBS-02" ;;
# platform/cnpg was UNMAPPED until 2026-09-06. It is the PostgreSQL DR path for keycloak-db /
# nextcloud-db / semaphore-db (barmanObjectStore WAL archiving + ScheduledBackup), so a change
# there is an R-IDP-05 / R-STATE-03 change; it now also carries the platform-health alert rules
# for that path (R-OBS-04). Without this line an edit under platform/cnpg/ contributed ZERO
# requirements unless it happened to retype an R-ID on a non-comment line, and escaped the gate.
platform/cnpg/*) echo "R-IDP-05 R-STATE-03 R-OBS-04" ;;
# vpn-egress was UNMAPPED until 2026-09-06. platform/vpn-egress/egress-steering.yaml IS the
# R-NET-04 datapath (the rule that selects between the two egress paths) and the R-NET-05/06/07
# steering policy — it even self-declares those IDs in its `homelab.keithjackson.dev/requirements`
# annotation. But the content scan only reads ADDED/REMOVED lines, so any edit that does not
# happen to retype an R-ID (a selector change, an excludedCIDRs change, a comment) contributed
# ZERO requirements and escaped the gate entirely. Mapped explicitly; the annotation still wins
# when a diff does carry the tags.
platform/vpn-egress/*) echo "R-NET-04 R-NET-05 R-NET-06 R-NET-07" ;;
# Keycloak is the whole SSO surface: the realm import (declarative identity config),
# the SOPS client secrets, and the reconciler that heals them. Added 2026-09-06 after
# the client-secret incident, in which editing platform/keycloak/ attributed NO
# requirement, so the L2 keycloak tests were filtered out of the merge gate's touched
# subset — the change that could break every OIDC login was gated by nothing.
# R-IDP-05 is in scope because platform/keycloak/ also carries the CNPG DB Cluster and
# its off-cluster backup credential, not only the realm.
platform/keycloak/*) echo "R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06" ;;
platform/oauth2-proxy/*) echo "R-IDP-02 R-IDP-06" ;;
platform/image-automation/*) echo "R-UPD-01 R-UPD-02" ;;
conformance/*) echo "R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04" ;;
# infra/ was UNMAPPED until 2026-09-12 — the OpenTofu + Ansible that provision the dev
# nodes. conformance/tests/l0-static/dev-nodes-provisioned-as-code reads NOTHING but
# infra/, yet an infra/ edit attributed ZERO requirements, so the one check that asserts
# the nodes are provisioned as code was filtered out of every change that could break it.
# That test's invariant IS path-scoped (it is deliberately NOT whole-repo-scoped; see its
# chainsaw-test.yaml), and the correct fix for a path-scoped invariant is a path-map entry.
infra/*) echo "R-ENV-01" ;;
clusters/*) echo "R-GITOPS-01 R-ENV-02" ;;
.gitea/*|ci/*) echo "R-GITOPS-01 R-TEST-02 R-TEST-03" ;;
# bootstrap/ was UNMAPPED until 2026-09-06. The entry STAYS, but its original rationale
# was written against a file that is not here, and that is corrected on 2026-09-13:
# it claimed "bootstrap/standup.sh carries the placeholder guards that decide whether
# `velero` may be enabled against a real backup endpoint". bootstrap/standup.sh HAS NEVER
# BEEN ON `main` — it was not deleted (no deletion commit exists anywhere), it landed only
# on unmerged prod-standup branches. `git ls-tree main bootstrap/` lists README.md, flux/,
# gitea/, sops/. So R-STATE-03 here is ANTICIPATORY, not descriptive.
#
# WHY IT IS STILL RIGHT. conformance/tests/l0-static/placeholder-guard-coupling DISCOVERS
# its guard (any tracked *.sh defining `<name>_is_placeholder()`) instead of hardcoding a
# path, so the moment a standup guard lands under bootstrap/ that test's coupling half
# activates — and this entry is what selects the test for the change that lands it.
# Deleting the entry would mean the change introducing a placeholder guard is gated by
# nothing, which is the 2026-09-05 class (a prefix glob that stopped matching when the
# value it guards changed, in a different file) left to reintroduce itself. R-GITOPS-01 is
# descriptive today: bootstrap/flux/, gitea/ and sops/ ARE the GitOps bootstrap path.
# Locked down by ci/tests/touched-requirements.test.sh's path-map section.
bootstrap/*) echo "R-STATE-03 R-GITOPS-01" ;;
# R-TEST-06 added 2026-09-12. The comment in changed_diff()'s rule 3 above records why it
# was ABSENT: R-TEST-06 used to be enforced by the spec-immutability STAGE and carried by no
# Chainsaw test, so attributing it made the coverage `--require` check false-block. It is now
# carried by conformance/tests/l0-static/spec-immutability-enforced — whose entire input
# surface is ci/ — so a ci/ change must attribute it, or the test that proves the
# spec-immutability mechanism is still load-bearing runs on nothing that can break it.
.gitea/*|ci/*) echo "R-GITOPS-01 R-TEST-02 R-TEST-03 R-TEST-06" ;;
*) : ;;
esac
}
# ---------------------------------------------------------------------------------
# A CHANGED TEST ATTRIBUTES THE REQUIREMENTS IT CERTIFIES (detection rule 3).
#
# THE GAP THIS CLOSES (2026-09-12). The path map keys `conformance/*` to
# R-TEST-01..04 — the requirements about the HARNESS. It says nothing about what a given
# test PROVES. So editing `conformance/tests/l2-behavioral/tenant-isolation/check.sh` to
# weaken or delete an assertion attributed R-TEST-01..04 and nothing else: the merge gate
# then ran the harness's self-tests and NOT the weakened test's own requirement subset.
# Weakening a test is the one change that most needs the requirement it certifies to be
# re-verified, and it was the change least likely to select it.
#
# The `conformance.home.k3s/R-*: "true"` labels on a test ARE the authoritative declaration of
# what it verifies (lib/coverage.py reads the same labels); this reads them directly. Note the
# content scan in main() only sees ADDED/REMOVED lines, so it catches a change to the LABEL
# BLOCK but not a change to the test BODY — which is exactly the case that matters here.
#
# WHY IT READS HEAD *OR* THE BASE: a change that DELETES a test, or deletes its
# chainsaw-test.yaml, must still attribute what that test used to certify — otherwise removing
# the only verifier of a requirement is the cheapest way to make the gate stop checking it.
# HEAD first (the post-change truth), base as the fallback (the deleted case). Fail-closed:
# this function can only ADD requirements, never remove one.
: "${CONF_TEST_LABEL_PREFIX:=conformance.home.k3s}" # == CONF_LABEL_PREFIX, conformance/lib/common.sh
file_text_head_or_base() {
local path="$1"
git -C "$REPO_ROOT" show "HEAD:${path}" 2>/dev/null && return 0
git -C "$REPO_ROOT" show "${BASE}:${path}" 2>/dev/null && return 0
return 0
}
# test_requirements_for_path <changed-path> -> space-separated R-IDs (possibly empty).
# Walks UP from the changed file to the nearest ancestor directory carrying a
# chainsaw-test.yaml, so it works for any file in a test dir (check.sh, mutations.py,
# fixtures, the chainsaw-test.yaml itself) and for nested test layouts.
test_requirements_for_path() {
local f="$1" dir yaml text
case "$f" in conformance/tests/*/*) ;; *) return 0 ;; esac
dir="${f%/*}"
while :; do
case "$dir" in
conformance/tests|conformance|.|"") return 0 ;;
*/*) ;;
*) return 0 ;;
esac
yaml="${dir}/chainsaw-test.yaml"
text="$(file_text_head_or_base "$yaml")"
if [ -n "$text" ]; then
# Whole-line `#` comments are dropped first — same "reference, not declaration" rule the
# content scan applies (changed_diff() rule 3): a label shown inside a header example is
# documentation. Only a real YAML mapping line counts.
printf '%s\n' "$text" \
| grep -vE '^[[:space:]]*#' \
| sed -nE "s#^[[:space:]]*${CONF_TEST_LABEL_PREFIX}/(R-[A-Z0-9]+-[0-9]+):[[:space:]]*[\"']?true.*#\1#p" \
| tr '\n' ' '
return 0
fi
dir="${dir%/*}"
done
}
main() {
local files reqs="" rc=0 base_sha head_sha local_main merge_base
@@ -221,6 +339,9 @@ main() {
while IFS= read -r f; do
[ -n "$f" ] || continue
reqs+=" $(path_requirements "$f")"
# Detection rule 3: a change under conformance/tests/<test>/ attributes THAT TEST'S own
# requirement labels, so weakening a test is gated by what the test certifies.
reqs+=" $(test_requirements_for_path "$f")"
done <<< "$files"
# Self-declared R-IDs on the ADDED/REMOVED lines of the change (see changed_diff).
+9
View File
@@ -21,6 +21,15 @@ YAMLLINT_VERSION="1.35.1"
KUBECTL_VERSION="1.36.4"
YQ_VERSION="4.44.6"
# Alert-rule unit tests (promtool test rules) — the L0 check
# conformance/tests/l0-static/alert-rules-unit-tested runs every
# platform/**/*.promtool-test.yaml through it. MUST stay in lockstep with
# infra/devcontainer/versions.env (which bakes it into the image) AND with the Prometheus
# version running on prod, because promtool IS the rule engine: verified 2026-09-12 as
# v3.14.0-distroless via `kubectl get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'`.
# Shipped inside the prometheus release tarball, not as a separate release.
PROMTOOL_VERSION="3.14.0"
# ServiceDefinition JSON-Schema validation (schemas/validate.sh). Pip-installed;
# pinned here so CI and a laptop validate against the exact same validator.
CHECK_JSONSCHEMA_VERSION="0.29.4"
+21
View File
@@ -97,6 +97,27 @@ Kustomization: apps-wave-1 (./apps/wave-1, dependsOn apps-wave-0)
Kustomization: apps-wave-N (./apps/wave-N, dependsOn apps-wave-N-1)
```
### Second-order CRDs need a THIRD pass (`platform-config-cdi`, added 2026-09-05)
The controllers/configs split (`platform` → `platform-config`) assumes a CRD is registered by the
controller **install**. CDI breaks that: `cdi-operator.yaml` registers only `cdis.cdi.kubevirt.io`,
and cdi-operator creates `storageprofiles` / `datavolumes` / `datasources` **only after it has
reconciled the `CDI` CR**. Since kustomize-controller server-side **dry-runs every object in a pass
before applying any of them**, having the `CDI` CR and the CDI `StorageProfile`s in one pass was a
**deadlock, not a race**: unknown kind → pass aborts → CR never applied → CRD never created →
`platform-config` can never converge → every tier below it blocked. Invisible on dev (its CDI CRDs
have existed for months); it cost 12h on prod's cold start.
So there is a third tier, `platform-config-cdi` (`./platform-config-cdi`, `dependsOn:
platform-config`), and a rule:
> A CR belongs in `platform-config-cdi`, not `platform-config`, when the CRD it needs is created by a
> **controller at runtime** rather than by a manifest in git.
`apps-vms` `dependsOn` it as well as `platform-config`, because CDI reads
`StorageProfile/longhorn-vm` at DataVolume **provisioning** time — a VM disk provisioned before that
profile exists is stamped RWO/Filesystem and is permanently not live-migratable.
Wave 0 `dependsOn` `platform-config` (transitively `platform`) so CNI/storage/TLS/DNS/gateway
exist before any workload reconciles. The app tier is split into dependency-ordered WAVES
(`catalog/render.py --emit-waves`): each `apps-wave-<N>` health-gates (wait:true) ONLY its own
+27 -2
View File
@@ -4,8 +4,11 @@
# NON-GATING (wait:false): unlike the app waves, this does NOT health-wait. A VM's first-boot image
# import (CDI, minutes) + in-guest software install mean a VM can be intentionally long-not-Ready, so
# health-gating it would wedge reconciliation for no benefit. VMs have no in-cluster dependents.
# dependsOn platform-config so KubeVirt/CDI + the longhorn-vm StorageClass + CDI StorageProfiles
# exist first. prune:true — this Kustomization solely owns the VM manifests (no adoption concern).
# dependsOn platform-config so KubeVirt/CDI + the longhorn-vm StorageClass exist first, AND
# platform-config-cdi so the CDI StorageProfiles do (2026-09-05: the StorageProfiles MOVED OUT of
# platform-config into their own tier, so `dependsOn: platform-config` alone NO LONGER guarantees
# them — see the dependsOn block). prune:true — this Kustomization solely owns the VM manifests (no
# adoption concern).
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
@@ -19,6 +22,28 @@ spec:
wait: false
dependsOn:
- name: platform-config
# LOAD-BEARING, not decoration. apps/vms/unifi/20-datavolume.yaml provisions a DataVolume on
# storageClassName `longhorn-vm`, and CDI reads StorageProfile/longhorn-vm AT PROVISIONING TIME to
# decide the claim's accessModes/volumeMode. Without that profile's `claimPropertySets`
# (ReadWriteMany + Block) the PVC is stamped RWO/Filesystem and the VM disk is PERMANENTLY not
# live-migratable — a silent, unrecoverable-without-recreate loss. The profiles used to arrive
# with platform-config; since 2026-09-05 they arrive with `platform-config-cdi`, so the edge has
# to move with them. platform-config-cdi runs wait:true and StorageProfile has no status
# conditions (kstatus -> Current on apply), so this edge cannot stall this tier.
#
# HONEST LIMIT: CDI provisions from the profile's STATUS, which its storageprofile-controller
# derives from our spec. This edge proves the spec is applied, not that CDI has folded it into
# status — that is a watch-triggered round-trip of seconds against a ~30s dependency requeue, so
# it is a timing argument rather than a guarantee. See clusters/base/platform-config-cdi.yaml.
#
# This edge also makes the StorageProfile prune/recreate transition safe: platform-config's prune
# deletes the two profiles for the ~30-60s before platform-config-cdi re-applies them, and this
# dependsOn is what guarantees Flux cannot create a DataVolume inside that window.
#
# The platform-config edge above is kept although platform-config-cdi already dependsOn it: the
# edges are a set, there is no cycle, and stating both means a future edit to platform-config-cdi
# cannot silently drop this tier's KubeVirt/StorageClass gate.
- name: platform-config-cdi
sourceRef:
kind: GitRepository
name: flux-system
+6 -3
View File
@@ -34,6 +34,9 @@ spec:
substituteFrom:
- kind: ConfigMap
name: cluster-vars
# SOPS decryption for velero-cloud.sops.yaml (the scoped MinIO `cloud` credential) is layered per
# environment by the overlay (clusters/dev patches THIS Kustomization by name), identical to how
# headlamp/oauth2-proxy/crafty decrypt their own SOPS secrets. See clusters/dev/kustomization.yaml.
# SOPS decryption for the `cloud` INI credentials — velero-cloud.sops.yaml (dev's scoped MinIO key)
# and velero-cloud-prod.sops.yaml (prod's Cloudflare R2 token) — is layered per environment by the
# overlay (clusters/dev and clusters/prod patch THIS Kustomization by name), identical to how
# headlamp/oauth2-proxy/crafty decrypt their own SOPS secrets. Both Secrets are applied on both
# clusters; ${BACKUP_CREDENTIALS_SUFFIX} (substituted from cluster-vars above) decides which one the
# velero Deployment actually mounts. See clusters/dev/kustomization.yaml.
+60
View File
@@ -0,0 +1,60 @@
---
# Flux Kustomization: CILIUM (the CNI) — reconciles ./platform/cilium.
# Serves R-PLAT-01 (platform components reconciled by GitOps) and R-GITOPS-01/03.
#
# WHY CILIUM HAS ITS OWN KUSTOMIZATION (split out of `platform`, 2026-09-01)
#
# This is the MIRROR IMAGE of the oauth2-proxy / headlamp splits, not a repeat of them. Those are
# leaves that must NEVER gate a tier, so they left `platform` and run `wait: false` below it.
# Cilium is the one component that legitimately SHOULD gate everything — but inside the shared
# `platform` Kustomization its health was indistinguishable from eleven unrelated components:
#
# * `platform`'s Ready condition could not tell you "the CNI is down" from "longhorn is slow".
# * The only emergency valve was `flux suspend kustomization platform` — freezing the ENTIRE
# tier (storage, TLS, DNS, ingress, monitoring, Keycloak, KubeVirt) to stop touching the CNI.
# `flux suspend kustomization cilium` now freezes exactly the CNI and nothing else, which is
# the valve the 2026-09-01 substitution incident wanted and did not have.
# * `platform`'s 10m interval + 5m timeout are wrong for a CNI: a 3-node DaemonSet rollout with
# `wait: true` can exceed 5m, and re-rendering the CNI every 10m buys nothing.
#
# GATING IS PRESERVED, NOT WEAKENED: `wait: true` here plus `platform.dependsOn: [cilium]` means
# nothing in the graph reconciles until the CNI is healthy — the same guarantee as before, now
# expressed as an explicit edge instead of an accident of co-tenancy.
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: cilium
namespace: flux-system
spec:
# A CNI does not need a 10m re-render cadence. A git push still reconciles immediately —
# kustomize-controller is source-triggered; `interval` is only the drift-correction floor.
interval: 1h
retryInterval: 2m
# 10m (not the 5m `platform` uses): a 3-node Cilium DaemonSet rollout under `wait: true`,
# rolled ONE NODE AT A TIME (maxUnavailable: 1, P2), legitimately exceeds 5m.
timeout: 10m
prune: true
# TRUE, deliberately: this is the ONE component whose health should gate the whole platform.
wait: true
# Same revision-gating reason `platform` has it (2026-09-01 incident): `cluster-vars` is APPLIED
# by the root `flux-system` Kustomization, and this Kustomization SUBSTITUTES from it. `dependsOn`
# is revision-gated (kustomize-controller v1.4.0 `checkDependencies`), so this cannot render
# revision N of helmrelease.yaml against the revision N-1 `cluster-vars`. That race is exactly how
# `k8sServiceHost: ""` once reached the Cilium HelmRelease and took down 2/3 agents.
dependsOn:
- name: flux-system # the ROOT Kustomization (sync.yaml) that applies cluster-vars
sourceRef:
kind: GitRepository
name: flux-system
path: ./platform/cilium
# MANDATORY, not decorative: platform/cilium/helmrelease.yaml references ${K3S_API_HOST}.
# kustomize-controller now runs --feature-gates=StrictPostBuildSubstitutions=true, so without
# this stanza an undefined ${K3S_API_HOST} FAILS this Kustomization outright (it no longer
# renders empty). That is the intended fail-closed behaviour — but it means the stanza is
# load-bearing for the CNI coming up at all. Do not remove it.
postBuild:
substituteFrom:
- kind: ConfigMap
name: cluster-vars
# No `decryption` stanza: platform/cilium carries no SOPS-encrypted resource (unlike
# oauth2-proxy/headlamp/crafty/velero, whose overlays patch one in by name).
+243 -11
View File
@@ -17,9 +17,63 @@ data:
CLUSTER_ENV: "base" # dev | prod (overlays set this)
# --- DNS / TLS ---
BASE_DOMAIN: "home.keithjackson.dev" # platform hostname suffix (same both envs; split-horizon)
DNS_ZONE: "keithjackson.dev" # the actual Cloudflare ZONE (BASE_DOMAIN is a subdomain of it);
# ExternalDNS domain-filter must be the zone, not the subdomain
# THE platform hostname suffix. Every public hostname the cluster serves is `<svc>.${BASE_DOMAIN}`,
# the wildcard cert is `*.${BASE_DOMAIN}`, coredns-custom answers the `${BASE_DOMAIN}` zone with
# ${GATEWAY_VIP}, and every OIDC issuer/redirect URI is built from it.
#
# THIS KNOB WAS DEAD UNTIL 2026-09-04 — declared here but referenced by NO manifest, because every
# hostname was hardcoded. That is what made the 2026-09-04 incident possible: prod and dev were
# authored to serve the SAME hostnames out of the SAME Cloudflare zone, so prod's external-dns
# (policy: sync, owner `default`) concluded it owned dev's public records and DELETED ALL OF THEM.
# See infra/PROD-PROVISIONING.md §10. The fix is architectural: each environment gets its OWN
# subdomain, so the two clusters can no longer describe the same name.
#
# dev -> home.keithjackson.dev (UNCHANGED — see below)
# prod -> prod.home.keithjackson.dev
#
# DEV DELIBERATELY STAYS ON THE APEX `home.keithjackson.dev`. The target architecture puts dev on
# `dev.home.keithjackson.dev` with apex CNAMEs `<svc>.home` -> `<svc>.prod.home`, but moving dev
# relocates its Keycloak issuer, every OIDC client redirect URI, kubectl SSO, its wildcard cert and
# the Gitea host that IS the GitOps source — on a live cluster. Deferred to phase 2
# (infra/PROD-PROVISIONING.md §10). Base default is therefore the DEV value, the same convention as
# GATEWAY_VIP / LONGHORN_DATA_PATH: the base default is the known-good working value, dev restates
# it verbatim, and prod overrides it. Dev's rendered output is byte-identical to the pre-
# tokenization hardcoded manifests (verified 2026-09-04, all 12 substituted paths).
BASE_DOMAIN: "home.keithjackson.dev"
# ExternalDNS `--domain-filter` (platform/externaldns/helmrelease.yaml). SAFETY NET #2 behind
# EXTERNALDNS_TXT_OWNER_ID: a record whose name falls outside this filter is not even considered,
# so it cannot be created, updated or deleted regardless of TXT ownership.
#
# DEV KEEPS THE ZONE APEX `keithjackson.dev`, and must: dev serves the `home.keithjackson.dev`
# apex itself, and the Cloudflare provider filters the ACCOUNT'S ZONES by this value — a filter
# that is a strict subdomain of the zone dropped the parent zone outright ("no hosted zone matching
# record DNS Name", DECISIONS-LOG 2026-08) on the version in use then. So dev's value is unchanged
# from the retired `DNS_ZONE` knob this replaces, and dev renders byte-identically.
#
# PROD NARROWS TO ITS OWN SUBTREE `prod.home.keithjackson.dev`. VERIFY ON PROD'S FIRST RECONCILE
# (infra/PROD-PROVISIONING.md §10): if the zone-lookup limitation above still applies to the pinned
# chart, prod's external-dns will select NO zone and create NO records. That failure is FAIL-CLOSED
# — it can never delete another environment's records — and the fallback is to set prod's value
# back to "keithjackson.dev", leaving EXTERNALDNS_TXT_OWNER_ID as the sole (and sufficient) guard.
EXTERNALDNS_DOMAIN_FILTER: "keithjackson.dev"
# ExternalDNS TXT registry owner id (`--txt-owner-id`). THE ROOT CAUSE OF THE 2026-09-04 INCIDENT:
# neither cluster set it, so both inherited the chart default `default`, and ExternalDNS treats a
# record whose TXT registry entry carries ITS OWN owner id as its own to reconcile — and with
# `policy: sync`, to DELETE when no local resource claims it. Two clusters sharing an owner id on
# one zone therefore garbage-collect each other. Each environment MUST have a distinct value.
#
# DEV MUST RENDER EXACTLY `default`. Dev's LIVE public records are already registered in Cloudflare
# under owner `default`; changing dev's value would make dev's own external-dns stop recognising
# them, and it would re-create duplicates alongside the orphaned originals. Dev's overlay pins it.
#
# THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default`. A new environment that
# forgets this knob must not silently inherit dev's identity — that is precisely the incident. With
# this sentinel such a cluster owns nothing, so it can DELETE nothing; it would publish under a
# visibly wrong owner id (grep the TXT records) instead of destroying a live environment's DNS.
EXTERNALDNS_TXT_OWNER_ID: "unset-see-cluster-vars"
ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006)
GATEWAY_VIP: "192.168.1.53" # in-cluster split-horizon target (R-DNS-01/03): the L2-announced
# ingress VIP of the `shared` Gateway. MUST equal the FIRST address of
@@ -42,15 +96,55 @@ data:
#
# Substitution coverage (verify before adding a knob here): every path that consumes these declares
# postBuild.substituteFrom cluster-vars — platform (K3S_API_HOST), platform-config
# (CILIUM_LB_POOL_*, S3_BACKUP_ENDPOINT), crafty (CRAFTY_L4_VIP), apps-wave-0 (S3_BACKUP_ENDPOINT),
# velero (S3_BACKUP_ENDPOINT), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an UNSUBSTITUTED path would
# ship the literal string into the cluster.
# (CILIUM_LB_POOL_*, and keycloak-db's S3_BACKUP_ENDPOINT + CNPG_BACKUP_PATH: the Cluster lives at
# platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
# tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
# S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH), velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
# VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
# UNSUBSTITUTED path would ship the literal string into the cluster.
#
# ADDING A KNOB IS SAFE UNDER StrictPostBuildSubstitutions ONLY BECAUSE OF THE ROOT EDGE. An
# UNDEFINED variable is a HARD Kustomization failure, so a manifest that references a knob the
# live ConfigMap does not yet carry breaks its whole tier. `clusters/base/platform.yaml` gained
# `dependsOn: [flux-system]` for exactly this: dependsOn is REVISION-gated, so no substituting
# tier can reconcile revision N until the ROOT Kustomization has APPLIED revision N — i.e. until
# the new cluster-vars key EXISTS. Every substituting consumer chains transitively through
# `platform`, so one edge covers the graph, and a new knob plus its first reference may ship in
# ONE commit. (A hand-run `kubectl apply -k clusters/<env>` bypasses that ordering entirely: apply
# the overlay BEFORE the tier that references the knob, or the tier fails until it retries.)
# k3s API endpoint Cilium's kube-proxy replacement dials (platform/cilium/helmrelease.yaml).
# [parity] MUST equal the Ansible bootstrap's k3s_api_host (infra/ansible/roles/cilium/defaults/
# main.yml -> cilium-values.yaml.j2, already inventory-parameterized). Wrong value = Cilium cannot
# reach the API server on adopt and the CNI breaks. Port stays 6443 in both envs, so it is NOT a knob.
K3S_API_HOST: "192.168.1.50"
#
# *** THIS IS THE CONTROL-PLANE VIP (kube-vip), NOT A NODE ADDRESS. *** Operator decision
# 2026-09-06. It was each environment's PRIMARY NODE IP until then, which made the CNI's
# control-plane path depend on one machine in a cluster whose entire purpose is HA: verified on
# live prod, all three cilium-agents carried KUBERNETES_SERVICE_HOST=192.168.2.10 (k3s-prod-1's
# own address) while kube-vip held 192.168.2.9. Losing that node leaves the datapath running off
# Cilium's cache while NO service or policy update lands — a silent, unbounded staleness, not an
# outage you would notice. A virtual address that follows the control plane is the fix, and it is
# what kube-vip was stood up to provide.
#
# dev -> 192.168.1.49 (= kube_vip_address, group_vars/all.yml; base default, below)
# prod -> 192.168.2.9 (= kube_vip_address, group_vars/k3s_prod/vars.yml)
#
# THE NAME IS NOW SLIGHTLY WRONG AND IS KEPT ANYWAY: renaming the knob would be a second,
# unrelated change to every substituting Kustomization at the same time as a live CNI endpoint
# move. Read it as "the API host Cilium dials", which is what it always meant.
#
# [parity] MUST equal the Ansible bootstrap's **k3s_apiserver_endpoint** for the SAME environment
# (infra/ansible/inventory/group_vars/all.yml -> roles/cilium/templates/cilium-values.yaml.j2,
# already inventory-parameterized). NOT k3s_api_host — that variable still exists and still holds
# the FOUNDER's address, because the cold-bootstrap `--server` join needs a machine that is
# already up before any VIP exists. The two are deliberately separate; see the TWO API ADDRESSES
# block at the top of group_vars/all.yml.
#
# Wrong value = Cilium cannot reach the API server on adopt and the CNI breaks. Port stays 6443
# in both envs, so it is NOT a knob.
#
# TLS needs nothing: k3s_tls_sans has always carried kube_vip_address alongside every node IP, so
# both addresses are already in the serving cert (verified live 2026-09-06 — prod answers through
# https://192.168.2.9:6443 under full CA verification). No cert reissue, no k3s restart.
K3S_API_HOST: "192.168.1.49"
# CiliumLoadBalancerIPPool `default-pool` (platform-config/cilium/l2-ip-pool.yaml): the VIP range
# L2-announced on the flat LAN. Must sit OUTSIDE the node IPs and any DHCP scope, and must match the
@@ -66,14 +160,59 @@ data:
UNIFI_L4_VIP: "192.168.1.54" # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi)
CRAFTY_L4_VIP: "192.168.1.55" # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports
# --- Bulk (external NFS) storage host (docs/04 bulk tier) ---------------------------------------
# THE address of the Proxmox host whose ZFS datasets back every `class: bulk` volume: the media
# library (Storage/plex, 8.33 T — plex, sonarr, radarr) and the legacy Nextcloud archive
# (Storage/nextcloud, 270 G). The renderer emits it into each static NFS PersistentVolume's
# `spec.nfs.server` (catalog/render.py, BULK_NFS_SERVER); the export PATH is per-dataset and is
# declared in the ServiceDefinition, because the path is identical in every environment and only
# the ADDRESS is not.
#
# WHY THIS IS A KNOB AND NOT A LITERAL — the two clusters are on OPPOSITE SIDES of pfSense.
# The storage host has exactly one interface, vmbr0 192.168.1.9/24, on pfSense's LAN.
# dev nodes are 192.168.1.40/.41/.42 — the SAME L2 segment. They reach it directly, and one
# NFS export to them already exists on that host (/mnt/Storage/k8s-volumes in
# /etc/exports), so this path is proven, not theoretical. Base default = the dev value,
# per the convention of the LAN block above.
# prod nodes are 192.168.2.10/.11/.12 — pfSense's WAN. tcp/2049 does NOT traverse that today;
# 22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. Opening
# the storage path is a PENDING OPERATOR DECISION with two candidate outcomes that give
# DIFFERENT values here: 192.168.2.102 (another port-forward) or 192.168.1.9 (a route).
# So prod holds a loud TBD-PROD-* placeholder — see clusters/prod/kustomization.yaml for the
# blast radius of shipping it unset, which is NOT the usual "fails to apply".
BULK_NFS_SERVER: "192.168.1.9"
# Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style.
# Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db,
# nextcloud-db, semaphore-db) and Velero's backupStorageLocation s3Url. Full URL incl. scheme+port.
# Prod needs its OWN target: a backup that lands on the dev LAN's MinIO is not off-cluster DR.
# Prod needs its OWN target: a backup that lands on the dev LAN's MinIO is not off-cluster DR —
# and, measured 2026-09-05 from a pod INSIDE prod, it CANNOT reach it anyway (prod is on the SOHO
# network 192.168.2.0/24, which is pfSense's WAN, while this MinIO sits inside pfSense's LAN:
# `curl http://192.168.1.9:9000/minio/health/live` timed out after 15s). Prod uses Cloudflare R2.
S3_BACKUP_ENDPOINT: "http://192.168.1.9:9000"
# S3 region. Every S3 client REQUIRES one; MinIO ignores whatever it is; Cloudflare R2 demands
# exactly `auto` (its `head-bucket` reports the physical hint `WNAM`, which is NOT a usable region
# string). This was hardcoded `us-east-1` inside platform/velero/helmrelease.yaml — correct-by-
# accident for MinIO and wrong for anything else, which is why it is a knob now. The base default
# is dev's LIVE value, deliberately: dev's Velero has been signing requests with `us-east-1`
# against that MinIO since bootstrap, and this repo's rule is that base == the known-good dev
# value. Prod overrides to `auto`.
S3_BACKUP_REGION: "us-east-1"
# --- storage (ADR-0003) ---
LONGHORN_DEFAULT_REPLICA_COUNT: "3" # default StorageClass replica count (3 on 3 nodes)
# Longhorn's defaultDataPath (platform/longhorn/helmrelease.yaml): the on-NODE directory every
# replica is written into. It must be the path the NODE's own data disk is actually mounted at —
# roles/node_baseline owns that half (node_baseline_longhorn_data_path), and the two are ONE
# fact stated in two places, so [parity] holds per environment or replicas land on the OS disk.
# Base default = the DEV value, same convention as GATEWAY_VIP / the LAN block: the known-good
# working value. Dev's dedicated /dev/vdb IS mounted here and the LIVE cluster has 18 volumes
# on it — changing dev's value would orphan every replica, so the dev overlay restates it
# verbatim. PROD DIFFERS: its data partition serves TWO tenants (Longhorn plus k3s's whole data
# dir, moved off the 11.8 GiB /var — infra/PROD-PROVISIONING.md §2), so it is mounted at the
# neutral /data and Longhorn gets the /data/longhorn SUBDIRECTORY, not the mount root.
LONGHORN_DATA_PATH: "/var/lib/longhorn"
# --- observability sizing (docs/10) ---
MONITORING_PROMETHEUS_RETENTION: "7d" # dev short | prod long
@@ -113,6 +252,99 @@ data:
VELERO_BACKUP_SCHEDULE: "0 3 * * *" # cron; overlays may differ
VELERO_RETENTION: "168h" # dev short | prod long
# ==========================================================================================
# OBJECT-STORE TENANCY — the three knobs that keep two clusters and two backup tools from
# writing into each other's data. THESE ARE SAFETY KNOBS, not preferences.
# ==========================================================================================
# Nothing about the object layout was per-environment before this: Velero's bucket was the
# hardcoded literal `velero` with no prefix, and every CNPG Cluster used the hardcoded
# `s3://cnpg` with barman's `serverName` defaulting to the CLUSTER NAME — and BOTH environments
# have a keycloak-db, a nextcloud-db and a semaphore-db. Point two clusters at one bucket under
# that layout and:
# (a) each cluster's Velero ADOPTS the other's backups and TTL-DELETES them on retention; and
# (b) two different Postgres servers stream WAL into ONE Barman prefix, which does not merely
# mix files — it CORRUPTS THE CHAIN and destroys PITR for both. That is the worse half:
# dev's DR restore is PROVEN (R-STATE-03) and this would silently un-prove it.
#
# WHY THERE IS NO SINGLE `S3_BACKUP_BUCKET`. The two consumers take differently-SHAPED values:
# the Velero chart wants `bucket` and `prefix` as separate fields, while CNPG's
# barmanObjectStore.destinationPath is ONE full s3:// URL whose first path segment IS the bucket.
# CNPG's bucket therefore already lives inside CNPG_BACKUP_PATH; adding a second knob that had to
# hold the same bucket name a second time would be a rotation waiting to drift — the exact class
# of duplication this repo has been bitten by. Three knobs, each shaped for its own consumer, and
# no two of them can contradict each other.
#
# BASE = DEV'S LIVE LAYOUT, BYTE-FOR-BYTE, AND MUST STAY THAT WAY. Dev's existing backups are at
# `s3://velero/` (bucket root, no prefix) and `s3://cnpg/<clusterName>/` on that MinIO. Giving dev
# a prefix, or renaming its buckets, would ORPHAN the very backups its proven restore depends on —
# the same reason dev's EXTERNALDNS_TXT_OWNER_ID stays `default` and its LONGHORN_DATA_PATH stays
# /var/lib/longhorn. Verified: with an EMPTY prefix the Velero chart omits `objectStorage.prefix`
# from the rendered BackupStorageLocation entirely (helm template, chart 12.1.0), so dev's live
# BSL is unchanged by this parameterization rather than merely equivalent to it.
#
# PROD, which has no history to preserve, gets ONE bucket with TWO disjoint top-level prefixes:
#
# s3://k3s-prod-backup/velero/… Velero (kopia repo + backup metadata)
# s3://k3s-prod-backup/barman/<clusterName>/… CNPG WAL + base backups
#
# The operator created exactly one R2 bucket, so velero-vs-barman separation has to come from the
# prefix rather than the bucket. The BUCKET ROOT IS DELIBERATELY LEFT EMPTY: if a prefix ever
# renders wrong, its objects land somewhere obviously bogus instead of silently colliding with a
# real tenant's data. Dev↔prod isolation is doubly independent — different endpoint AND different
# bucket — so neither cluster can reach the other's objects even with a mis-substituted prefix.
VELERO_BACKUP_BUCKET: "velero" # dev: the `velero` MinIO bucket | prod: k3s-prod-backup
VELERO_BACKUP_PREFIX: "" # dev: bucket root (live layout) | prod: "velero"
CNPG_BACKUP_PATH: "s3://cnpg" # dev: the `cnpg` MinIO bucket root | prod: s3://k3s-prod-backup/barman
# ------------------------------------------------------------------------------------------------
# WHICH CREDENTIAL SECRET THE BACKUP CONSUMERS READ (added 2026-09-07 — the per-env credential split)
# ------------------------------------------------------------------------------------------------
# The three knobs above say WHERE the backups go. This one says WHICH KEY signs the requests, and
# it exists because the endpoint moved per-environment while the credential did not.
#
# THE BUG IT FIXES. The four backup-credential Secrets — velero-cloud.sops.yaml and the three
# <db>-backup-credentials.sops.yaml — are SHARED BASE manifests: platform/ and apps/ are rendered
# by the same Flux tiers on both clusters, and cluster-vars substitution is the ONLY thing that
# differs (R-ENV-02). So writing prod's R2 token into those four files did not GIVE prod a
# credential, it REPLACED DEV'S: dev's Velero and all three dev CNPG Clusters would have started
# signing MinIO requests with an R2 key id. For Velero that is a broken BSL; for CNPG it is worse,
# because `archive_mode=on` means Postgres REFUSES TO RECYCLE a WAL segment it could not archive,
# so a 403 loop is a monotonically filling pgdata PVC ending in a PANIC — the same failure prod
# spent its whole bring-up in with a placeholder endpoint. One shared Secret also means rotating
# either environment's key silently breaks the other; that is the structural reason for the split,
# independent of dev's eventual decommission.
#
# THE MECHANISM. Each credential is now TWO Secret objects, both applied to BOTH clusters (they are
# base resources; nothing here forks a manifest per environment). The consumer names one:
#
# velero: credentials.existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}
# CNPG: s3Credentials.*.name: <db>-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
#
# ONE knob, not four, ON PURPOSE. Four independent secret-name knobs could be updated three-
# quarters of the way, leaving one consumer pointed at the OTHER environment's key with nothing
# rendering wrong — and StrictPostBuildSubstitutions cannot catch that, because it only fails on an
# UNDEFINED variable, never on a defined-but-stale one. With a single suffix the four move together
# or not at all. The cost is that the rendered Secret name is not greppable from the consumer; the
# names are spelled out in full in each kustomization.yaml beside the resources.
#
# THE BASE DEFAULT IS THE EMPTY STRING, AND THAT IS LOAD-BEARING. It renders the UNSUFFIXED names,
# which are the objects that already exist and that the live dev cluster is running. A default
# naming a Secret that does not exist would be worse than the shared-Secret state it replaces:
# Velero's pod would sit in ContainerCreating forever on a missing `existingSecret` volume, and a
# CNPG Cluster whose s3Credentials Secret is absent stops archiving WAL — i.e. the disk-full clock
# again — so an unresolvable default breaks the referencing cluster HARDER than a wrong key does.
# Empty is also the value dev must carry, so base and dev agree, exactly like VELERO_BACKUP_PREFIX.
#
# An EMPTY VALUE IS SAFE UNDER StrictPostBuildSubstitutions: strict mode fails on a variable that is
# not DEFINED, not on one defined as "". VELERO_BACKUP_PREFIX above has shipped empty since the
# gate was enabled and proves it. Both overlays state this knob explicitly — dev "" and prod
# "-prod" — so neither inherits it silently.
#
# ADDING A NEW BACKUP CREDENTIAL: create BOTH halves (`x-credentials.sops.yaml` and
# `x-credentials-prod.sops.yaml`), list both in the kustomization, and reference
# `x-credentials${BACKUP_CREDENTIALS_SUFFIX}`. Never a single shared Secret again.
BACKUP_CREDENTIALS_SUFFIX: "" # dev: "" (MinIO creds) | prod: "-prod" (R2 creds)
# --- placement ---
PLATFORM_NODE_SELECTOR_KEY: "kubernetes.io/os" # base no-op selector; prod may pin roles
PLATFORM_NODE_SELECTOR_VALUE: "linux"
+23
View File
@@ -5,8 +5,24 @@ apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- flux-system.yaml # GitRepository (reconciliation source)
# Cilium (the CNI) — its OWN Kustomization, split OUT of `platform` (2026-09-01). The mirror image
# of the oauth2-proxy/headlamp splits: those are leaves that must never gate a tier (wait:false);
# Cilium is the one component that legitimately SHOULD gate everything, so it keeps wait:true and
# `platform` dependsOn it. Splitting it out buys a per-CNI Ready condition, CNI-appropriate
# interval/timeout, and `flux suspend kustomization cilium` as an emergency valve that does not
# freeze the other eleven platform components. See clusters/base/cilium.yaml.
- cilium.yaml # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
- platform.yaml # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
- platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
# SECOND-ORDER CDI configs — CRs whose CRD is created by a controller AT RUNTIME, not by any
# manifest in git. cdi-operator.yaml registers only `cdis.cdi.kubevirt.io`; `storageprofiles` (and
# datavolumes, datasources, …) appear only after cdi-operator reconciles the `CDI` CR that
# platform-config applies. Because Flux dry-runs every object in a pass before applying any of them,
# keeping the StorageProfiles in platform-config was a TRUE DEADLOCK (unknown kind -> pass aborts ->
# CR never applied -> CRD never created -> tier never converges), invisible on dev and fatal on any
# cold cluster — observed on prod 2026-09-05. This tier is the durable fix; `standup.sh crds` stays
# as the belt-and-braces repair for clusters already wedged. See clusters/base/platform-config-cdi.yaml.
- platform-config-cdi.yaml # Flux Kustomization -> ./platform-config-cdi (dependsOn platform-config)
# App tier, split into dependency-ordered WAVES (was the single ./apps `apps` Kustomization).
# Each apps-wave-<N> dependsOn wave N-1 (wave 0 dependsOn platform-config) and health-gates
# (wait:true) ONLY its own wave, so a crashloop in a later wave can't stall an earlier one.
@@ -34,6 +50,13 @@ resources:
# reverse-proxy sidecar (Option B). POST-platform tier (dependsOn platform-config, wait:false); same
# non-gating leaf-UI rationale as headlamp/homepage/oauth2-proxy (a UI must never gate the platform).
- crafty.yaml # Flux Kustomization -> ./platform/crafty (dependsOn platform-config)
# Alert runbooks — POST-platform tier (dependsOn platform-config, wait:false). Serves the 34
# runbooks the alert rules' `runbook_url` annotations point at, at <apex>/runbooks/<name>; before
# this tier existed that path was a 404 and every alert handed the operator a dead link. Same
# non-gating leaf rationale as headlamp/homepage/crafty. Carries NO postBuild substitution on
# purpose — it renders documentation containing `${...}` shell examples that envsubst would empty;
# see clusters/base/runbooks.yaml.
- runbooks.yaml # Flux Kustomization -> ./runbooks (dependsOn platform-config)
# Velero backup controller — POST-platform tier (dependsOn platform-config, wait:false). A backup
# controller must NEVER gate the platform chain; same non-gating rationale as headlamp/oauth2-proxy.
# Carries velero-cloud.sops.yaml, so clusters/dev adds a SOPS decryption patch for it (by name).
+135
View File
@@ -0,0 +1,135 @@
---
# Flux Kustomization: SECOND-ORDER CDI CONFIGS — reconciles ./platform-config-cdi.
# Serves R-GITOPS-01 (all desired state in git, reconciled by Flux) and R-ENV-02.
#
# WHY THIS TIER EXISTS — the durable fix for the 2026-09-05 prod CDI DEADLOCK
#
# `platform-config` applied the `CDI` CR and the CDI `StorageProfile`s in the SAME pass.
# kustomize-controller server-side DRY-RUNS EVERY object before applying ANY of them, and
# `storageprofiles.cdi.kubevirt.io` does not exist until cdi-operator has reconciled that `CDI` CR.
# So the unknown kind aborted the whole tier → the CR was never applied → the CRD was never created →
# platform-config could NEVER converge and every tier below it stayed blocked. Not a race: a
# deadlock. cdi-operator was Available for 12h on prod with no CR. Invisible on dev (its CDI CRDs
# have existed for months), FATAL on any cold cluster. See DECISIONS-LOG 2026-09-05.
#
# `bootstrap/standup.sh crds` pre-applies the `CDI` CR to break it by hand. That is a WORKAROUND and
# stays as belt-and-braces for clusters already wedged; THIS Kustomization is the fix that makes the
# graph order itself, so a cold cluster needs no human in the loop.
#
# THE ORDERING GUARANTEE, precisely: `dependsOn` is REVISION-gated, not merely Ready-gated (verified
# against kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies`, our
# version): sharing a GitRepository with the dependency, this Kustomization cannot reconcile revision
# N until `platform-config` has APPLIED revision N — and, because platform-config runs `wait: true`,
# not until the `CDI` CR it applied is HEALTHY. cdi-operator creates the remaining CDI CRDs as part of
# reaching that state, so by the time this tier's first dry-run runs, `StorageProfile` is a known
# kind. One edge, no pre-apply.
#
# WHY NOT just move the CDI CR earlier, into `platform`? Because `platform` is the tier the CNI gates
# and everything else waits on: putting CDI's health on that critical path is the mistake the
# oauth2-proxy/headlamp splits were made to undo. The CR is correctly a platform-config CR; it is the
# StorageProfiles that were in the wrong pass.
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: platform-config-cdi
namespace: flux-system
spec:
interval: 10m
# 2m (short, deliberately). kustomize-controller caches API discovery, so on a COLD cluster the
# very first attempt here can still fail `no matches for kind "StorageProfile"` even though the CRD
# now exists — a stale RESTMapper, not a missing CRD (the failure mode `standup.sh crds` warns
# about). The controller resets the mapper and the retry succeeds, so the only cost of that stale
# cache is one retryInterval. Keep it short.
retryInterval: 2m
# 2m is ample: this tier applies two tiny cluster-scoped CRs. There is no workload to roll out, no
# image to pull and no controller to become Available, so a slow apply here means a real error, not
# a slow rollout — failing fast surfaces it instead of burning 5m per attempt.
timeout: 2m
# TRUE, and this tier is the SOLE declarer of both StorageProfiles (checked repo-wide), so prune is
# component-scoped: the stale set can only ever be a StorageProfile this file stopped rendering.
# It is NOT the platform/cilium case — nothing here declares a shared Namespace, or any object a
# second path also declares. See the transition note at the bottom of this file.
prune: true
# TRUE, but understand exactly what it buys and what it does NOT.
#
# CANNOT STALL: `StorageProfile`'s status type (CDI v1beta1 StorageProfileStatus: storageClass,
# provisioner, claimPropertySets, cloneStrategy, dataImportCronSourceFormat) carries NO
# `conditions` and NO `observedGeneration`, so kstatus's generic path reports it Current the moment
# it is applied. `wait: true` therefore adds a Ready condition without adding a way to hang — which
# is why it is safe to let `apps-vms` gate on it (contrast the headlamp gating stall).
# If a resource that CAN be unhealthy is ever added to ./platform-config-cdi, re-derive this:
# `apps-vms` waits on it.
#
# WHAT IT DOES NOT PROVE (honest gap): CDI reads the profile's **status**, not its spec, when it
# provisions a claim (status.claimPropertySets / status.cloneStrategy), and CDI's
# storageprofile-controller computes that status from our spec override plus inference. So Ready
# here means "the spec override is accepted by the API server", not "CDI has folded it into
# status". That reconcile is a watch-triggered round-trip of seconds, and the dependency requeue
# below it is ~30s, so a DataVolume cannot realistically beat it — but it is a timing argument, not
# a guarantee, and no `wait` setting can turn it into one (there is no condition to wait on).
wait: true
dependsOn:
# platform-config applies the `CDI` CR and (wait:true) waits for it to be healthy — which is what
# causes cdi-operator to create `storageprofiles.cdi.kubevirt.io`. Revision-gated, so this is an
# ordering guarantee and not a hope. Transitively this also chains platform -> cilium ->
# flux-system, so `cluster-vars` at revision N is applied before this renders revision N.
- name: platform-config
sourceRef:
kind: GitRepository
name: flux-system
path: ./platform-config-cdi
# NO SOPS `decryption` stanza: ../platform/kubevirt/cdi-storageprofiles.yaml is two plain CRs with
# no Secret and no *.sops.yaml, so neither overlay needs to patch one in (unlike
# oauth2-proxy/headlamp/crafty/velero, which each carry an encrypted Secret). Nothing to add to
# clusters/dev or clusters/prod for this tier.
#
# substituteFrom cluster-vars: the manifest currently contains NO `${TOKEN}`, so this stanza
# substitutes nothing today. It is declared anyway for two concrete reasons, not for symmetry:
# 1. COVERAGE. conformance/tests/l0-static/flux-postbuild-substitution/check.sh only audits paths
# whose Kustomization declares substituteFrom. Omitting it would move
# cdi-storageprofiles.yaml OUT of the audited set — a blind spot where a future `${TOKEN}`
# could ship as a literal string, precisely the class of bug that check exists to stop.
# 2. FAIL-CLOSED. kustomize-controller runs --feature-gates=StrictPostBuildSubstitutions=true, so
# with this stanza an undefined token FAILS this Kustomization loudly instead of rendering
# empty. Without the stanza there is no substitution at all and the literal ships silently.
postBuild:
substituteFrom:
- kind: ConfigMap
name: cluster-vars
# ── FIRST RECONCILE AFTER MERGE: a TRANSIENT DELETE-THEN-RECREATE. READ BEFORE MERGING. ──────────
# Moving cdi-storageprofiles.yaml out of ./platform-config drops both StorageProfiles from THAT
# Kustomization's inventory, and platform-config has `prune: true` — so its next apply DELETES
# StorageProfile/longhorn and StorageProfile/longhorn-vm, and this Kustomization recreates them on
# the following reconcile (revision-gated: strictly after platform-config's apply, so there is no
# dual-ownership fight — the objects are pruned before they are re-applied under this tier's
# `kustomize.toolkit.fluxcd.io/name=platform-config-cdi` label).
#
# Why that is acceptable here — and it is NOT the platform/cilium case:
# * Neither object is the sole (or any) declarer of a shared object. No Namespace, no ConfigMap,
# nothing else in the repo declares a StorageProfile. The stale set is exactly two
# cluster-scoped CDI CRs.
# * The objects do not actually go missing: CDI's storageprofile-controller creates a
# StorageProfile per StorageClass and recreates a deleted one, with an EMPTY `.spec` and
# inferred `.status`. So the transient is "our .spec overrides are absent", not "the object is
# absent".
# * A StorageProfile is read at DataVolume/PVC PROVISIONING and CLONE time only. Existing PVCs,
# existing DataVolumes and running VMs do not consult it again, so nothing already provisioned
# is disturbed and no data path is touched.
# * The only exposure is a DataVolume/clone STARTED inside the window: it would clone by `copy`
# instead of `snapshot`, and on `longhorn-vm` be stamped RWO/Filesystem instead of RWX/Block —
# permanently non-live-migratable for that disk. Flux cannot cause that, because `apps-vms` now
# dependsOn THIS tier (clusters/base/apps-vms.yaml), so no Flux-managed DataVolume can be
# created while the overrides are absent. A human running `virtctl`/`kubectl` in that minute
# could.
#
# PRECONDITION THE OPERATOR MUST CHECK, because it is what bounds the window: the recreate cannot
# happen until `platform-config` is Ready, and platform-config runs `wait: true` over the whole
# config tier (ClusterIssuers, wildcard Certificate, Gateway, Keycloak, CNPG, KubeVirt, CDI…). On a
# healthy cluster that is seconds and the window is ~30-60s (kustomize-controller's
# --requeue-dependency default is 30s). If platform-config is NOT Ready when this merges, the
# StorageProfiles are pruned and stay at CDI defaults until it recovers. So:
# `flux get kustomization platform-config` must be Ready BEFORE merging.
# If it is not, do the two-step the cilium move used instead: merge with `platform-config.prune`
# temporarily set to `false` (the objects are then dropped from its inventory but NOT deleted, and
# this tier ADOPTS them with zero window), verify the ownership label transferred, then restore
# `prune: true` in a second commit.
+19 -1
View File
@@ -11,7 +11,18 @@ spec:
interval: 10m
retryInterval: 2m
timeout: 5m
prune: true # drift correction: delete resources removed from git (R-GITOPS-03)
# Cilium was moved OUT of this tier into its own Kustomization (clusters/base/cilium.yaml)
# on 2026-09-01. `prune: true` is restored here now that the ownership transfer is VERIFIED
# live: Namespace/kube-system, the Cilium HelmRelease and the HelmRepository all carry
# `kustomize.toolkit.fluxcd.io/name=cilium`, this Kustomization's inventory no longer lists
# any of the three, and the `cilium` Kustomization's inventory holds exactly those three.
#
# LESSON WORTH KEEPING: `platform/cilium/namespace.yaml` was the ONLY declaration of
# Namespace/kube-system under ./platform. Moving that path out of a prune:true Kustomization
# made the stale set `delete namespace kube-system` — coredns, metrics-server, local-path AND
# Cilium. Before moving ANY platform/* component in future, check whether it is the sole
# declarer of a shared Namespace; that turns a component-scoped prune into a cluster-scoped one.
prune: true
wait: true # block `apps` until platform is healthy
# ORDERING FIX (2026-09-01 incident: a substitution race that failed OPEN broke Cilium).
#
@@ -40,6 +51,13 @@ spec:
# health-gates its own children (see clusters/dev/sync.yaml).
dependsOn:
- name: flux-system # the ROOT Kustomization (sync.yaml) that applies cluster-vars
# THE CNI GATES THE TIER. Cilium was split out of ./platform into its own Kustomization
# (clusters/base/cilium.yaml, wait:true) so its health has its own Ready condition and its own
# suspend valve. This edge preserves the ordering that co-tenancy used to provide implicitly:
# nothing in `platform` — and therefore nothing anywhere downstream — reconciles until the CNI
# is healthy. It is also revision-gated, which is what makes the prune-safety argument above
# deterministic rather than a race.
- name: cilium
sourceRef:
kind: GitRepository
name: flux-system
+57
View File
@@ -0,0 +1,57 @@
---
# Flux Kustomization: alert runbooks — reconciles ./runbooks.
#
# WHAT THIS SERVES. Every alert rule in this repo carries
# `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations over 34 distinct names.
# Until this tier existed that path returned 404 on the live cluster, so every alert that reached a
# human handed them a dead link at the moment they needed help. This Kustomization renders the
# markdown under ./runbooks into a ConfigMap and a two-replica nginx that serves it at that exact
# prefix. See runbooks/README.md and runbooks/serving/httproute.yaml.
#
# POST-PLATFORM TIER (dependsOn platform-config), NON-GATING (wait:false) — the same placement and
# the same reasoning as homepage/headlamp/oauth2-proxy. It needs the wildcard TLS cert, the shared
# Gateway (including the `https-apex` listener) and split-horizon DNS, all of which arrive with
# `platform` + `platform-config`; and a leaf web server must never be able to stall the platform
# chain, which is the whole lesson of DECISIONS-LOG's "Headlamp gating stall". wait:false guarantees
# its health can never gate anything even if something is later mistakenly pointed dependsOn it.
#
# NO postBuild.substituteFrom — THE ONE DELIBERATE DIVERGENCE FROM EVERY OTHER TIER, AND THE MOST
# IMPORTANT LINE IN THIS FILE.
# This tier renders the runbook PROSE into a ConfigMap, and runbooks are full of shell: `$POD`,
# `$(kubectl ...)`, and — unavoidably, since they document these very annotations — the literal text
# `${BASE_DOMAIN}`. Flux's postBuild substitution runs envsubst over the ENTIRE built stream, and it
# FAILS OPEN: an undefined `${X}` becomes the empty string while the Kustomization still reports
# Ready (DECISIONS-LOG 2026-09-01 — that is how `k8sServiceHost` was emptied and Cilium broke on 2
# of 3 dev nodes). Enabled here it would silently mutilate the documentation: every braced expansion
# in every command example blanked, in a file nobody re-reads after writing. Worse, the L0 check
# `conformance/tests/l0-static/flux-postbuild-substitution` scans only files reachable through
# kustomization `resources:`, not `configMapGenerator` inputs, so the damage would be invisible to
# the gate as well.
# The fix is to have nothing to substitute. The only knob this tier would have wanted is the apex
# hostname, and runbooks/serving/httproute.yaml avoids needing it by declaring NO `hostnames:` and
# inheriting the `https-apex` listener's own hostname instead. So this tier is
# environment-independent by construction and R-ENV-02 ("authored once, promoted dev -> prod") holds
# without substitution rather than because of it.
# IF A FUTURE KNOB EVER GENUINELY NEEDS SUBSTITUTION HERE: do not add substituteFrom to this
# Kustomization. Split the serving manifests into their own substituted tier and leave the content
# ConfigMap in an unsubstituted one, or the runbooks lose their shell examples.
#
# NO SOPS decryption stanza: this tier carries no encrypted Secret (markdown plus five plain
# manifests), so — like homepage — neither clusters/dev nor clusters/prod patches it.
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: runbooks
namespace: flux-system
spec:
interval: 10m
retryInterval: 2m
timeout: 5m
prune: true
wait: false
dependsOn:
- name: platform-config # wildcard TLS + Gateway (incl. the https-apex listener) first
sourceRef:
kind: GitRepository
name: flux-system
path: ./runbooks
+115 -5
View File
@@ -106,8 +106,11 @@ patches:
secretRef:
name: sops-age
# velero (backup controller) has its own post-platform Flux Kustomization (clusters/base/backup.yaml).
# It carries velero-cloud.sops.yaml (the scoped MinIO `cloud` credential), so it needs its own SOPS
# decryption stanza (same as headlamp/oauth2-proxy/crafty). The `sops-age` Secret is out-of-band (name only).
# It carries BOTH halves of the per-env `cloud` INI credential — velero-cloud.sops.yaml (dev's scoped
# MinIO key) and velero-cloud-prod.sops.yaml (prod's R2 token) — so it needs its own SOPS decryption
# stanza (same as headlamp/oauth2-proxy/crafty). Which one the Deployment mounts is chosen by
# BACKUP_CREDENTIALS_SUFFIX in the cluster-vars patch below, not here. The `sops-age` Secret is
# out-of-band (name only).
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
@@ -148,6 +151,33 @@ patches:
- op: replace
path: /data/ACME_CLUSTER_ISSUER
value: "letsencrypt-prod"
# --- DNS identity (R-DNS-01/02/03) -----------------------------------------------------
# DEV STAYS ON THE APEX. Restated explicitly even though it equals the base default, exactly
# like the LAN block and LONGHORN_DATA_PATH below: this is what the LIVE cluster is running.
# Every one of dev's public DNS records, its `*.home.keithjackson.dev` wildcard cert, its
# Keycloak issuer, every OIDC client redirect URI, kubectl SSO, and gitea.home.keithjackson.dev
# (which IS this repo's GitOps source) are all on this exact suffix RIGHT NOW. Prod moved to
# prod.home.keithjackson.dev instead (2026-09-04) — that alone ended the collision, so dev did
# not have to move. Relocating dev to dev.home.keithjackson.dev is phase 2 and is a migration,
# not a variable change: infra/PROD-PROVISIONING.md §10.
- op: replace
path: /data/BASE_DOMAIN
value: "home.keithjackson.dev"
# ExternalDNS scope. MUST stay the zone apex: dev serves the home.keithjackson.dev APEX itself,
# and the Cloudflare provider selects zones with this value (a strict-subdomain filter dropped
# the parent zone — DECISIONS-LOG 2026-08). Same value the retired DNS_ZONE knob held.
- op: replace
path: /data/EXTERNALDNS_DOMAIN_FILTER
value: "keithjackson.dev"
# DO NOT CHANGE THIS VALUE. Dev's LIVE public records are registered in Cloudflare's TXT
# registry under owner `default` (the external-dns chart default, which is what both clusters
# silently shared until the 2026-09-04 incident). Dev's external-dns identifies its own records
# BY this string: any other value and it stops recognising every record it manages, then
# re-creates duplicates beside the orphaned originals. Prod is `prod` — distinct, which is the
# whole point. The base default is a non-colliding sentinel, so this line is load-bearing.
- op: replace
path: /data/EXTERNALDNS_TXT_OWNER_ID
value: "default"
# --- dev LAN addressing (192.168.1.0/24 on the Proxmox LAN) -----------------------------
# EVERY knob in cluster-vars' "LAN addressing" block is restated here EXPLICITLY, even where it
# equals the base default. These values are what the LIVE dev cluster is running right now, so
@@ -160,11 +190,25 @@ patches:
- op: replace
path: /data/GATEWAY_VIP
value: "192.168.1.53"
# k3s API endpoint Cilium's kube-proxy replacement dials — the primary control-plane node
# (k3s-1). [parity] equals the Ansible inventory's k3s_api_host for dev.
# k3s API endpoint Cilium's kube-proxy replacement dials — DEV'S CONTROL-PLANE VIP,
# the address kube-vip floats, NOT the primary node .50.
# [parity] equals the Ansible inventory's k3s_apiserver_endpoint for dev
# (= kube_vip_address, group_vars/all.yml). See clusters/base/cluster-vars.yaml for why.
#
# *** THE ONE EXCEPTION TO "NEVER LET THIS BLOCK DRIFT FROM THE LIVE CLUSTER" ***
# Every other value in this block is what dev is running right now. This one is not:
# the live dev cilium-agents dial 192.168.1.50 (k3s-1). Changing it here is a
# DELIBERATE, OPERATOR-APPROVED move (2026-09-06) and reconciling it ROLLS THE CILIUM
# AGENT DAEMONSET on dev — one node at a time (updateStrategy.maxUnavailable: 1, kept
# in parity for exactly this reason). Do not push this to dev-live casually, and not
# while dev's DNS migration is mid-flight; the L2 datapath survives an agent restart
# but a wrong value here is the 2026-09-01 incident. .49 is in dev's serving-cert SANs
# already (k3s_tls_sans) and dev's own kubeconfig validates against it via
# tls-server-name, so TLS is not the risk — the risk is reachability, and .49 is on the
# same LAN as the nodes and ARP-announced by kube-vip on every server.
- op: replace
path: /data/K3S_API_HOST
value: "192.168.1.50"
value: "192.168.1.49"
# Cilium LB IP pool = the tofu reservation cilium_l2_vip_range .53-.60, outside the node IPs
# (.50/.51/.52) and outside the DHCP scope. .53 Gateway, .54 unifi, .55 crafty; .56-.60 free.
- op: replace
@@ -181,15 +225,81 @@ patches:
- op: replace
path: /data/CRAFTY_L4_VIP
value: "192.168.1.55"
# Bulk (external NFS) storage host: the Proxmox host itself, on the SAME LAN as dev's nodes
# (192.168.1.40/.41/.42), which already hold an NFS export from it (/mnt/Storage/k8s-volumes).
# Restated verbatim rather than inherited, per this overlay's "supply every knob" convention.
# Dev is therefore the environment in which the bulk manifests can actually be VALIDATED
# end-to-end before prod's network path exists — provided the additive `zfs set sharenfs=`
# change described in docs/BULK-NFS-STORAGE.md is applied for the dev node addresses.
- op: replace
path: /data/BULK_NFS_SERVER
value: "192.168.1.9"
# Off-cluster backup target: the on-Proxmox MinIO (plain HTTP, path-style) that every CNPG
# barmanObjectStore and Velero's backupStorageLocation write to.
#
# DEV IS DELIBERATELY STAYING ON THIS MinIO while prod goes to Cloudflare R2 (2026-09-07).
# Prod could not use it at all — it cannot even route to it — but dev's move is a separate,
# operator-gated decision, because R2 would start dev on an EMPTY prefix: Velero would write
# a fresh kopia repository, CNPG would take new base backups, and the 30-day Barman window
# would rebuild from zero. Dev's DR restore is the one PROVEN recovery path in this repo
# (R-STATE-03); resetting its PITR window is not a side effect to take while fixing prod.
- op: replace
path: /data/S3_BACKUP_ENDPOINT
value: "http://192.168.1.9:9000"
# DO NOT CHANGE THESE FOUR. They are dev's LIVE object layout, restated verbatim for the same
# reason as LONGHORN_DATA_PATH and EXTERNALDNS_TXT_OWNER_ID below/above: they equal the base
# defaults today, and a silent drift in base would ORPHAN dev's existing backups — the ones
# its proven restore depends on — without any manifest looking wrong.
#
# `us-east-1` is what dev's Velero has signed every request with since bootstrap; MinIO
# ignores the value, but changing it is a needless risk against a live, working backup path.
# An EMPTY VELERO_BACKUP_PREFIX means the bucket root, which is where dev's backups already
# are — and the Velero chart omits `objectStorage.prefix` entirely when it is empty (verified
# against chart 12.1.0), so dev's live BackupStorageLocation is byte-unchanged by this.
- op: replace
path: /data/S3_BACKUP_REGION
value: "us-east-1"
- op: replace
path: /data/VELERO_BACKUP_BUCKET
value: "velero"
- op: replace
path: /data/VELERO_BACKUP_PREFIX
value: ""
- op: replace
path: /data/CNPG_BACKUP_PATH
value: "s3://cnpg"
# WHICH CREDENTIAL SIGNS THOSE REQUESTS. Empty suffix = the UNSUFFIXED Secrets
# (velero-cloud-credentials, <db>-backup-credentials) — the scoped MinIO keys dev's Velero and
# its three CNPG Clusters are running RIGHT NOW, byte-identical to what is on the cluster. Prod
# renders "-prod" and reads the Cloudflare R2 token from the parallel `-prod` Secrets instead.
#
# RESTATED EXPLICITLY even though it equals the base default, for the same reason as
# EXTERNALDNS_TXT_OWNER_ID and the four backup-layout knobs above: the four Secrets are SHARED
# base manifests, so before this split prod's R2 token landed in the objects DEV reads. Dev
# signing MinIO with an R2 key id is 403 on every barman-cloud-wal-archive, and archive_mode=on
# turns that into a filling pgdata PVC — the disk-full clock, on the one cluster whose DR
# restore is PROVEN (R-STATE-03). This line is what keeps dev on its own credential.
#
# DEV IS NOT BEING BACKED UP GOING FORWARD and is slated for decommission (operator,
# 2026-09-07), so this is not protecting a future backup — it is protecting the EXISTING
# backups and the WAL chain until dev is retired. Do not "simplify" it away while dev runs.
- op: replace
path: /data/BACKUP_CREDENTIALS_SUFFIX
value: ""
# dev mirrors prod's 3-node topology, so the replicated tier keeps 3 replicas.
- op: replace
path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
value: "3"
# DO NOT CHANGE THIS VALUE. The LIVE dev cluster has 18 Longhorn volumes whose replicas are
# on disk under /var/lib/longhorn — the dedicated /dev/vdb ext4 disk roles/node_baseline
# mounts there. Longhorn addresses a replica by dataPath; pointing defaultDataPath anywhere
# else would orphan every one of them (they are not recoverable from the old path by
# rescheduling). Restated explicitly even though it equals the base default, exactly like the
# LAN block above: this is what the live cluster is running. Prod uses /data/longhorn because
# its data partition also carries k3s's data dir — see clusters/prod and PROD-PROVISIONING §2.
- op: replace
path: /data/LONGHORN_DATA_PATH
value: "/var/lib/longhorn"
# smaller retention / volumes to keep the dev workstation footprint light.
- op: replace
path: /data/MONITORING_PROMETHEUS_RETENTION
+350 -14
View File
@@ -115,8 +115,20 @@ patches:
name: sops-age
# --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) ---
#
# *** BOOTSTRAP BLOCKER — this URL CANNOT work on a cold prod cluster. READ BEFORE BRING-UP. ***
# ############################################################################################
# ### ###
# ### *** T E M P O R A R Y B O O T S T R A P S O U R C E — NOT PERMANENT *** ###
# ### ###
# ### The url below is a Gitea container running ON THE OPERATOR'S WORKSTATION, outside ###
# ### any cluster. It exists ONLY to get prod's first reconcile off the ground. It MUST ###
# ### be cut over to the in-cluster URL once Gitea runs on prod. If you are reading this ###
# ### and prod has been serving traffic for months, THE CUTOVER WAS NEVER DONE — that is ###
# ### a live single point of failure, not a settled design. See "CUTOVER" below. ###
# ### ###
# ############################################################################################
#
# WHY THE IN-CLUSTER URL CANNOT BE USED YET (blocker found 2026-09-01, still true; the
# unusable value was `https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`).
# Two independent reasons, both consequences of the split-horizon design:
# 1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER. Prod syncing from it means prod's
# GitOps source lives inside the very cluster slated for decommission. Gitea must move to
@@ -126,13 +138,46 @@ patches:
# 192.168.2.20 — PROD's own gateway, where no Gitea is listening. Prod's Flux would resolve
# gitea.home.keithjackson.dev to itself and never reach a repository.
#
# Bring-up therefore needs a TEMPORARY source (the bootstrap host-Gitea, or an off-cluster
# mirror), with a cutover to the in-cluster URL once Gitea actually runs on prod — exactly the
# sequence docs/05 describes for dev's own Phase-6 cutover. Tracked in PROD-PROVISIONING.md §7.
# WHAT THE TEMPORARY SOURCE IS (stood up 2026-09-04 on explicit operator direction):
# bootstrap/gitea/bootstrap-gitea.sh with HOST_NAT_IP=192.168.2.132 — a podman container on the
# operator's workstation, Gitea 1.22.6, listening on 192.168.2.132:3000. All three prod nodes
# reach it DIRECTLY on the LAN: no libvirt NAT hop (that network is retired) and no firewall
# change was needed. This is the same "local bootstrap source" pattern dev itself used before
# its Phase-6 cutover (docs/05 "Source-of-truth").
#
# PLAIN HTTP, AND WHY THAT IS ACCEPTED HERE: the bootstrap Gitea has no TLS, so there is no cert
# for Flux to trust and no caCertSecretRef to wire. Traffic is LAN-local between prod nodes and
# the workstation, the repo carries NO plaintext secrets (everything sensitive is SOPS-encrypted,
# .sops.yaml), and the window is the bring-up only. Do NOT extend this posture past cutover.
#
# NO secretRef, DELIBERATELY: the repo is PUBLIC on the bootstrap Gitea, so anonymous clone
# works and Flux needs no git credentials. VERIFIED 2026-09-04 by unauthenticated request:
# curl http://192.168.2.132:3000/api/v1/repos/Home-Network/home-network-k3s-impl -> "private":false
# git ls-remote http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git -> refs/heads/main
# If the GitRepository ever reports an auth failure that looks like a network problem, RE-CHECK
# THAT FIRST: a private repo here fails exactly that way.
#
# *** CUTOVER — the exit condition, mirroring dev's Phase-6 cutover (docs/05) ***
# WHEN: Gitea runs on prod (reconciled in-cluster) and answers gitea.home.keithjackson.dev
# through prod's own Gateway VIP with a publicly-trusted cert.
# DO: 1. push/mirror this repo into the prod in-cluster Gitea;
# 2. change the url below to
# https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
# and DELETE this whole TEMPORARY block (dev's equivalent stanza, ~15 lines above in
# clusters/dev/kustomization.yaml, is what "done" looks like);
# 3. confirm `flux get sources git -n flux-system` reconciles the new revision;
# 4. only THEN stop the workstation container. Keep it stopped-not-deleted as recovery,
# exactly as dev retained its bootstrap Gitea.
# UNTIL THEN: prod's reconciliation depends on a container on a workstation that may be
# powered off. Flux keeps the last-reconciled state running, but NO new revision lands while
# it is down. Tracked as an open blocker in infra/PROD-PROVISIONING.md §7.
#
# The BRANCH below is settled and correct: prod tracks `main`, the same branch as dev.
# CONSEQUENCE, recorded deliberately: once prod is live, EVERY MERGE TO main RECONCILES BOTH
# CLUSTERS. There is no promotion step and no soak window — dev and prod move together.
# NOTE for bring-up: bootstrap-gitea.sh pushes the operator's CURRENT HEAD to the bootstrap
# Gitea's `main`, whatever branch that is. Before bootstrapping prod, confirm that remote `main`
# is the revision you intend prod to run (`git ls-remote` above, against local `main`).
- patch: |-
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
@@ -140,7 +185,56 @@ patches:
name: flux-system
namespace: flux-system
spec:
url: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
# CUTOVER DONE 2026-09-06 (operator direction: "move the repo to prod"). Prod now reconciles
# from PROD'S OWN in-cluster Gitea. The workstation bootstrap container
# (http://192.168.2.132:3000) is retired as prod's source — it was a podman container on the
# operator's workstation holding a DHCP lease on WIFI, i.e. production's source of truth was a
# laptop. It is kept only as a recovery path (bootstrap/gitea/bootstrap-gitea.sh).
#
# THIS LINE IS SELF-REFERENTIAL AND THAT IS THE WHOLE DIFFICULTY. The root Kustomization
# renders ./clusters/prod, which contains this patch — so a live `kubectl patch` of
# GitRepository.spec.url is REVERTED on the next reconcile unless the authored value moves
# too. The cutover therefore had to be: (1) push the repo to prod's Gitea, (2) fast-forward
# its `main` to the exact commit prod was already running so the source swap changed nothing
# else, (3) live-patch the url to open a window, (4) land THIS commit through that window so
# the authored value agrees. Dev hit the identical trap on 2026-09-05
# (infra/PROD-PROVISIONING.md §10a "D0/D1 as executed").
#
# Verified before the switch: a throwaway pod INSIDE prod resolved this host through
# in-cluster CoreDNS to the Gateway VIP and cloned over the public LE cert with no CA
# supplied. The workstation being able to reach it proves nothing about pod egress.
# MOVED WITH THE APEX FLIP (2026-09-13) — and it had to move in the SAME fetch.
# This URL is a hardcoded literal, NOT templated on ${BASE_DOMAIN}, so the flip did not
# carry it. But the flip DOES rename prod's own HTTPRoute hostnames, so:
# - external-dns (policy=sync) deletes gitea.prod.home.keithjackson.dev, and
# - platform/coredns/coredns-custom.yaml answers the whole ${BASE_DOMAIN} zone with
# ${GATEWAY_VIP}, so in-cluster resolution moves from prod.home. to home.
# Leaving this on the old name would therefore have severed prod from its OWN GitOps
# source moments after the flip reconciled — self-reference, the same class of trap as
# patching this url live and having the root Kustomization revert it. The fix has to land
# THROUGH THE OLD SOURCE, which is why it is in the same push as the flip rather than
# after it.
# FOLLOW-UP worth doing: point this at the in-cluster Service
# (gitea-http.gitea.svc.cluster.local) so prod's GitOps stops depending on public DNS and
# on its own Gateway at all. Deferred: it needs the TLS/credential path rethought.
# IN-CLUSTER SERVICE, NOT A PUBLIC HOSTNAME (2026-09-13). This is the durable fix for a
# deadlock the apex flip walked straight into, and it is worth stating precisely.
#
# The url used to be a public name. The flip renamed prod's HTTPRoute hostnames, so the
# name this GitRepository pointed at stopped serving — but the HTTPRoute that WOULD have
# served the new name lives in a DOWNSTREAM tier, which cannot reconcile while the source
# is broken. Source down -> routes never rename -> source stays down. Observed live:
# `srcReady=False`, "unable to list remote for https://gitea.home.keithjackson.dev".
#
# Pointing at the Service breaks the cycle permanently: prod's GitOps no longer depends on
# public DNS, on Cloudflare, on its own Gateway, or on ${BASE_DOMAIN} at all. A future
# domain change cannot sever prod from its own source again.
#
# Plain http is deliberate and safe here: it is cluster-internal traffic to a ClusterIP,
# TLS terminates at the Gateway for external clients only, the GitRepository has no
# secretRef (the repo is cloned anonymously), and every secret in the tree is SOPS-encrypted
# at rest regardless of transport.
url: http://gitea.gitea.svc.cluster.local:3000/Home-Network/home-network-k3s-impl.git
ref:
branch: main
- target:
@@ -154,9 +248,97 @@ patches:
- op: replace
path: /data/ACME_CLUSTER_ISSUER
value: "letsencrypt-prod"
# ==========================================================================================
# PROD DNS IDENTITY (R-DNS-01/02/03) — the fix for the 2026-09-04 incident.
# ==========================================================================================
# PROD SERVES ITS OWN SUBDOMAIN. Until 2026-09-04 every hostname was HARDCODED to
# home.keithjackson.dev in the base manifests, so prod was authored to serve the EXACT SAME
# public names as the live dev cluster out of the SAME Cloudflare zone. Two consequences, one
# of which already happened:
# 1. HAPPENED: prod's external-dns (policy: sync, owner id `default` — the same implicit
# default dev had) concluded it owned dev's records and DELETED EVERY ONE of them.
# 2. QUEUED BEHIND IT: platform-config would have requested a `*.home.keithjackson.dev`
# wildcard from letsencrypt-prod for a cert dev already holds, and both clusters would
# have raced each other's `_acme-challenge` TXT records on that one zone.
# Moving prod to its own suffix dissolves both: the two clusters can no longer name the same
# record. This is what makes ACME_CLUSTER_ISSUER: letsencrypt-prod above safe here.
#
# dev -> home.keithjackson.dev (UNCHANGED — dev is live; phase 2 moves it)
# prod -> prod.home.keithjackson.dev (this knob)
#
# CONSEQUENCE — prod's hostnames are NOT the apex names. Prod's Gitea is
# gitea.prod.home.keithjackson.dev, its Keycloak keycloak.prod.home.keithjackson.dev, and so
# on; the wildcard cert becomes `*.prod.home.keithjackson.dev`. The apex CNAMEs that make
# `<svc>.home.keithjackson.dev` point at prod are PHASE 2, deliberately not done here — they
# would take traffic off the live dev cluster. infra/PROD-PROVISIONING.md §10.
#
# [parity] MUST equal k3s_base_domain in infra/ansible/inventory/group_vars/k3s_prod/vars.yml.
# The kube-apiserver's oidc-issuer-url is an INSTALL-TIME k3s flag, not a manifest, so it is
# set Ansible-side from that variable and cannot read this one. If they disagree, kubectl SSO
# trusts a different Keycloak than the one prod actually runs.
- op: replace
path: /data/BASE_DOMAIN
# APEX AS OF 2026-09-13. Prod now serves home.keithjackson.dev directly; dev has been
# decommissioned and released the name. Prerequisites completed and VERIFIED before this
# flip, in this order:
# 1. dev's 18 A records released by its own external-dns (policy=sync, sources deleted)
# 2. dev's 18 LEGACY UNPREFIXED TXT registry entries deleted from Cloudflare by hand —
# external-dns v0.21.0 removed only the `a-` prefixed form and then reported "All
# records are already up to date", so it would NEVER have cleared them. They carried
# `owner=default`, and prod runs `owner=prod`, so prod could not have claimed these
# names while they existed. Confirmed gone at the AUTHORITY (betty.ns.cloudflare.com);
# 1.1.1.1 served stale cache for minutes afterwards and must not be trusted here.
# 3. dev's `*.home.keithjackson.dev` + apex Certificate deleted, so it stops renewing.
# It was issued ~18d ago, so prod's request is #1 this week against LE's duplicate
# limit of 5 per identical name set per week.
# NOTE the zone has wildcards `*.keithjackson.dev` AND `*.home.keithjackson.dev` pointing
# at the public IP 136.51.115.252 (legacy non-k3s prod). Dev's specific records were
# SHADOWING that wildcard; prod's new specific records shadow it again. The legacy stack
# lives on `*.keithjackson.dev` and is untouched by any of this.
value: "home.keithjackson.dev"
# SAFETY NET #2 (independent of the owner id below): narrow ExternalDNS's name scope to prod's
# OWN subtree, so prod cannot create, update or delete any record outside it no matter what the
# TXT registry says. VERIFY ON FIRST RECONCILE — the Cloudflare provider uses this value for
# ZONE selection too, and on the version in use in 2026-08 a strict-subdomain filter dropped
# the parent zone ("no hosted zone matching record DNS Name"). If prod's external-dns logs no
# matching zone, it will create NOTHING — fail-closed, and it can still never delete dev's
# records. Fallback in that case: set this to "keithjackson.dev" (dev's value) and rely on
# EXTERNALDNS_TXT_OWNER_ID below, which is sufficient on its own.
- op: replace
path: /data/EXTERNALDNS_DOMAIN_FILTER
# MUST be the Cloudflare ZONE, not prod's subdomain. FINDING F-DNS1 (DECISIONS-LOG,
# 2026-08-17) proved a subdomain filter DROPS THE PARENT ZONE: ExternalDNS generated the
# record then skipped it with "no hosted zone matching record DNS Name", so prod would
# publish NOTHING. Isolation from dev does NOT come from this filter — it comes from
# (1) prod only requesting *.prod.home.keithjackson.dev hostnames, and
# (2) txtOwnerId: prod, so prod's registry never claims dev's `default`-owned records.
# That pairing is what the 2026-09-04 incident needed and this filter cannot provide.
value: "keithjackson.dev"
# SAFETY NET #1 — THE ACTUAL ROOT CAUSE OF THE INCIDENT. Both clusters left this unset and so
# both inherited the chart default `default`; prod then treated dev's TXT-registry entries as
# its own and, under policy: sync, deleted the records behind them. This value MUST differ from
# dev's (`default`, pinned in clusters/dev — dev's live records are registered under it and
# cannot be renamed without orphaning them). `prod` is distinct, which is the entire fix.
- op: replace
path: /data/EXTERNALDNS_TXT_OWNER_ID
value: "prod"
- op: replace
path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
value: "3"
# Longhorn's on-node replica directory: a SUBDIRECTORY of the claimed data partition, not
# the mount root. Prod's ~424 GB partition (LABEL=k3s-data) is mounted at the neutral /data
# and serves TWO tenants — /data/longhorn here, and /data/k3s for k3s's entire data dir
# (embedded etcd + the containerd image store), which had to move off the 11.8 GiB /var:
# measured on dev, containerd alone is 21-25 GB per node, so prod would have filled /var on
# its first image pulls and etcd could not have written its WAL. infra/PROD-PROVISIONING.md §2.
#
# [parity] MUST equal node_baseline_longhorn_data_path in
# infra/ansible/inventory/group_vars/k3s_prod/vars.yml — Ansible creates and mounts the
# directory, this tells Longhorn to use it. If they disagree, Longhorn writes replicas to a
# path on the OS filesystem instead of the data partition and /var fills up.
- op: replace
path: /data/LONGHORN_DATA_PATH
value: "/data/longhorn"
# longer retention / larger volumes on prod NVMe.
- op: replace
path: /data/MONITORING_PROMETHEUS_RETENTION
@@ -222,14 +404,32 @@ patches:
- op: replace
path: /data/CILIUM_LB_POOL_STOP
value: "192.168.2.30"
# --- still UNDECIDED -> loud placeholders (must be real before `flux bootstrap`) ------------
# k3s API endpoint Cilium's kube-proxy replacement dials. This is the prod CONTROL-PLANE VIP
# (kube-vip), NOT .20 (the ingress VIP) and NOT the single node address .10. It is owned
# Ansible-side (prod inventory k3s_api_host) and MUST equal that value — [parity] contract in
# platform/cilium/helmrelease.yaml. Not yet decided.
# (kube-vip), NOT .20 (the ingress VIP) and NOT the single node address .10.
#
# *** DECIDED 2026-09-06 by the operator: the VIP, as intended. *** (This block previously
# read "Not yet decided" while shipping .10 — the founder's own address — and that is what
# the LIVE prod cluster is running: all three cilium-agents carry
# KUBERNETES_SERVICE_HOST=192.168.2.10 while kube-vip holds .9, verified 2026-09-06. On a
# 3-node HA cluster that makes the CNI's control-plane path depend on k3s-prod-1: lose it and
# the datapath keeps running off Cilium's cache while no service or policy update lands, with
# nothing looking broken. The VIP is a virtual address that reaches the cluster wherever the
# control plane is, which is the whole reason kube-vip exists here.)
#
# It is owned Ansible-side by **k3s_apiserver_endpoint** (group_vars/all.yml, derived from
# this environment's kube_vip_address = 192.168.2.9 in group_vars/k3s_prod/vars.yml) and MUST
# equal that value — [parity] contract in platform/cilium/helmrelease.yaml. NOT k3s_api_host:
# that variable still exists and still holds the FOUNDER's .10, because the cold-bootstrap
# `--server` join needs a host that is up before any VIP is floated. Two addresses, two
# variables, deliberately.
#
# .9 sits OUTSIDE the Cilium LB pool (.20-.30) and is not a node address, so kube-vip and
# LB-IPAM can never be handed the same IP — asserted by
# conformance/tests/l0-static/ha-control-plane-as-code.
- op: replace
path: /data/K3S_API_HOST
value: "192.168.2.10"
value: "192.168.2.9"
# --- still UNDECIDED -> loud placeholders (must be real before `flux bootstrap`) ------------
# Pinned L4 VIPs, both carved out of the pool by lbipam.cilium.io/ips. Not yet assigned for
# prod. Leaving the dev literals in place would hand a prod LoadBalancer a dev-LAN address, so
# they stay placeholders; whatever is chosen MUST fall inside [POOL_START, POOL_STOP] and not
@@ -240,10 +440,146 @@ patches:
- op: replace
path: /data/CRAFTY_L4_VIP
value: "192.168.2.22"
# Off-cluster DR target. Dev's versitygw/MinIO (192.168.1.9:9000) sits on the DEV side of the
# network — a prod backup written there is not off-cluster DR and would mix two environments'
# PITR streams in one bucket. Operator supplies the prod endpoint (full URL incl. scheme+port).
# ------------------------------------------------------------------------------------------
# BULK (EXTERNAL NFS) STORAGE HOST — *** NOT YET DECIDED. THIS PLACEHOLDER IS LOAD-BEARING. ***
# ------------------------------------------------------------------------------------------
# Backs every `class: bulk` volume: the 8.33 T media library (Storage/plex — plex, sonarr,
# radarr) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
#
# THE STORAGE HOST IS NOT REACHABLE FROM PROD AT ALL RIGHT NOW. It has one interface,
# vmbr0 192.168.1.9/24, on pfSense's LAN; prod's nodes are on 192.168.2.0/24, pfSense's WAN.
# 22 and 8006 answer from prod only because pfSense port-forwards them via 192.168.2.102;
# 2049 and 111 do not. Opening that path is an OPERATOR decision, and the two candidate
# outcomes give DIFFERENT values here:
# port-forward 192.168.2.102:2049 -> 192.168.1.9:2049 => "192.168.2.102"
# a WAN->LAN route to 192.168.1.0/24 => "192.168.1.9"
# Guessing either one authors a prod cluster that mounts nothing, or — worse, if the guess
# ever became reachable — mounts something unintended. So it stays a placeholder.
#
# WHICH PORTS THE DECISION MUST OPEN: exactly ONE, tcp/2049. The PVs mount nfsvers=4.1, and
# NFSv4 carries mount and lock in-protocol — no rpcbind/111, no mountd, no statd. Choosing v4
# over v3 was in part this decision's cost: v3 would have needed four ports and three of them
# on ephemeral numbers.
#
# WHAT THE EXPORT MUST ALLOW IS ALREADY SETTLED, and it is the same under BOTH outcomes.
# Measured 2026-09-07 from this workstation (192.168.2.132) through the existing port-forward:
# the storage host logged the connection as coming FROM 192.168.2.132, not from a pfSense LAN
# address (`ss -tn` peer 192.168.2.132, and the host's only route off-LAN is its default via
# 192.168.1.1). pfSense DNATs WITHOUT source NAT, so the NFS server will see prod's REAL node
# addresses either way. The additive `zfs set sharenfs=` values are in docs/BULK-NFS-STORAGE.md.
#
# *** BLAST RADIUS — READ BEFORE MERGING THIS TO A BRANCH PROD RECONCILES. ***
# A PersistentVolume's spec.nfs.server is an unvalidated string: the API server ACCEPTS this
# placeholder, the PV is created and the PVC BINDS to it (binding is pure control-plane). The
# failure surfaces only when a pod mounts — MountVolume.SetUp "Failed to resolve server
# TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr and nextcloud would sit in
# ContainerCreating on prod. Unlike the LAN knobs above, this placeholder does NOT fail at
# apply time and there is no CRD validation to lean on. Set this value BEFORE prod reconciles
# a revision containing the bulk volumes; the dev overlay renders a working value, which is
# where these manifests are meant to be proven first.
- op: replace
path: /data/BULK_NFS_SERVER
# 192.168.2.102 is pfSense's WAN address, which DNATs tcp/2049 through to the Proxmox
# host's REAL address 192.168.1.9 on pfSense's LAN. Prod sits on the WAN side and cannot
# reach 192.168.1.9 directly, so the cluster mounts the NAT address. Dev is on the LAN and
# keeps the base value 192.168.1.9.
#
# PROVEN from k3s-prod-1 on 2026-09-12, not inferred:
# - mount of 192.168.2.102:/mnt/Storage/plex negotiated nfs4 vers=4.1 proto=tcp
# - a write as uid 1000 succeeded. Tested as 1000 and NOT root deliberately: the export
# carries no_root_squash, so a root write would have passed and proved nothing, while
# LSIO images default to PUID 911 and would have gone Ready then failed every import
# with EACCES. The tree is 0755 1000:3000, so uid 1000 is the only writer.
# - ln() across downloads/ -> tv/ returned link count 2 (same inode), so the arrs get
# real atomic moves instead of silently degrading to copy-then-delete on EXDEV.
# pfSense DNATs WITHOUT source NAT (the host logs prod's own node addresses as the peer),
# so the ZFS export ACL names 192.168.2.10/.11/.12 directly rather than one translated IP.
value: "192.168.2.102"
# ==========================================================================================
# OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05; bucket + token 2026-09-07).
# ==========================================================================================
# Dev's versitygw/MinIO (192.168.1.9:9000) sits on the DEV side of the network and is NOT an
# option for prod, for two independent reasons:
# 1. UNREACHABLE. Prod is on the SOHO network 192.168.2.0/24 — pfSense's WAN — while that
# MinIO is inside pfSense's LAN. Measured from a pod INSIDE prod (the workstation proves
# nothing about pod egress): `curl http://192.168.1.9:9000/minio/health/live` → connection
# timed out after 15s. Reaching it would need a WAN→LAN rule that does not exist.
# 2. NOT OFF-CLUSTER DR. It runs on the same Proxmox host as dev's VMs, so it already shares
# a physical failure domain with dev. R2 is genuinely off-site, and its free egress means
# restore REHEARSALS cost nothing — R-STATE-03 has no financial disincentive to stay proven.
#
# THIS UNBLOCKS AN ACTIVE INCIDENT, not just a roadmap item. All three prod CNPG Clusters ship
# `archive_mode=on` with `endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`, so barman-cloud-wal-archive
# exits 4 on every segment and Postgres — correctly — REFUSES TO RECYCLE WAL. Verified live
# 2026-09-07: keycloak-db / nextcloud-db / semaphore-db all report
# ContinuousArchiving=False/ContinuousArchivingFailing, and keycloak-db's 5 GiB PVC is at
# 579 MiB used / 4360 MiB free and climbing monotonically. A disk-full PANIC there takes the
# cluster's SSO database down, and with it kubectl OIDC, Grafana, Headlamp, Gitea and
# forward-auth. Fixing the endpoint lets Postgres drain the whole backlog.
#
# ENDPOINT IS ACCOUNT-SCOPED, so it stays a prod-overlay value even though it is not a LAN
# address: this is the operator's Cloudflare account id, confirmed from inside a prod pod
# (`head-bucket` on k3s-prod-backup → BucketRegion WNAM; PUT and DELETE both succeed).
# It is NOT put in base, because base is dev's live MinIO and dev is deliberately STAYING
# there — moving dev to R2 would reset its PITR window and orphan the backups its PROVEN
# restore depends on. Two environments, two targets, isolated by URL *and* by bucket.
- op: replace
path: /data/S3_BACKUP_ENDPOINT
value: "TBD-PROD-S3-BACKUP-ENDPOINT"
value: "https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com"
# R2 requires the region string to be exactly `auto`. (`head-bucket` answers
# `{"BucketRegion":"WNAM"}` — that is a PHYSICAL LOCATION HINT, not a signing region; using it
# here produces AuthorizationHeaderMalformed.) Dev keeps the base `us-east-1` its live MinIO
# has always been signed with.
- op: replace
path: /data/S3_BACKUP_REGION
value: "auto"
# ------------------------------------------------------------------------------------------
# OBJECT TENANCY INSIDE THE ONE BUCKET. The operator created a SINGLE R2 bucket, so velero and
# barman cannot be separated by bucket the way they are on dev's MinIO (`velero` and `cnpg`).
# They are separated by disjoint TOP-LEVEL PREFIXES instead:
#
# s3://k3s-prod-backup/velero/… Velero: kopia repo + backup metadata
# s3://k3s-prod-backup/barman/<clusterName>/… CNPG: WAL + base backups
#
# The bucket ROOT is left empty on purpose — a misrendered prefix then lands somewhere
# obviously wrong instead of silently colliding with a live tenant. Velero's retention TTL
# deletes objects under its own prefix only, so it can never reap a WAL segment; and barman's
# per-cluster serverName subdirectory keeps the three Postgres servers' chains apart, which
# matters because interleaving two servers' WAL into one prefix corrupts the chain outright.
# Dev cannot be reached from any of these: different endpoint AND different bucket names.
# ------------------------------------------------------------------------------------------
- op: replace
path: /data/VELERO_BACKUP_BUCKET
value: "k3s-prod-backup"
- op: replace
path: /data/VELERO_BACKUP_PREFIX
value: "velero"
- op: replace
path: /data/CNPG_BACKUP_PATH
value: "s3://k3s-prod-backup/barman"
# ------------------------------------------------------------------------------------------
# WHICH CREDENTIAL SIGNS THE REQUESTS TO THAT BUCKET (per-env credential split, 2026-09-07).
# ------------------------------------------------------------------------------------------
# The knobs above moved the ENDPOINT and the OBJECT LAYOUT per environment. The credential did
# not move with them, and it could not: the four backup-credential Secrets are SHARED BASE
# manifests, applied to both clusters by the same Flux tiers. Writing the R2 token into them
# gave prod a working key by TAKING DEV'S AWAY — dev would have signed MinIO requests with an
# R2 key id, and under archive_mode=on that is a filling pgdata PVC, not a missing backup.
#
# So each credential is now two objects, and this suffix selects prod's:
# velero-cloud-credentials-prod the R2 token in the AWS INI `cloud` shape
# <db>-backup-credentials-prod the same token as ACCESS_KEY_ID / ACCESS_SECRET_KEY
# for keycloak-db, nextcloud-db and semaphore-db. Both halves ship to both clusters (they are
# base resources); ONLY this value decides which one each consumer reads. It is a SUFFIX rather
# than four secret-name knobs deliberately — four could be updated three-quarters of the way,
# leaving one consumer on dev's MinIO key with nothing rendering wrong, and strict-mode
# substitution cannot catch a defined-but-stale variable.
#
# THE R2 TOKEN IS THE ONE PROVEN FROM INSIDE PROD (chore/r2-credentials 0a6a862): head-bucket on
# k3s-prod-backup returns BucketRegion WNAM, and PUT and DELETE both succeed. Rotating it now
# touches ONLY the four `-prod` files; dev's MinIO keys are untouched objects. That isolation is
# the structural reason for the split, independent of dev's eventual decommission.
- op: replace
path: /data/BACKUP_CREDENTIALS_SUFFIX
value: "-prod"
# TODO(WS-INFRA, Phase 7): pin prod node placement once the bare-metal role labels exist.
+19
View File
@@ -8,3 +8,22 @@ _artifacts/
# Python bytecode from lib/ (status.py imports coverage.py); never committed.
__pycache__/
# --- THE ONE EXCEPTION: the committed run ledger (runs/) -------------------------------
# `runs/<UTC>-<short-sha>/{report.junit.xml,run.log}` + `runs/ledger.jsonl` are the
# machine-readable proof-of-pass record that lib/status.py's docstring asks for. They are
# SOURCE, not regenerated artifacts: a run's evidence cannot be reconstructed after the
# fact, so it must survive in git.
#
# The negations are needed because the ROOT .gitignore ignores `*.junit.xml` globally
# (and `coverage/`). A deeper .gitignore overrides a shallower one, and nothing ignores
# the `conformance/runs/` directory itself, so re-including the files here works (git
# cannot re-include a file whose PARENT DIRECTORY is excluded — that is why this must not
# live under _artifacts/).
#
# The run LOG is the load-bearing half: Chainsaw's JUnit carries no stdout, and the L2/L3
# tests self-skip GREEN, so the sentinel token in run.log is the only thing separating
# "proven" from "skipped". Never commit a report.junit.xml without its run.log.
!runs/
!runs/**
!runs/**/*.junit.xml
+103
View File
@@ -22,6 +22,10 @@ behavioral / L3 fault-injection** (this harness).
./run.sh --lint # structural-validate test defs (no cluster)
./run.sh --strict-coverage # hygiene gate: fail on tag/annotation drift
./run.sh --require-all # phase-exit gate: every requirement must be covered
./run.sh --record # run AND capture the evidence (see "Proof of pass")
./run.sh --layer l0 --include-scope whole-repo # union: the tagged subset PLUS every test
# declaring `conformance.home.k3s/scope: whole-repo`
./run.sh --layer l0 --print-selection # print the dirs the selection resolves to; run nothing
```
**Gating is scoped, not "all 51 always"** (R-TEST-03). Three distinct checks:
@@ -31,6 +35,29 @@ behavioral / L3 fault-injection** (this harness).
- `--requirements R-…` — **merge gate**: selects the tagged tests to run **and** asserts
those touched requirements are covered (a listed safety-critical one must have an L3 test).
- `--require-all` — **phase-exit gate**: every registry requirement must be covered.
- `--include-scope whole-repo` — **the escape hatch scoping needs**, because scoping has one
blind spot and it is the worst-shaped kind. Some L0 invariants quantify over the WHOLE
tracked tree (no plaintext secrets anywhere; no RBAC wildcard anywhere), or over subtrees
the coarse path→requirement map keys to **nothing** (`apps/`, `infra/`, most of
`platform/`). For those, *the change most likely to break the invariant attributes no
requirement at all*, so requirement-scoped selection filters the check out of exactly that
change. Such a test **declares** its scope:
```yaml
metadata:
annotations:
conformance.home.k3s/scope: "whole-repo"
```
and `--include-scope whole-repo` unions every declaring test into the selection. The CI
stages (`ci/l0.sh`, `ci/l1.sh`, `ci/conformance.sh`) all pass it, so **adding the
annotation is the only step** — there is nothing to remember and nothing to hand-write.
Two consequences, both deliberate: a declaring test always runs, and a **SKIP of a
declaring test is UNWIRED, not green** (the runner exits 3, which `ci/l0.sh` turns into the
gate's own rc 3 — never a pass). `tests/l0-static/whole-repo-invariants-wired` proves the
mechanism end-to-end and falsifiably, so it cannot rot silently. This replaced a
hand-written "UNCONDITIONAL" stage per check in `ci/l0.sh`; see `lib/common.sh` for the
full rationale and the incidents behind it.
- Documented out-of-scope gaps (e.g. `R-ENV-02` dev→prod promote and `R-GITOPS-01`
state-in-git, owned by WS-INFRA/WS-GITOPS and keyed wholesale by the coarse path map)
are exempted from the completeness gates via `--allow-gap`, wired in `run.sh` as
@@ -105,13 +132,89 @@ lib/
common.sh tagging convention + test discovery/selection helpers
coverage.py requirement-coverage generator (stdlib only)
status.py requirement-STATUS generator (stdlib only; --selftest)
record_run.py run-ledger writer: PROOF-of-pass (stdlib only; --selftest)
emit_skipped_junit.py Phase-0 skipped-JUnit shim (no cluster)
runs/
ledger.jsonl TRACKED append-only proof-of-pass ledger (one row per
testcase per run) — the only source status.py trusts
<UTC>-<short-sha>/ TRACKED per-run evidence: run.log + report.junit.xml
tests/
l1-rendering/ l2-behavioral/ l3-fault-injection/ meta/ (see tests/README.md)
incluster/
chainsaw-job.yaml rbac.yaml in-cluster L2/L3 Job + scoped test SA (ADR-0011)
```
## Proof of pass — the run ledger (`runs/`)
`--status` used to report **PROVEN: 0** for all 51 requirements, and said so honestly:
proof-of-pass was not recoverable from repo state. `_artifacts/` is gitignored and the
root `.gitignore` kills `*.junit.xml` globally, so no run result ever reached git; the
only record of a green run was DECISIONS-LOG.md prose. `runs/` is the fix `lib/status.py`
asked for — **a committed, machine-readable run ledger**.
```bash
./run.sh --record # runs normally, and captures the evidence
# -> conformance/runs/<UTC>-<short-sha>/run.log (tee'd, carries the sentinels)
# -> conformance/runs/<UTC>-<short-sha>/report.junit.xml (that run's JUnit)
# -> conformance/runs/ledger.jsonl (one appended row per testcase)
python3 lib/record_run.py --run-dir runs/<id> [--dry-run] # or record an existing run dir
python3 lib/record_run.py --selftest # deterministic, no cluster
```
**Recording is explicit, never implicit.** Plain `./run.sh` writes no ledger row: a run
that silently records itself is how a bad run becomes a permanent false proof. **Commit
the run dir and the ledger** — an uncommitted ledger proves nothing.
### Why a committed JUnit alone would be WORSE than nothing
The L2/L3 tests are runnable and **self-skipping**: when a precondition is missing (no
`longhorn` StorageClass, no Velero CRDs, fewer than 3 Ready nodes) they echo a skip token
and `exit 0`, and their `check:` accepts either outcome —
```yaml
(contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')
|| contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true
```
So **a self-skip exits 0 and Chainsaw records `passed` exactly as a real run does**, and
Chainsaw's JUnit carries no stdout — its `<testcase>` elements hold only `<properties>`.
A committed JUnit on its own would therefore record `DR_RESTORE_SKIPPED_OK` as *proof of
R-STATE-03*: fake green, and worse than no ledger because it would look authoritative.
**The sentinel token in `run.log` is the only thing that separates proven from skipped.**
`record_run.py` derives each test's tokens from its own `chainsaw-test.yaml` (every
`contains($stdout, 'TOKEN')`; `*_SKIP*_OK` is a skip token, the last non-skip token is the
verdict token), so the table below is never hand-maintained:
| Test | pass token (proof) | skip token (never proof) |
|---|---|---|
| `l3/node-loss` | `NODE_LOSS_VOLUME_AVAILABLE_OK` | `NODE_LOSS_SKIPPED_OK` |
| `l3/dr-restore` | `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK` (+ `OFF_CLUSTER_TARGET_OK`, `BACKUPS_ARE_SCHEDULED_OK`) | `DR_RESTORE_SKIPPED_OK`, `DR_SCHEDULE_SKIPPED_OK` |
| `l3/data-survival` | `DATA_SURVIVED_OK` | `DATA_SURVIVAL_SKIPPED_OK` |
| `l3/vm-data-survival` | `VM_DATA_SURVIVED_OK` | `VM_DATA_SURVIVAL_SKIPPED_OK` |
| `l3/hands-off-update` | `HANDS_OFF_UPDATE_OK` | `HANDS_OFF_UPDATE_SKIPPED_OK` |
| `l2/vm-lifecycle` | `VM_LIFECYCLE_OK` | `VM_LIFECYCLE_SKIPPED_OK` |
| `l2/vm-default-deny` | `VM_DEFAULT_DENY_OK` | `VM_DEFAULT_DENY_SKIPPED_OK` |
A row reaches `proof: "proven"` on **one** path: JUnit `passed`, **every** declared pass
token present in that test's log lines, and **no** skip token anywhere in them. A skip
token beats a pass token (worst wins). Anything else is `skipped`, `failed`,
`junit-skipped`, `unknown-no-sentinel-declared` or `unknown-sentinel-absent` — all
non-green. L0/L1 need no sentinel because `lib/run_l1_offline.py` already renders their
skips as JUnit `<skipped/>`, never as a green pass.
### Staleness — a proof is a claim about a tree, not a badge
`--status` honours a `proven` row only while the run's `git_sha` is a **descendant of the
last commit touching that requirement's implementing manifests or its verifying test**.
Otherwise the row degrades to **`stale-proof`**, which is not green. A run captured from a
dirty working tree is recorded as `<sha>-dirty`, which no ancestry check can satisfy — so
it never counts. Re-run with `--record` to refresh.
A requirement reads `proven` only when **every** verifying test does; one skipped, failed
or unrecorded test drags the whole requirement down. With an empty ledger `--status`
behaves exactly as before: every requirement `not-derivable`, headline **PROVEN 0**.
## How CI invokes it
CI (Gitea Actions, wired in T-0.2 / migrated in T-6.1b) calls **this same `run.sh`**:
+7
View File
@@ -57,6 +57,13 @@ rules:
verbs: [create, delete, get, list, watch, patch]
- apiGroups: [""]
resources: [pods, services, configmaps, secrets, persistentvolumeclaims]
# LOAD-BEARING `get` ON CONFIGMAPS, BEYOND FIXTURE CREATION (R-ENV-02): the environment-
# agnostic L2 tests (dns-exposure, dns-internal-resolution, gateway-ingress-reachable,
# oidc-sso-grafana, trusted-tls) resolve the cluster's own BASE_DOMAIN / GATEWAY_VIP at run
# time from `flux-system/cluster-vars` — the same ConfigMap Flux substitutes from — via
# chainsaw `x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars')` bindings.
# If this rule is ever narrowed to the test's own namespace, those bindings fail and the
# tests error out. Narrow it only alongside a namespace-scoped read of that ConfigMap.
verbs: [create, delete, get, list, watch, patch]
- apiGroups: [""]
resources: [pods/exec]
+56
View File
@@ -47,6 +47,62 @@ conf_dirs_for_requirements() {
done < <(conf_all_test_files) | sort -u
}
# --- DECLARED SCOPE: whole-repo invariants -------------------------------------------
# A test may also declare the SCOPE of the invariant it asserts:
#
# metadata:
# annotations:
# conformance.home.k3s/scope: "whole-repo"
#
# WHY THIS EXISTS — the requirement-scoped-selection blind spot (R-TEST-03)
# ------------------------------------------------------------------------
# The merge gate runs the conformance subset tagged with the requirements a change
# TOUCHES (ci/touched-requirements.sh -> `--requirements`). That is right for an invariant
# whose input surface is the subtree its requirements map to. It is WRONG — silently, and in
# the worst direction — for an invariant that quantifies over the WHOLE tracked tree, or over
# subtrees (apps/, platform/<most>, infra/) that the coarse path->requirement map keys to
# NOTHING. For those, the change most likely to break the invariant attributes no requirement
# at all, so the requirement-scoped selection filters the check out of EXACTLY that change.
# Three real incidents came out of that blind spot; each is recorded in the header of the
# test that now guards it (flux-postbuild-substitution, keycloak-client-secret-reconciler,
# bulk-tier-distinct-from-replicated).
#
# The repo's previous answer was a hand-written "UNCONDITIONAL" stage per such test in
# ci/l0.sh — which does not scale and fails silently in the worst way: whoever adds the NEXT
# whole-repo invariant must REMEMBER to hand-write a stage, and if they forget, the check runs
# only when some unrelated requirement happens to be touched. The scope annotation replaces
# that with a DECLARATION the runner enforces: `run.sh --include-scope whole-repo` unions
# every test carrying it into the selection, so adding the annotation is the only step.
#
# TWO things follow from the annotation, and both are deliberate:
# 1. the test is ALWAYS selected (union with the requirement-scoped subset), and
# 2. a SKIP of that test is UNWIRED, NOT GREEN (lib/run_l1_offline.py --unwired-scope).
# A whole-repo invariant that reports "nothing to audit" is reporting that the subtree
# it audits has VANISHED, which is not a pass. This is the rc-3 semantics the bespoke
# stages encoded ("check SKIPPED (clusters/ absent) — not a pass"), preserved verbatim.
export CONF_SCOPE_ANNOTATION="${CONF_LABEL_PREFIX}/scope"
# Directories of tests DECLARING the given scope (e.g. "whole-repo"). The annotation is
# matched on a real YAML mapping line only: a `#` comment cannot start with the key, so a
# scope named in a test's header prose is documentation, not a declaration (the same
# "reference, not declaration" rule ci/touched-requirements.sh applies to R-IDs).
conf_dirs_for_scope() {
local scope="$1" f
[ -n "$scope" ] || return 0
while IFS= read -r f; do
[ -n "$f" ] || continue
if grep -Eq "^[[:space:]]*${CONF_SCOPE_ANNOTATION}:[[:space:]]*[\"']?${scope}[\"']?[[:space:]]*$" "$f"; then
dirname "$f"
fi
done < <(conf_all_test_files) | sort -u
}
# Union of conf_dirs_for_scope over a comma/space separated list of scopes.
conf_dirs_for_scopes() {
local scopes="$1" s
for s in ${scopes//,/ }; do conf_dirs_for_scope "$s"; done | sort -u
}
# Directories for a given layer (subtree): l0 | l1 | l2 | l3 | meta | all.
# Accepts a comma/space separated LIST of layers (e.g. "l2,l3") and returns the UNION of
# their test dirs, deduplicated. A single layer keeps its prior behavior (backward compatible);
+956
View File
@@ -0,0 +1,956 @@
#!/usr/bin/env python3
"""Append a conformance RUN LEDGER row per testcase — the committed proof-of-pass record.
WHY THIS EXISTS
---------------
`lib/status.py` reports PROOF as `not-derivable` for every requirement, and names its own
fix in its docstring: *"a committed, machine-readable run ledger"*. There was none, because
`conformance/_artifacts/` is git-ignored and the root `.gitignore` kills `*.junit.xml` and
`coverage/` globally, so nothing about a green run ever reached repo state. This tool is
that ledger's writer: it reads one run's JUnit + run log out of `conformance/runs/<run-id>/`
and appends one JSON object per testcase to `conformance/runs/ledger.jsonl`, which IS
tracked (see conformance/.gitignore for the negations that let it survive).
THE TRAP THIS TOOL EXISTS TO AVOID — READ BEFORE CHANGING ANYTHING
------------------------------------------------------------------
The L2/L3 tests are RUNNABLE / SELF-SKIPPING: when a precondition is absent (no `longhorn`
StorageClass, no Velero CRDs, <3 Ready nodes) they echo a skip token and `exit 0`, and their
`check:` block is deliberately written to accept either outcome, e.g. tests/l3-fault-
injection/dr-restore:
(contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')
|| contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true
So a SELF-SKIP exits 0 and Chainsaw records `passed` EXACTLY as a real run does. And
Chainsaw's JUnit carries no stdout — its `<testcase>` elements hold only `<properties>`
(verified against the committed reports). Therefore:
*** A COMMITTED JUnit ALONE WOULD RECORD `DR_RESTORE_SKIPPED_OK` AS PROOF OF R-STATE-03. ***
AND THE SECOND TRAP, WHICH IS THE MIRROR OF THE FIRST
------------------------------------------------------
Chainsaw echoes each script's SOURCE into an indented `=== COMMAND` block before running
it, so `echo NODE_LOSS_SKIPPED_OK` — the skip branch that was NOT taken — is present in
the log of every run of that test. A section-blind grep therefore reads a skip token out
of the source of every skip-gated test and can never record `proven` for any of them: the
ledger fails closed so completely that it records nothing and proves nothing. Tokens are
consequently read from SECTIONS (see classify_sections): a pass token only from
`=== STDOUT`, a skip token from any section that is not script source. Skip still beats
pass — within genuine output, which is the only place either was ever evidence.
That is the fake-green this project refuses, and it would be worse than no ledger at all
because it would look authoritative. THE SENTINEL TOKEN IS THE ONLY THING THAT SEPARATES
PROVEN FROM SKIPPED, AND IT EXISTS ONLY IN THE RUN LOG. Hence: this tool requires BOTH
files, and for the Chainsaw layers it will NEVER emit `proof: "proven"` on a JUnit `passed`
unless the log carries that test's own pass token and carries NO skip token.
HOW THE TOKENS ARE OBTAINED (derived, never hand-listed)
--------------------------------------------------------
Hard-coding a token table would rot the moment a test is edited. Instead every token is
parsed out of the test's own `chainsaw-test.yaml`: each `contains($stdout, 'TOKEN')` in a
`check:` block is a declared sentinel. Classification is by the project's naming convention
(`*_SKIP*_OK` is a skip token, anything else is a pass token), and the TERMINAL pass token
— the last non-skip token in file order — is the test's verdict token. Edit a test's tokens
and this tool follows automatically; delete them and the test simply becomes unprovable
rather than silently proven.
node-loss pass NODE_LOSS_VOLUME_AVAILABLE_OK skip NODE_LOSS_SKIPPED_OK
dr-restore pass DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK skip DR_RESTORE_SKIPPED_OK
(+ OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK / DR_SCHEDULE_SKIPPED_OK)
WHY L0/L1 NEED NO SENTINEL (and why that is not a loophole)
-----------------------------------------------------------
The offline layers are not run by Chainsaw. `lib/run_l1_offline.py` shells out to each
dir's `check.sh` / `render-check.sh` and maps exit 3 to a JUnit `<skipped/>` element and
any other non-zero to `<failure>`. A self-skip there is ALREADY a JUnit skip, never a green
pass, so the JUnit is a truthful record on its own. Only the Chainsaw layers (L2/L3/META)
can express "skipped" as "passed", and only they require a sentinel — see
SENTINEL_REQUIRED_LAYERS.
LEDGER ROW (conformance/runs/ledger.jsonl — append-only, one object per testcase per run)
ts_utc run-dir timestamp, else the JUnit's mtime (when)
git_sha `git rev-parse HEAD`, suffixed `-dirty` when tracked files were
modified — a proof captured from a dirty tree is NOT honoured by
status.py, because what ran is not what is committed (what)
env --env / CONF_ENV (cluster overlay the run targeted)
chainsaw_version CHAINSAW_VERSION from versions.env (the pin IS the truth)
kube_version `kubectl version` server gitVersion, else "unknown"
layer L0|L1|L2|L3|META, from the test's `layer` annotation / JUnit property
test testcase name (== the test dir basename, asserted by convention)
requirements[] the `conformance.home.k3s/R-*` labels on the test — the SAME
authoritative source coverage.py uses, not the JUnit classname
junit_result passed | failed | error | skipped
sentinel the DECIDING token grepped from run.log, or null
proof proven | skipped | failed | junit-skipped
| unknown-no-sentinel-declared | unknown-sentinel-absent
run_id the runs/<dir> name (provenance: which log backs this row)
note one line saying WHY this row landed on that proof state
`proven` is emitted on exactly one path (see decide()); every other path is non-green.
RECORDING IS EXPLICIT, NEVER IMPLICIT
-------------------------------------
Invoke this tool by hand, or via `run.sh --record`. A run that silently records itself is
how a bad run becomes a permanent false proof, so `run.sh` without `--record` writes no
ledger row at all.
EXIT CODES (ci/lib.sh convention: 0 pass / 1 fail / 3 skipped-not-green)
0 rows appended.
1 a real failure: unparsable JUnit, unreadable test tree, unwritable ledger.
3 the inputs a ledger row needs are absent (no report.junit.xml, or no run.log).
NOTHING is recorded and this is NEVER green — a run we cannot evidence is not a run.
Pure standard library, matching the harness's no-pip-dependency posture (coverage.py,
status.py). PyYAML lives in catalog/render.py; the harness does not depend on it.
"""
from __future__ import annotations
import argparse
import json
import os
import re
import subprocess
import sys
import time
import xml.etree.ElementTree as ET
from pathlib import Path
_LIB = Path(__file__).resolve().parent
if str(_LIB) not in sys.path:
sys.path.insert(0, str(_LIB))
# Importing a sibling would otherwise litter conformance/lib/__pycache__/ into the working
# tree, and the lint stage scans untracked files. Same guard as status.py.
sys.dont_write_bytecode = True
import coverage as cov # noqa: E402 — reuse the registry/test parsers, don't fork them
CONF_ROOT = _LIB.parent
REPO_ROOT = CONF_ROOT.parent
DEFAULT_LEDGER = CONF_ROOT / "runs" / "ledger.jsonl"
# The ONLY layers exempt from needing a pass sentinel: the offline ones, whose skips
# lib/run_l1_offline.py already renders as JUnit <skipped/> rather than as a green pass
# (see docstring). Written as an EXEMPT set, not a REQUIRED set, so that an unrecognised
# or new layer fails CLOSED — it needs a sentinel — instead of quietly becoming provable
# by JUnit alone.
SENTINEL_EXEMPT_LAYERS = frozenset({"L0", "L1"})
# Layer from the test's subtree. Authoritative over the annotation: coverage.py's layer
# regex stops at the first non-[L/A/M/0-9] character, so `layer: "META"` parses as "M" —
# harmless for its own report, but here a mis-read layer decides whether a sentinel is
# required at all, so it is derived from the path exactly as emit_skipped_junit.py does.
_LAYER_BY_SUBTREE = (("/l0-static/", "L0"), ("/l1-rendering/", "L1"),
("/l2-behavioral/", "L2"), ("/l3-fault-injection/", "L3"),
("/meta/", "META"))
def layer_of(path: Path) -> str:
rel = str(path)
for frag, layer in _LAYER_BY_SUBTREE:
if frag in rel:
return layer
return ""
# A declared sentinel: `contains($stdout, 'TOKEN')` inside a Chainsaw `check:` block.
# Both quote styles are accepted; the repo uses single quotes throughout.
_SENTINEL_RE = re.compile(r"contains\(\s*\$stdout\s*,\s*['\"]([A-Za-z0-9_]+)['\"]\s*\)")
# Naming convention for a SKIP token (tests/l3-fault-injection/*: NODE_LOSS_SKIPPED_OK,
# DR_RESTORE_SKIPPED_OK, DR_SCHEDULE_SKIPPED_OK, VM_DATA_SURVIVAL_SKIPPED_OK, ...).
# Deliberately broad: mis-classifying a PASS token as a skip only ever loses a proof;
# mis-classifying a SKIP token as a pass would manufacture one.
_SKIP_TOKEN_RE = re.compile(r"_SKIP")
# A Chainsaw log line header, as it really appears (Go `t.Log` prefixes the source site and
# pads it with backspaces, and the fields are ANSI-coloured — both are stripped first):
# l.go:52: | 23:51:18 | node-loss | volume-stays-available... | SCRIPT | LOG |
# Field 2 is the test name; it is how output is attributed back to a testcase. The
# `<file>.go:<line>:` prefix is optional so a redirected/plain log still attributes.
_LOG_HEADER_RE = re.compile(
r"^\s*(?:[\w./-]+\.go:\d+:\s*)?"
r"\|\s*\d{1,2}:\d{2}:\d{2}(?:\.\d+)?\s*\|\s*([A-Za-z0-9][A-Za-z0-9._-]*)\s*\|")
# ANSI colour + the backspace padding Go writes ahead of a `t.Log` continuation. Stripped
# before ANY parsing; neither ever appears at the head of a line, so indentation survives.
_ANSI_RE = re.compile(r"\x1b\[[0-9;]*[A-Za-z]")
def _strip_noise(line: str) -> str:
return _ANSI_RE.sub("", line).replace("\b", "")
# ===================================================================== output sections
# THE BUG THIS SECTION MACHINERY EXISTS TO KILL — READ BEFORE CHANGING ANYTHING
# -----------------------------------------------------------------------------
# Chainsaw echoes the script it is ABOUT TO RUN, verbatim, in an indented `=== COMMAND`
# block, and only afterwards prints what the script actually wrote in `=== STDOUT`. Every
# skip-gated test contains its own skip token as SOURCE TEXT (`echo NODE_LOSS_SKIPPED_OK`),
# so a raw whole-log grep finds that token in EVERY run of that test — including runs that
# never took the skip branch. Combined with decide()'s (correct) rule that a skip token
# outranks a pass token, a section-blind scan can never record `proven` for ANY test that
# has a skip gate: the ledger fails closed so hard it records nothing. Observed on
# runs/20260901T235117Z-e9a4b66, where NODE_LOSS_SKIPPED_OK appears twice as source at
# lines 51/62 (`=== COMMAND`) and never in output, yet the run genuinely proved R-STATE-02.
#
# So tokens are attributed to SECTIONS, never to raw lines:
# l.go:52: | 23:51:18 | node-loss | step | CMD | RUN | <- header, no section
# === COMMAND <- marker (indent 8)
# /usr/bin/sh -c set -euo pipefail <- source, indent >= 8
# echo NODE_LOSS_SKIPPED_OK <- SOURCE, not output
# l.go:52: | 23:52:42 | node-loss | step | SCRIPT | LOG | <- indent 4 => section ends
# === STDOUT <- marker
# NODE_LOSS_VOLUME_AVAILABLE_OK <- the REAL emission
#
# A marker is an INDENTED `=== NAME` on a line of its own. Go's own test markers
# (`=== RUN chainsaw`, `=== PAUSE`, `=== CONT`) are at column 0 and carry a test name
# after the verb, so they match neither the indent nor the end-of-line anchor; they are
# additionally named in _GO_TEST_MARKERS so a future indented variant still cannot open a
# section that would swallow the whole log.
_SECTION_MARKER_RE = re.compile(r"^(\s+)=== ([A-Z][A-Z0-9_]*)\s*$")
_GO_TEST_MARKERS = frozenset({"RUN", "PAUSE", "CONT"})
# Pseudo-section for a log that contains NO section markers at all (the plain/redirected
# logs the earlier ledger rows were written from, and the selftest fixtures). Such a log
# never echoes script source, so there is nothing to be confused by and the legacy
# whole-log scan is exactly as safe as it was — narrowing is only needed where a COMMAND
# echo exists. Lines OUTSIDE a section in a log that DOES use sections are attributed to
# "" and are evidence for nothing.
UNSECTIONED = "*"
# A pass token counts ONLY where Chainsaw's own `contains($stdout, ...)` check looks:
# real stdout. Anything else — source echo, stderr, an unrecognised future section — can
# never manufacture a proof.
PASS_SECTIONS = frozenset({"STDOUT", UNSECTIONED})
# Sections that carry SOURCE rather than output. A skip token is honoured in every OTHER
# section (STDOUT, STDERR, any section name added by a later Chainsaw release), because
# over-reading a skip only ever COSTS a proof, while over-reading a pass would fabricate
# one. Unknown section names therefore fail closed in both directions.
SOURCE_SECTIONS = frozenset({"COMMAND"})
def classify_sections(log_text: str) -> list[tuple[str, str]]:
"""[(section, line)] for every line of a run log.
`section` is the enclosing `=== NAME` block, "" for a line inside no block, or
UNSECTIONED for every line when the log declares no sections at all. A block runs from
its marker until the first non-blank line indented LESS than the marker (blank lines do
not end it) or until the next marker; the marker line itself is content of nothing.
"""
rows: list[tuple[str, str]] = []
cur, cur_indent, sectioned = "", 0, False
for raw in log_text.splitlines():
line = _strip_noise(raw)
m = _SECTION_MARKER_RE.match(line)
if m and m.group(2) not in _GO_TEST_MARKERS:
sectioned = True
cur, cur_indent = m.group(2), len(m.group(1))
rows.append(("", line))
continue
if cur:
if not line.strip() or len(line) - len(line.lstrip()) >= cur_indent:
rows.append((cur, line))
continue
cur = ""
rows.append(("", line))
if not sectioned:
return [(UNSECTIONED, line) for _, line in rows]
return rows
def scan_texts(lines) -> tuple[str, str]:
"""(text a PASS token may be read from, text a SKIP token may be read from).
Accepts either [(section, line)] pairs from classify_sections() or a plain list of
strings, which is treated as an unsectioned log (legacy whole-log scanning).
"""
pass_lines, skip_lines = [], []
for item in lines:
section, line = item if isinstance(item, tuple) else (UNSECTIONED, item)
if section in PASS_SECTIONS:
pass_lines.append(line)
if section and section not in SOURCE_SECTIONS:
skip_lines.append(line)
return "\n".join(pass_lines), "\n".join(skip_lines)
_RUN_ID_TS_RE = re.compile(r"^(\d{8}T\d{6}Z)-")
# ========================================================================= sentinels
def parse_sentinels(test_yaml: Path) -> dict:
"""Declared sentinel tokens of one Chainsaw test, derived from the test file itself.
Returns {"pass": [...], "skip": [...], "terminal_pass": str|None}. `terminal_pass` is
the LAST non-skip token in file order — the test's verdict token, i.e. the one its
final `check:` block asserts (dr-restore -> DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK).
A test with no `contains($stdout, ...)` assertion declares no sentinel and is reported
unprovable rather than proven; see decide().
"""
try:
text = test_yaml.read_text()
except (OSError, UnicodeDecodeError):
return {"pass": [], "skip": [], "terminal_pass": None}
toks = _SENTINEL_RE.findall(text)
passes, skips = [], []
for t in toks:
(skips if _SKIP_TOKEN_RE.search(t) else passes).append(t)
# De-duplicate, preserving order (dr-restore names DR_RESTORE_SKIPPED_OK twice).
def uniq(xs: list[str]) -> list[str]:
return list(dict.fromkeys(xs))
passes, skips = uniq(passes), uniq(skips)
return {"pass": passes, "skip": skips,
"terminal_pass": passes[-1] if passes else None}
def sentinel_map(tests_dir: Path) -> dict[str, dict]:
"""{test-name: sentinels} for every test in the tree.
Keyed on the Chainsaw `metadata.name`, which is what lands in the JUnit `<testcase
name=...>`. This repo keeps that equal to the test's directory basename; the mapping
is by name, so it stays correct if that ever drifts.
"""
out: dict[str, dict] = {}
for f in sorted(tests_dir.rglob("chainsaw-test.yaml")):
meta = cov.parse_test(f)
name = meta["name"] or f.parent.name
s = parse_sentinels(f)
s["dir"] = str(f.parent)
s["layer"] = layer_of(f) or meta["layer"]
s["requirements"] = meta["labels"]
out[name] = s
return out
# =============================================================================== log
def attribute_log(log_text: str, known: set[str]) -> tuple[dict[str, list[tuple[str, str]]],
bool]:
"""Split a run log into per-testcase buckets of (section, line) pairs.
Chainsaw prefixes its log lines with `| HH:MM:SS | <test> | <step> | ...` and prints a
script's sections underneath that header, so the test name is carried by a state
machine: a header naming a KNOWN test switches the current bucket, every following line
lands in it. Requiring the name to be known is what stops run.sh's own `[run.sh]`
chatter (and any other pipe-shaped line) from opening a bogus bucket.
Section labels come from classify_sections() and travel WITH the lines, so a bucket
still knows which of its lines were script source and which were real output.
Returns (buckets, attributed) where `attributed` is False when the log contained no
recognisable header at all — the signal for the degraded global scan in decide().
"""
buckets: dict[str, list[tuple[str, str]]] = {}
cur: str | None = None
attributed = False
for section, line in classify_sections(log_text):
m = _LOG_HEADER_RE.match(line)
if m and m.group(1) in known:
cur = m.group(1)
attributed = True
if cur is not None:
buckets.setdefault(cur, []).append((section, line))
return buckets, attributed
def unique_tokens(smap: dict[str, dict]) -> set[str]:
"""Tokens claimed by EXACTLY ONE test in the tree.
Only these may be used by the degraded global scan: `DNS_OK` is echoed by two different
tests, so in an unattributable log it can never be evidence for either; but
`NODE_LOSS_VOLUME_AVAILABLE_OK` names precisely one test and still can.
"""
owners: dict[str, set[str]] = {}
for name, s in smap.items():
for t in s["pass"] + s["skip"]:
owners.setdefault(t, set()).add(name)
return {t for t, o in owners.items() if len(o) == 1}
# ============================================================================ decide
def decide(layer: str, junit_result: str, sent: dict, lines: list[str],
scope: str) -> tuple[str | None, str, str]:
"""The ONE place a testcase becomes `proven` — or, far more often, does not.
Returns (sentinel, proof, note).
Ordering is worst-first ON PURPOSE, so that ambiguity can only ever cost a proof:
1. JUnit failure/error -> failed (a red run is recorded as red)
2. JUnit <skipped/> -> junit-skipped (never green; ci/lib.sh rule)
3. ANY declared skip token seen -> skipped (BEATS a green JUnit and beats
IN REAL OUTPUT the pass token: if any gated
step self-skipped, the run did
not prove the requirement)
4. no sentinel declared -> proven only for L0/L1, whose skips are already
JUnit skips; unknown-no-sentinel-declared for
the Chainsaw layers
5. EVERY declared pass token seen -> proven
6. anything else -> unknown-sentinel-absent
Step 5 requires ALL pass tokens, not just the terminal one: every Chainsaw step here
asserts `pass || skip`, so a missing pass token means a step did not reach its proven
branch, whatever the JUnit says.
Steps 3 and 5 read from SECTION-SCOPED text, never from raw lines: a pass token counts
only in `=== STDOUT`, a skip token counts in any section that is not script source.
That narrows WHERE a token is read from and changes nothing about what it means — skip
still outranks pass, and it outranks it within genuine output, which is the only place
either of them was ever evidence. `lines` may be [(section, line)] pairs or plain
strings (an unsectioned log; see scan_texts / UNSECTIONED).
"""
if junit_result in ("failed", "error"):
return (None, "failed", f"JUnit recorded {junit_result}")
if junit_result == "skipped":
return (None, "junit-skipped",
"JUnit <skipped/> — the test did not execute; a skip is never green")
pass_text, skip_text = scan_texts(lines)
seen_skip = [t for t in sent["skip"] if t in skip_text]
if seen_skip:
return (seen_skip[0], "skipped",
f"run log OUTPUT carries the SELF-SKIP token {seen_skip[0]!r} — the test "
f"exited 0 without exercising the behaviour, so this is NOT proof")
if not sent["pass"] and not sent["skip"]:
if layer not in SENTINEL_EXEMPT_LAYERS:
return (None, "unknown-no-sentinel-declared",
f"{layer or 'unknown-layer'} test declares no contains($stdout, ...) "
f"sentinel, so a green JUnit cannot be told apart from a self-skip — "
f"refusing to call it proven")
return (None, "proven",
f"{layer} offline check exited 0; run_l1_offline.py renders its skips as "
f"JUnit <skipped/>, so a green JUnit is truthful without a sentinel")
missing = [t for t in sent["pass"] if t not in pass_text]
if not missing:
return (sent["terminal_pass"], "proven",
f"run log OUTPUT carries every declared pass token "
f"({', '.join(sent['pass'])}) and no skip token [{scope}]")
return (None, "unknown-sentinel-absent",
f"run log OUTPUT lacks pass token(s) {', '.join(missing)} for this test "
f"[{scope}] — refusing to infer proof from the JUnit alone")
# ============================================================================= junit
def junit_cases(path: Path) -> list[dict]:
"""[{name, layer, requirements, result}] for every <testcase> in a JUnit report."""
root = ET.parse(path).getroot()
cases: list[dict] = []
for c in root.iter("testcase"):
if c.find("failure") is not None:
result = "failed"
elif c.find("error") is not None:
result = "error"
elif c.find("skipped") is not None:
result = "skipped"
else:
result = "passed"
props = {p.get("name"): (p.get("value") or "")
for p in c.iter("property") if p.get("name")}
reqs = sorted(set(cov.RID_RE.findall(props.get("requirements", ""))))
layer = (props.get("layer") or "").upper()
if not layer:
layer = (c.get("classname") or "").split(":", 1)[0].upper()
cases.append({"name": c.get("name") or "?", "layer": layer,
"requirements": reqs, "result": result})
return cases
# =============================================================================== git
def git_sha(repo: Path) -> str:
"""HEAD, suffixed `-dirty` when TRACKED files differ from it.
Untracked files are deliberately ignored, matching status.py's `tracked_files()`
posture (untracked scratch is not repo state) — and because a `--record` run creates
untracked `runs/<id>/` files as it goes, which would otherwise mark every run dirty.
A `-dirty` sha is not a valid rev, so status.py's ancestry check cannot resolve it and
the proof degrades to `stale-proof`: correct, since what ran is not what is committed.
"""
try:
sha = subprocess.run(["git", "-C", str(repo), "rev-parse", "HEAD"],
capture_output=True, text=True, check=True).stdout.strip()
except (subprocess.CalledProcessError, OSError):
return "unknown"
# EXCLUDE conformance/runs/. It is TRACKED, so `--record` writing ledger.jsonl dirtied the
# tree and every LATER run in the same session was stamped `-dirty` — recording one test
# poisoned the next, and check.sh refuses to honour a dirty proven row. That cost a second
# live fault-injection run on prod to work around. The docstring above already states the
# intent for the untracked runs/<id>/ half of this same artifact; the ledger is the tracked
# half. Mirrors status.py's IMPL_EXCLUDE_PREFIXES, which lists this path for the same reason.
dirty = subprocess.run(["git", "-C", str(repo), "diff", "--quiet", "HEAD", "--",
".", ":(exclude)conformance/runs/"],
capture_output=True).returncode != 0
return f"{sha}-dirty" if dirty else sha
def pinned_chainsaw_version(conf_root: Path) -> str:
"""CHAINSAW_VERSION from versions.env — the pin is the single source of truth."""
f = conf_root / "versions.env"
if f.is_file():
m = re.search(r'^CHAINSAW_VERSION="?([^"\n]+)"?', f.read_text(), re.M)
if m:
return m.group(1)
return "unknown"
def server_kube_version() -> str:
"""Server gitVersion via `kubectl version` (read-only), else "unknown"."""
for args in (["kubectl", "version", "-o", "json"],
["kubectl", "version", "--output=json"]):
try:
p = subprocess.run(args, capture_output=True, text=True, timeout=15)
except (OSError, subprocess.TimeoutExpired):
continue
if p.returncode != 0 or not p.stdout.strip():
continue
try:
v = json.loads(p.stdout).get("serverVersion") or {}
except json.JSONDecodeError:
continue
if v.get("gitVersion"):
return v["gitVersion"]
return "unknown"
def run_timestamp(run_dir: Path, junit: Path) -> str:
"""UTC stamp: the run-dir name's own `<UTC>-<sha>` prefix, else the JUnit's mtime."""
m = _RUN_ID_TS_RE.match(run_dir.name)
if m:
t = m.group(1)
return f"{t[0:4]}-{t[4:6]}-{t[6:8]}T{t[9:11]}:{t[11:13]}:{t[13:15]}Z"
return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(junit.stat().st_mtime))
# ============================================================================ record
def build_rows(run_dir: Path, tests_dir: Path, env: str, chainsaw_version: str,
kube_version: str, sha: str) -> list[dict]:
junit = run_dir / "report.junit.xml"
log = run_dir / "run.log"
smap = sentinel_map(tests_dir)
log_text = log.read_text(errors="replace")
buckets, attributed = attribute_log(log_text, set(smap))
uniq = unique_tokens(smap)
all_pairs = classify_sections(log_text)
sectioned = any(sec not in ("", UNSECTIONED) for sec, _ in all_pairs)
section_note = ("tokens read from real-output sections only"
if sectioned else "log declares no === sections; whole-log scan")
ts = run_timestamp(run_dir, junit)
rows: list[dict] = []
for case in junit_cases(junit):
name = case["name"]
sent = smap.get(name, {"pass": [], "skip": [], "terminal_pass": None,
"layer": case["layer"], "requirements": []})
layer = sent.get("layer") or case["layer"] or "?"
# Requirements come from the test's LABELS (coverage.py's authoritative source);
# the JUnit property is the fallback for a testcase with no matching test file.
reqs = sent.get("requirements") or case["requirements"]
if name in buckets:
lines, scope = buckets[name], f"log scoped to this testcase; {section_note}"
elif not attributed:
# Degraded mode: the log had no recognisable Chainsaw header (a redirected or
# reformatted log). Fall back to a whole-log scan, but ONLY for tokens owned by
# exactly one test, so a token another test emitted can never prove this one.
lines = all_pairs
sent = dict(sent,
**{"pass": [t for t in sent["pass"] if t in uniq],
"skip": [t for t in sent["skip"] if t in uniq]})
sent["terminal_pass"] = sent["pass"][-1] if sent["pass"] else None
scope = ("DEGRADED: unattributable log, globally-unique tokens only; "
+ section_note)
else:
lines, scope = [], "no log lines attributed to this testcase"
sentinel, proof, note = decide(layer, case["result"], sent, lines, scope)
rows.append({
"ts_utc": ts,
"git_sha": sha,
"env": env,
"chainsaw_version": chainsaw_version,
"kube_version": kube_version,
"layer": layer,
"test": name,
"requirements": reqs,
"junit_result": case["result"],
"sentinel": sentinel,
"proof": proof,
"run_id": run_dir.name,
"note": note,
})
return rows
def append_rows(ledger: Path, rows: list[dict]) -> None:
"""Append-only, one compact JSON object per line. Never rewrites existing rows."""
ledger.parent.mkdir(parents=True, exist_ok=True)
with ledger.open("a", encoding="utf-8") as fh:
for r in rows:
fh.write(json.dumps(r, sort_keys=True) + "\n")
# ========================================================================== selftest
def _selftest() -> int:
"""Deterministic, no-cluster self-test of every decision rule (house pattern:
`status.py --selftest`, `catalog/render.py --selftest`). Throwaway tree only.
The load-bearing case is `SKIP sentinel does NOT produce proven`: it is the single
assertion standing between this ledger and a permanent false proof of R-STATE-03.
"""
import tempfile
ok = True
def check(label: str, cond: bool) -> None:
nonlocal ok
if not cond:
sys.stderr.write(f"record_run.py selftest: FAIL — {label}\n")
ok = False
# --- token derivation, against the dr-restore/node-loss shape ---------------------
with tempfile.TemporaryDirectory() as td:
root = Path(td)
tdir = root / "tests" / "l3-fault-injection" / "dr-restore"
tdir.mkdir(parents=True)
(tdir / "chainsaw-test.yaml").write_text(
"apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n"
" name: dr-restore\n labels:\n"
" conformance.home.k3s/R-STATE-03: \"true\"\n annotations:\n"
" conformance.home.k3s/requirements: \"R-STATE-03\"\n"
" conformance.home.k3s/layer: \"L3\"\nspec:\n steps:\n"
" - script:\n content: 'echo OFF_CLUSTER_TARGET_OK'\n"
" check:\n"
" (contains($stdout, 'OFF_CLUSTER_TARGET_OK')"
" || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true\n"
" - script:\n content: 'echo x'\n check:\n"
" (contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')"
" || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true\n")
s = parse_sentinels(tdir / "chainsaw-test.yaml")
check("pass tokens derived from the test file",
s["pass"] == ["OFF_CLUSTER_TARGET_OK",
"DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"])
check("skip token classified by the *_SKIP*_OK convention, de-duplicated",
s["skip"] == ["DR_RESTORE_SKIPPED_OK"])
check("terminal pass token is the LAST non-skip token",
s["terminal_pass"] == "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK")
smap = sentinel_map(root / "tests")
check("sentinel_map keys on metadata.name and carries layer + labels",
smap["dr-restore"]["layer"] == "L3"
and smap["dr-restore"]["requirements"] == ["R-STATE-03"])
# --- decide(): the SKIP-is-never-proven contract ----------------------------------
l3 = {"pass": ["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"],
"skip": ["DR_RESTORE_SKIPPED_OK"],
"terminal_pass": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"}
# *** THE CASE THIS WHOLE FILE EXISTS FOR ***
sent, proof, _ = decide("L3", "passed", l3, ["DR_RESTORE_SKIPPED_OK"], "t")
check("SKIP sentinel on a GREEN JUnit is 'skipped', NEVER 'proven'",
proof == "skipped" and sent == "DR_RESTORE_SKIPPED_OK")
_, proof, _ = decide("L3", "passed", l3,
["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"], "t")
check("pass sentinel on a green JUnit is 'proven'", proof == "proven")
_, proof, _ = decide("L3", "passed", l3,
["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK",
"DR_RESTORE_SKIPPED_OK"], "t")
check("a skip token BEATS a pass token in the same log (worst wins)",
proof == "skipped")
_, proof, _ = decide("L3", "passed", l3, ["nothing relevant here"], "t")
check("green JUnit with no sentinel in the log is unknown, not proven",
proof == "unknown-sentinel-absent")
_, proof, _ = decide("L3", "passed", l3, [], "t")
check("green JUnit with NO log lines attributed is unknown, not proven",
proof == "unknown-sentinel-absent")
_, proof, _ = decide("L3", "failed", l3,
["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"], "t")
check("a failing JUnit outranks a pass token in the log", proof == "failed")
_, proof, _ = decide("L3", "skipped", l3, [], "t")
check("JUnit <skipped/> records junit-skipped, never green",
proof == "junit-skipped")
multi = {"pass": ["A_OK", "B_OK"], "skip": [], "terminal_pass": "B_OK"}
_, proof, _ = decide("L3", "passed", multi, ["B_OK"], "t")
check("every declared pass token must be present, not just the terminal one",
proof == "unknown-sentinel-absent")
_, proof, _ = decide("L3", "passed", multi, ["A_OK", "B_OK"], "t")
check("all pass tokens present == proven", proof == "proven")
none = {"pass": [], "skip": [], "terminal_pass": None}
_, proof, _ = decide("L2", "passed", none, ["whatever"], "t")
check("a Chainsaw test declaring no sentinel is NOT proven by a green JUnit",
proof == "unknown-no-sentinel-declared")
_, proof, _ = decide("L1", "passed", none, [], "t")
check("L0/L1 offline checks are proven by JUnit (their skips are JUnit skips)",
proof == "proven")
_, proof, _ = decide("L0", "skipped", none, [], "t")
check("an L0 JUnit skip is still never green", proof == "junit-skipped")
_, proof, _ = decide("META", "passed", none, [], "t")
check("META (a Chainsaw layer) needs a sentinel like L2/L3",
proof == "unknown-no-sentinel-declared")
_, proof, _ = decide("", "passed", none, [], "t")
check("an unknown/blank layer fails CLOSED — sentinel required",
proof == "unknown-no-sentinel-declared")
# --- SECTION SCOPING: the COMMAND echo is SOURCE, never evidence -------------------
# Regression cases for the ledger's first real L3 run, which recorded a genuine PASS as
# `skipped` because `echo NODE_LOSS_SKIPPED_OK` — the skip branch NOT taken — is echoed
# inside `=== COMMAND` on every run of a skip-gated test.
nl = {"pass": ["NODE_LOSS_VOLUME_AVAILABLE_OK"],
"skip": ["NODE_LOSS_SKIPPED_OK"],
"terminal_pass": "NODE_LOSS_VOLUME_AVAILABLE_OK"}
_CMD = (" l.go:52: | 23:51:18 | node-loss | step | CMD | RUN |\n"
" === COMMAND\n"
" /usr/bin/sh -c set -euo pipefail\n"
" if ! kubectl get storageclass longhorn >/dev/null 2>&1; then\n"
" echo NODE_LOSS_SKIPPED_OK\n"
"\n"
" exit 0\n"
" fi\n"
" echo NODE_LOSS_VOLUME_AVAILABLE_OK\n"
" l.go:52: | 23:52:42 | node-loss | step | SCRIPT | LOG |\n")
ran = classify_sections(_CMD + " === STDOUT\n"
" ready nodes: 3\n"
" NODE_LOSS_VOLUME_AVAILABLE_OK\n"
" l.go:52: | 23:52:42 | node-loss | step | SCRIPT | DONE |\n")
sent, proof, _ = decide("L3", "passed", nl, ran, "t")
check("skip token ONLY in the COMMAND echo + pass token in STDOUT is PROVEN",
proof == "proven" and sent == "NODE_LOSS_VOLUME_AVAILABLE_OK")
selfskipped = classify_sections(_CMD + " === STDOUT\n"
" SKIP: 'longhorn' StorageClass absent\n"
" NODE_LOSS_SKIPPED_OK\n")
sent, proof, _ = decide("L3", "passed", nl, selfskipped, "t")
check("skip token GENUINELY in STDOUT is still 'skipped' (the guarantee holds)",
proof == "skipped" and sent == "NODE_LOSS_SKIPPED_OK")
neither = classify_sections(_CMD + " === STDOUT\n ready nodes: 3\n")
_, proof, _ = decide("L3", "passed", nl, neither, "t")
check("both tokens in COMMAND and neither in STDOUT is NOT proven",
proof == "unknown-sentinel-absent")
both_out = classify_sections(" === STDOUT\n"
" NODE_LOSS_VOLUME_AVAILABLE_OK\n"
" NODE_LOSS_SKIPPED_OK\n")
_, proof, _ = decide("L3", "passed", nl, both_out, "t")
check("skip still BEATS pass when both are in genuine output", proof == "skipped")
stderr_skip = classify_sections(" === STDERR\n NODE_LOSS_SKIPPED_OK\n"
" === STDOUT\n"
" NODE_LOSS_VOLUME_AVAILABLE_OK\n")
_, proof, _ = decide("L3", "passed", nl, stderr_skip, "t")
check("a skip token in a non-source section (STDERR) still costs the proof",
proof == "skipped")
stderr_pass = classify_sections(" === STDERR\n"
" NODE_LOSS_VOLUME_AVAILABLE_OK\n")
_, proof, _ = decide("L3", "passed", nl, stderr_pass, "t")
check("a pass token OUTSIDE stdout cannot prove anything",
proof == "unknown-sentinel-absent")
# --- classify_sections: markers, extents, and Go's own `=== RUN` --------------------
gorows = classify_sections("=== RUN chainsaw\n=== PAUSE chainsaw\n"
"=== CONT chainsaw/node-loss\n"
" === COMMAND\n echo NODE_LOSS_SKIPPED_OK\n"
" l.go:52: | 1:00:00 | node-loss | s | SCRIPT | DONE |\n"
"--- PASS: chainsaw/node-loss (131.09s)\n")
secs = [sec for sec, _ in gorows]
check("Go's column-0 `=== RUN/PAUSE/CONT` never open a section",
secs[:3] == ["", "", ""])
check("a section covers its indented body and ends at a less-indented line",
secs[3] == "" and secs[4] == "COMMAND" and secs[5] == "" and secs[6] == "")
check("a log with no === markers stays wholly scannable (UNSECTIONED)",
{sec for sec, _ in classify_sections("plain line\n another\n")} == {UNSECTIONED})
# --- the preserved run itself, as a fixture ----------------------------------------
# runs/20260901T235117Z-e9a4b66 IS the bug: NODE_LOSS_SKIPPED_OK twice in `=== COMMAND`
# (source), never in output; NODE_LOSS_VOLUME_AVAILABLE_OK once in `=== STDOUT`.
fixture = CONF_ROOT / "runs" / "20260901T235117Z-e9a4b66"
if (fixture / "run.log").is_file() and (fixture / "report.junit.xml").is_file():
rows = build_rows(fixture, CONF_ROOT / "tests", "dev", "v0.2.12", "v1.36.4", "sha")
row = next((r for r in rows if r["test"] == "node-loss"), None)
check("preserved real L3 run records node-loss/R-STATE-02 as proven",
row is not None and row["proof"] == "proven"
and row["sentinel"] == "NODE_LOSS_VOLUME_AVAILABLE_OK"
and row["requirements"] == ["R-STATE-02"])
check("preserved real L3 run attributes (no DEGRADED fallback needed)",
row is not None and "DEGRADED" not in row["note"])
pairs = classify_sections((fixture / "run.log").read_text(errors="replace"))
secs = {sec for sec, line in pairs if "NODE_LOSS_SKIPPED_OK" in line}
check("every NODE_LOSS_SKIPPED_OK in the preserved log is COMMAND source",
secs == {"COMMAND"})
secs = {sec for sec, line in pairs
if line.strip() == "NODE_LOSS_VOLUME_AVAILABLE_OK"}
check("the real emission in the preserved log is in STDOUT", secs == {"STDOUT"})
# --- log attribution --------------------------------------------------------------
log = ("[run.sh] Coverage report first\n"
" | 20:41:32 | node-loss | step | SCRIPT | RUN |\n"
" NODE_LOSS_SKIPPED_OK\n"
" | 20:44:01 | dr-restore | step | SCRIPT | RUN |\n"
" DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK\n")
b, attributed = attribute_log(log, {"node-loss", "dr-restore"})
check("log lines are attributed to the testcase whose header preceded them",
attributed
and any("NODE_LOSS_SKIPPED_OK" in l for _, l in b["node-loss"])
and not any("NODE_LOSS_SKIPPED_OK" in l for _, l in b["dr-restore"]))
check("a pass token does NOT leak across testcase boundaries",
not any("DR_RESTORED" in l for _, l in b["node-loss"]))
check("an unsectioned log keeps whole-log scanning (nothing narrowed away)",
all(sec == UNSECTIONED for sec, _ in b["node-loss"]))
# Chainsaw's real headers carry a Go `t.Log` source prefix and ANSI colour; the earlier
# regex matched neither, so every genuine Chainsaw log silently fell into DEGRADED mode.
real_hdr = (" l.go:52: \x08\x08\x08| 23:51:18 | \x1b[32;1mnode-loss\x1b[0;22m | "
"\x1b[32;1mstep\x1b[0;22m | SCRIPT | LOG |\n"
" === STDOUT\n NODE_LOSS_VOLUME_AVAILABLE_OK\n")
b2, attributed_real = attribute_log(real_hdr, {"node-loss"})
check("a REAL Chainsaw header (go source prefix + ANSI) attributes, not degrades",
attributed_real and "node-loss" in b2
and ("STDOUT", " NODE_LOSS_VOLUME_AVAILABLE_OK") in b2["node-loss"])
_, attributed2 = attribute_log("no chainsaw headers at all\n", {"node-loss"})
check("an unattributable log is reported as such (degraded mode)", not attributed2)
smap2 = {"a": {"pass": ["DNS_OK", "A_ONLY_OK"], "skip": []},
"b": {"pass": ["DNS_OK"], "skip": []}}
check("only tokens owned by exactly one test may be used in degraded mode",
unique_tokens(smap2) == {"A_ONLY_OK"})
# --- end-to-end: a self-skipped L3 run must land in the ledger as `skipped` --------
with tempfile.TemporaryDirectory() as td:
root = Path(td)
tdir = root / "tests" / "l3-fault-injection" / "node-loss"
tdir.mkdir(parents=True)
(tdir / "chainsaw-test.yaml").write_text(
"kind: Test\nmetadata:\n name: node-loss\n labels:\n"
" conformance.home.k3s/R-STATE-02: \"true\"\n annotations:\n"
" conformance.home.k3s/requirements: \"R-STATE-02\"\n"
" conformance.home.k3s/layer: \"L3\"\nspec:\n steps:\n"
" - script:\n content: 'echo hi'\n check:\n"
" (contains($stdout, 'NODE_LOSS_VOLUME_AVAILABLE_OK')"
" || contains($stdout, 'NODE_LOSS_SKIPPED_OK')): true\n")
rd = root / "runs" / "20260901T120000Z-abc1234"
rd.mkdir(parents=True)
(rd / "report.junit.xml").write_text(
'<testsuites><testsuite><testcase classname="L3:R-STATE-02" '
'name="node-loss"><properties>'
'<property name="requirements" value="R-STATE-02"/>'
'<property name="layer" value="L3"/></properties></testcase>'
"</testsuite></testsuites>")
(rd / "run.log").write_text(
" | 12:00:01 | node-loss | s | SCRIPT | RUN |\n"
" SKIP: 'longhorn' StorageClass absent\n"
" NODE_LOSS_SKIPPED_OK\n")
rows = build_rows(rd, root / "tests", "dev", "v0.2.12", "v1.36.4", "abc1234")
check("end-to-end: Chainsaw 'passed' + skip token => proof 'skipped'",
len(rows) == 1 and rows[0]["junit_result"] == "passed"
and rows[0]["proof"] == "skipped"
and rows[0]["sentinel"] == "NODE_LOSS_SKIPPED_OK")
check("end-to-end: row carries ts from the run-dir name and the R-ID label",
rows[0]["ts_utc"] == "2026-09-01T12:00:00Z"
and rows[0]["requirements"] == ["R-STATE-02"])
(rd / "run.log").write_text(
" | 12:00:01 | node-loss | s | SCRIPT | RUN |\n"
" NODE_LOSS_VOLUME_AVAILABLE_OK\n")
rows = build_rows(rd, root / "tests", "dev", "v0.2.12", "v1.36.4", "abc1234")
check("end-to-end: Chainsaw 'passed' + pass token => proof 'proven'",
rows[0]["proof"] == "proven"
and rows[0]["sentinel"] == "NODE_LOSS_VOLUME_AVAILABLE_OK")
ledger = root / "ledger.jsonl"
append_rows(ledger, rows)
append_rows(ledger, rows)
check("ledger is append-only JSONL (one parsable object per line)",
len(ledger.read_text().strip().splitlines()) == 2
and json.loads(ledger.read_text().splitlines()[0])["test"] == "node-loss")
if not ok:
return 1
sys.stderr.write("record_run.py selftest: PASS\n")
return 0
# ============================================================================== main
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--run-dir", type=Path,
help="conformance/runs/<UTC>-<sha>/ holding report.junit.xml + run.log")
ap.add_argument("--ledger", type=Path, default=DEFAULT_LEDGER,
help=f"append-only ledger (default {DEFAULT_LEDGER})")
ap.add_argument("--tests", type=Path, default=CONF_ROOT / "tests")
ap.add_argument("--repo", type=Path, default=REPO_ROOT)
ap.add_argument("--env", default=os.environ.get("CONF_ENV", "dev"))
ap.add_argument("--chainsaw-version", default=None,
help="override the versions.env pin (recorded verbatim)")
ap.add_argument("--kube-version", default=os.environ.get("CONF_KUBE_VERSION"),
help="override the `kubectl version` probe (recorded verbatim)")
ap.add_argument("--dry-run", action="store_true",
help="print the rows that WOULD be appended; write nothing")
ap.add_argument("--selftest", action="store_true", help="run the built-in self-test")
args = ap.parse_args()
if args.selftest:
return _selftest()
if args.run_dir is None:
sys.stderr.write("[record] --run-dir is required (or use --selftest)\n")
return 1
run_dir = args.run_dir.resolve()
junit, log = run_dir / "report.junit.xml", run_dir / "run.log"
# BOTH files or nothing. The JUnit alone cannot tell a self-skip from a real pass, so
# recording from it alone is exactly the false proof this tool exists to prevent.
missing = [p.name for p in (junit, log) if not p.is_file()]
if missing:
sys.stderr.write(
f"[record] {run_dir} is missing {', '.join(missing)} — nothing recorded. "
f"The JUnit alone cannot distinguish a self-skip from a real pass "
f"(see module docstring). SKIPPED (not green).\n")
return 3
try:
rows = build_rows(
run_dir, args.tests, args.env,
args.chainsaw_version or pinned_chainsaw_version(CONF_ROOT),
args.kube_version or server_kube_version(),
git_sha(args.repo))
except ET.ParseError as e:
sys.stderr.write(f"[record] {junit} is not parsable JUnit: {e}\n")
return 1
except OSError as e:
sys.stderr.write(f"[record] cannot read the run inputs: {e}\n")
return 1
if not rows:
sys.stderr.write(f"[record] {junit} contains no <testcase> — nothing to record. "
f"SKIPPED (not green).\n")
return 3
for r in rows:
print(f"[record] {r['layer']:4s} {r['proof']:28s} {r['test']} "
f"({','.join(r['requirements']) or 'untagged'}) "
f"junit={r['junit_result']} sentinel={r['sentinel'] or '—'}")
counts: dict[str, int] = {}
for r in rows:
counts[r["proof"]] = counts.get(r["proof"], 0) + 1
print(f"[record] {len(rows)} testcase(s): "
+ ", ".join(f"{k}={v}" for k, v in sorted(counts.items())))
if args.dry_run:
print("[record] --dry-run: ledger NOT written.")
return 0
try:
append_rows(args.ledger, rows)
except OSError as e:
sys.stderr.write(f"[record] cannot append to {args.ledger}: {e}\n")
return 1
print(f"[record] appended {len(rows)} row(s) -> {args.ledger}")
print("[record] COMMIT the run dir and the ledger; an uncommitted ledger proves nothing.")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+90 -8
View File
@@ -19,6 +19,30 @@ Exit 0 iff every selected offline check passes.
The per-dir check script is whichever of `render-check.sh` / `check.sh` the directory
carries; a dir with neither is reported SKIPPED (never a pass).
DECLARED SCOPE, AND WHY A SKIP IS NOT ALWAYS HARMLESS (--unwired-scope)
----------------------------------------------------------------------
Exit 0 here means "every selected check passed"; a check that exits 3 is SKIPPED and, by
default, does not fail the run (a scaffold with no check script, an audit whose subject
subtree does not exist yet). That default is WRONG for a WHOLE-REPO invariant. Those are the
checks ci/l0.sh used to run as hand-written "UNCONDITIONAL" stages, and each of those stages
mapped rc 3 to the GATE's rc 3 with the words "not a pass":
3) skip "substitution check SKIPPED (clusters/ absent) — not a pass"
summary "- subst-safety: **SKIPPED** (nothing to audit)"
if [ "$rc" -eq 0 ]; then rc=3; fi
A whole-repo check reporting "nothing to audit" is reporting that the tree it audits has
VANISHED — which is a louder problem than the check failing, not a quieter one. So:
--unwired-scope SCOPE a test DECLARING `<prefix>/scope: SCOPE` whose check SKIPS (rc 3,
or no check script at all) makes this runner exit 3 instead of 0.
Hard failures still win (exit 1); rc 3 never masks one.
Exit codes: 0 all selected checks passed / 1 at least one FAILED / 3 nothing failed but a
scope-declared check was UNWIRED. conformance/run.sh propagates 3 verbatim; ci/l0.sh turns it
into the gate's own rc 3. This is the ONE place the "unwired, not green" rule lives now,
instead of once per hand-written stage.
"""
from __future__ import annotations
@@ -36,6 +60,24 @@ LABEL_LINE_RE_TMPL = r"^\s*{prefix}/(R-[A-Z0-9]+-[0-9]+)\s*:\s*[\"']?true"
CHECK_SCRIPTS = ("render-check.sh", "check.sh")
def scope_of(test_yaml: Path, annotation: str) -> str:
"""The scope a test DECLARES, or "" when it declares none.
Matched on a real YAML mapping line: a `#` comment cannot begin with the annotation key,
so a scope named in the test's header prose is documentation, not a declaration — the
same "reference, not declaration" rule ci/touched-requirements.sh applies to R-IDs.
Mirrors conf_dirs_for_scope() in lib/common.sh; they must agree.
"""
if not test_yaml.exists():
return ""
rx = re.compile(r"^\s*" + re.escape(annotation) + r"\s*:\s*[\"']?([A-Za-z0-9_.-]+)[\"']?\s*$")
for line in test_yaml.read_text().splitlines():
m = rx.match(line)
if m:
return m.group(1)
return ""
def layer_of(d: Path) -> str:
"""Layer for the JUnit properties, from the subtree the test lives in."""
rel = str(d)
@@ -59,25 +101,38 @@ def main() -> int:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--junit", type=Path, required=True)
ap.add_argument("--label-prefix", default="conformance.home.k3s")
ap.add_argument("--scope-annotation", default="conformance.home.k3s/scope",
help="annotation key a test uses to DECLARE its invariant's scope")
ap.add_argument("--unwired-scope", action="append", default=[], metavar="SCOPE",
help="a test declaring this scope that SKIPS is UNWIRED, not green "
"(makes this runner exit 3); repeatable")
ap.add_argument("--summary-out", type=Path, default=None,
help="write one Markdown line per case here, for the CI step summary")
ap.add_argument("dirs", nargs="+", type=Path)
args = ap.parse_args()
cases = [] # (name, reqs, dir, rc, output)
strict_scopes = set(args.unwired_scope)
cases = [] # (name, reqs, dir, rc, output, scope)
for d in args.dirs:
check = next((d / s for s in CHECK_SCRIPTS if (d / s).exists()), None)
name, reqs = collect_meta(d / "chainsaw-test.yaml", args.label_prefix)
scope = scope_of(d / "chainsaw-test.yaml", args.scope_annotation)
if check is None:
cases.append((name, reqs, d, 3,
f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}"))
f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}",
scope))
continue
proc = subprocess.run(
["bash", str(check)], capture_output=True, text=True
)
out = (proc.stdout + proc.stderr).strip()
cases.append((name, reqs, d, proc.returncode, out))
cases.append((name, reqs, d, proc.returncode, out, scope))
failures = sum(1 for c in cases if c[3] not in (0, 3))
skipped = sum(1 for c in cases if c[3] == 3)
# A SKIP of a scope-declared check is UNWIRED, not green (see the module docstring).
unwired = [c for c in cases if c[3] == 3 and c[5] in strict_scopes]
suite = ET.Element(
"testsuite",
@@ -89,7 +144,8 @@ def main() -> int:
"skipped": str(skipped),
},
)
for name, reqs, d, rc, out in cases:
summary_lines = []
for name, reqs, d, rc, out, scope in cases:
tc = ET.SubElement(
suite,
"testcase",
@@ -100,17 +156,33 @@ def main() -> int:
ET.SubElement(props, "property", {"name": "requirements", "value": ",".join(reqs)})
ET.SubElement(props, "property", {"name": "layer", "value": layer_of(d)})
ET.SubElement(props, "property", {"name": "path", "value": str(d)})
ET.SubElement(props, "property", {"name": "scope", "value": scope})
if rc == 3:
ET.SubElement(tc, "skipped", {"message": out or "no offline check script"})
elif rc != 0:
fail = ET.SubElement(tc, "failure", {"message": f"offline check exit {rc}"})
fail.text = out
status = "PASS" if rc == 0 else ("SKIP" if rc == 3 else "FAIL")
mark = f" scope={scope}" if scope else ""
print(f"[offline] {layer_of(d):3s} {status:4s} {name} "
f"({','.join(reqs) or 'untagged'})",
f"({','.join(reqs) or 'untagged'}){mark}",
file=sys.stderr)
if rc not in (0, 3):
print(out, file=sys.stderr)
# One step-summary line per case, naming what ran — the generated replacement for the
# hand-written `summary "- subst-safety: **passed**"` line each deleted stage carried.
tag = f" ({scope})" if scope else ""
if rc == 0:
verdict = "**passed**"
elif rc == 3:
verdict = ("**SKIPPED** (UNWIRED, not green)" if scope in strict_scopes
else "**SKIPPED** (nothing to audit)")
else:
verdict = f"**FAILED** (exit {rc})"
summary_lines.append(f"- {layer_of(d).lower()}/{name}{tag}: {verdict}")
if rc == 3 and scope in strict_scopes:
print(f"[offline] {name}: SKIPPED, and it declares scope={scope!r} — "
f"UNWIRED, NOT GREEN. {out}", file=sys.stderr)
root = ET.Element(
"testsuites",
@@ -126,9 +198,19 @@ def main() -> int:
args.junit.parent.mkdir(parents=True, exist_ok=True)
xml = minidom.parseString(ET.tostring(root, "utf-8")).toprettyxml(indent=" ")
args.junit.write_text(xml)
print(f"[offline] wrote JUnit ({len(cases)} case(s), {failures} failure(s)) -> "
f"{args.junit}", file=sys.stderr)
return 1 if failures else 0
if args.summary_out is not None:
args.summary_out.parent.mkdir(parents=True, exist_ok=True)
args.summary_out.write_text("\n".join(summary_lines) + ("\n" if summary_lines else ""))
print(f"[offline] wrote JUnit ({len(cases)} case(s), {failures} failure(s), "
f"{skipped} skip(s), {len(unwired)} unwired) -> {args.junit}", file=sys.stderr)
if failures:
return 1
if unwired:
print("[offline] UNWIRED (declared scope, SKIPPED — not a pass): "
+ ", ".join(c[0] for c in unwired), file=sys.stderr)
return 3
return 0
if __name__ == "__main__":
+668 -59
View File
@@ -15,26 +15,42 @@ This generator derives, from the REPOSITORY ALONE, three orthogonal axes per req
from a Flux Kustomization entrypoint (i.e. actually reconciled)?
2. VERIFICATION — is there a requirement-tagged test, at which layer, and is it an
EXECUTABLE test or a TODO scaffold?
3. PROOF — has that test actually passed? *** DELIBERATELY NOT DERIVED. ***
3. PROOF — has that test actually passed? Derived ONLY from the committed
run ledger, and ONLY as far as that ledger's evidence reaches.
WHAT THIS TOOL REFUSES TO DERIVE (and why)
------------------------------------------
PROOF-OF-PASS IS NOT DERIVABLE FROM REPO STATE. Nothing tracked in git records a test
result: `conformance/_artifacts/` and `results/` are git-ignored, and the only prose
record of a green run lives in DECISIONS-LOG.md — hand-maintained, unstructured, and
exactly the kind of source the operator has ruled out. The project's own culture is that
"merged" is never "done" (agents/definition-of-done.md, R-TEST-03), so this report will
NOT infer, guess, or default a requirement to green. Every requirement's proof state is
reported as `not-derivable`, and the headline says so. If an untracked JUnit artifact
happens to be lying in the workspace it is surfaced ONLY as a quarantined advisory with
its mtime — never counted, never mixed into a status.
PROOF — WHAT CHANGED, AND WHAT DID NOT
--------------------------------------
This report used to say `not-derivable` for every requirement, because nothing tracked in
git recorded a test result and the only prose record of a green run lived in
DECISIONS-LOG.md — hand-maintained, unstructured, exactly the source the operator ruled
out. The fix this docstring asked for now exists: `conformance/runs/ledger.jsonl`, an
append-only ledger written by `lib/record_run.py` from a run's JUnit **and its run log**
(see conformance/.gitignore for the negations that keep both in git).
The honest fix is a committed, machine-readable run ledger; until that exists, the most
this report can say is "wired and testable", never "proven".
THE REFUSAL POSTURE IS UNCHANGED. Evidence replaced "not derivable" only where evidence
exists; nowhere is optimism substituted for it:
* A ledger row reaches `proven` only if `record_run.py` saw that test's own PASS
sentinel token in the run log. The L2/L3 tests SELF-SKIP GREEN — Chainsaw records
`passed` on `DR_RESTORE_SKIPPED_OK` exactly as on a real run, and its JUnit carries no
stdout — so a skip token maps to `skipped` and can never become proof.
* A proof is honoured only while it still describes the current code: if any of the
requirement's implementing manifests or verifying tests changed in a commit that is
NOT an ancestor of the run's `git_sha`, the row degrades to `stale-proof`, which is
not green. A run recorded from a dirty tree (`<sha>-dirty`) is never honoured.
* A requirement with no ledger row, or whose verifying tests did not ALL report proven,
stays `not-derivable` / `unknown` — never rounded up.
* The untracked `_artifacts/report.junit.xml` is STILL only a quarantined advisory with
its mtime. It is not the ledger, it is not evidence, and it feeds no status.
With an empty or absent ledger this report behaves exactly as it did before: every
requirement `not-derivable`, headline PROVEN 0, "the most this report can say is wired and
testable, never proven".
READS (all read-only)
requirements/registry.yaml the authoritative R-* mirror (via lib/coverage.py)
tests/**/chainsaw-test.yaml requirement tags + layer + executable annotation
runs/ledger.jsonl COMMITTED proof-of-pass rows (via lib/record_run.py)
clusters/<env>/ Flux entrypoint -> the reconciliation graph
`git ls-files` TRACKED files only; untracked scratch is not repo state
../home-network-k3s/docs/00-vision-and-scope.md OPTIONAL registry<->spec parity check
@@ -92,7 +108,14 @@ FLUX_API = "kustomize.toolkit.fluxcd.io"
# conformance/tests — those are the TESTS; they are the VERIFICATION axis, counted there.
# ci/tests — gate unit-test fixtures embed example R-IDs as DATA.
# registry.yaml — the requirement mirror itself, not an implementation of anything.
IMPL_EXCLUDE_PREFIXES = ("conformance/tests/", "ci/tests/")
# `conformance/runs/` is excluded for a subtler reason than the test dirs: the ledger and the
# preserved run logs CONTAIN the requirement IDs they are evidence FOR. Counting them as
# implementing evidence makes `last_change` move every time proof is recorded, so a row's
# git_sha always predates it and EVERY proof degrades to stale-proof the instant it is
# committed — recording the proof destroys it. The ledger is the proof record; it must not
# participate in the freshness computation about itself. (Found live 2026-09-01 on the first
# PROVEN row.)
IMPL_EXCLUDE_PREFIXES = ("conformance/tests/", "ci/tests/", "conformance/runs/")
IMPL_EXCLUDE_PATHS = ("conformance/requirements/registry.yaml",)
@@ -274,6 +297,109 @@ _EXECUTABLE_RE = re.compile(
_BLOCKED_RE = re.compile(
r"^\s*" + re.escape(LABEL_PREFIX) + r"/blocked-by:\s*(.+)$", re.M)
# ---------------------------------------------------------- scaffold-marker detection
# WHY THIS IS NOT A WHOLE-FILE `"TODO" in text`
# ---------------------------------------------------
# It used to be, and that mis-reported a SAFETY-CRITICAL requirement. A test whose SUBJECT
# is placeholder detection has to name the idioms it hunts for:
# `l0-static/placeholder-guard-coupling/check.sh` carries
# `SEED = ["TBD", "TODO", "FIXME", "PLACEHOLDER", ...]` — its detection vocabulary, as data.
# A whole-file substring test read that as the test declaring itself unfinished, classified
# a 32-assertion executable check as a scaffold, raised a permanent `--strict` annotation
# conflict, and (because the same file is tagged R-STATE-03) reported R-STATE-03 as
# `exec=1 scaffold=1`. A previous engineer worked around it by rewording prose.
#
# The rule below is derived from the corpus (43 tests), not invented. Every genuine
# scaffold in this tree — `l2-behavioral/onboarding-completeness`, `l3-fault-injection/
# alerting`, `l3-fault-injection/gitops-drift`, and no others — declares itself in BOTH of
# two structural positions, and no executable test does either:
#
# 1. a `# STATUS: ... scaffold` line in the file's documentation header. This is the
# repo's DEDICATED declaration slot (tests/README.md): 3 of 43 tests say
# `STATUS: scaffold` and they are exactly the three scaffolds; 19 others declare
# `STATUS: EXECUTABLE` / `IMPLEMENTED` / `RUNNABLE`; the rest declare nothing.
# 2. a TODO/FIXME marker in the BODY that CARRIES A MESSAGE — `echo "TODO: assert PVC
# bound ..."`, `TODO(T-4.1): hermetic SMTP capture sink`. A marker that carries a
# message is a note about work that is missing. A marker that is the ENTIRE content of
# a quoted literal (`"TODO",` in a vocabulary list) carries no message: it is the
# marker as a *datum*, which is what a placeholder DETECTOR necessarily contains.
#
# Both are still evaluated over the body of `chainsaw-test.yaml` AND of any
# `render-check.sh` / `check.sh`, and either one still OUTRANKS an
# `executable: "true"` annotation (see classify_test) — the invariant that an annotation
# must never outrank the body is unchanged. Ambiguity still only ever DOWNGRADES: an
# unrecognised marker shape, a marker in body code outside a quoted literal, or a marker
# carrying any trailing text all classify as scaffold.
#
# KNOWN, ACCEPTED IMPRECISION (stated rather than hidden): a test that embeds a marker with
# text attached but as data — e.g. `re.compile(r"TODO|FIXME")`, or a single string holding
# several idioms — is still called a scaffold. That is the conservative direction, and the
# author's remedy is the same structural discipline the corpus already uses: keep each
# idiom a standalone literal in a list.
# Vocabulary: only the idioms that DECLARE unfinished work. Deliberately excludes `TBD`,
# `PLACEHOLDER`, `XXX`, `REPLACE-ME` — tests in this tree legitimately carry those in
# executable code as data (`l0-static/egress-paths-selected-by-rules/check.sh` compares
# authored values against `"TBD"`; `l0-static/no-plaintext-secrets/check.sh` matches
# `placeholder.*`), and none of them is used here to mark a placeholder body.
# Case-insensitive: no English word contains either token, and the corpus carries no
# lower-case occurrence, so folding case only ever ADDS detection (`echo "todo: ..."`).
_MARKER_RE = re.compile(r"\b(?:TODO|FIXME)\b", re.I)
# The dedicated declaration slot, on a comment line, case-insensitive. The declared state
# must be the FIRST token after `STATUS:` — a `STATUS:` line declares exactly one state,
# and all three real scaffolds read `# STATUS: scaffold — <what is missing>`. Anchoring it
# there is what keeps `# STATUS: EXECUTABLE ... this is not a scaffold` from matching; a
# loose `.*scaffold` would turn any prose that merely uses the word into a declaration.
_STATUS_SCAFFOLD_RE = re.compile(
r"^[ \t]*#[ \t]*STATUS:[ \t]*(?:scaffold|placeholder)\b", re.M | re.I)
# A quoted literal whose entire content is one marker word: the marker as data.
_MARKER_AS_DATUM_RE = re.compile(r"(['\"])[ \t]*(?:TODO|FIXME)[ \t]*\1", re.I)
def _split_doc_header(text: str) -> tuple[str, str]:
"""Split a test file into (documentation header, body).
Both file kinds in this tree open with a banner comment block that is PROSE about the
test: the Given/When/Then, the incident being closed, the mandatory "what this does NOT
prove" list (tests/README.md, L0 authoring rule 1). Prose necessarily discusses
placeholder idioms when the test hunts them, so an incidental mention there is
documentation, not a placeholder body.
The body starts at the first line that is neither blank nor a `#` comment:
*.sh -> `set -euo pipefail` (the shebang is a comment)
*.yaml -> `apiVersion: chainsaw.kyverno.io/v1alpha1`
Comments BELOW that point are body, deliberately: a scaffold marks a missing fixture
with an inline comment (`l3-fault-injection/alerting`: `file: fixtures/smtp-sink.yaml
# TODO(T-4.1): hermetic SMTP capture sink`), and that must keep counting.
"""
lines = text.splitlines(keepends=True)
for i, ln in enumerate(lines):
s = ln.strip()
if s and not s.startswith("#"):
return "".join(lines[:i]), "".join(lines[i:])
return text, "" # no executable line at all: it is all header, body is empty
def scaffold_marker(text: str) -> str:
"""Why `text` declares itself a placeholder body, or "" if it does not.
Returns a short human-readable reason (used verbatim in the hygiene report) so the
defect is actionable rather than a bare path.
"""
head, body = _split_doc_header(text)
m = _STATUS_SCAFFOLD_RE.search(head) or _STATUS_SCAFFOLD_RE.search(body)
if m:
return m.group(0).strip().lstrip("#").strip()[:80]
# Blank out markers held as pure data before looking for a marker that has a message.
m2 = _MARKER_RE.search(_MARKER_AS_DATUM_RE.sub('""', body))
if m2:
line = body[:m2.start()].count("\n")
snippet = body.splitlines()[line].strip()
return f"placeholder marker in body: {snippet[:80]}"
return ""
def classify_test(t: dict) -> dict:
"""Decide EXECUTABLE vs SCAFFOLD for one discovered test, conservatively.
@@ -281,14 +407,18 @@ def classify_test(t: dict) -> dict:
A tagged test is only worth counting if it can actually assert something. Rules, in
order, chosen so ambiguity NEVER upgrades a test:
scaffold the body contains TODO markers (a placeholder), *even if* it
is annotated executable — an annotation must never outrank the
body. That mismatch is also reported as a hygiene defect.
executable annotated `conformance.home.k3s/executable: "true"` and no TODOs.
executable-untagged no TODOs and a real runnable body, but the annotation is
missing (true of the L1 render tests, which execute a
render-check.sh + golden diff, of the L0 static checks, and
of the META self-check).
scaffold the test DECLARES itself a placeholder — a `# STATUS: scaffold`
header line, or a TODO/FIXME marker carrying a message in the
body (see `scaffold_marker` for the rule and the corpus it was
derived from) — *even if* it is annotated executable. An
annotation must never outrank the body. That mismatch is also
reported as a hygiene defect.
executable annotated `conformance.home.k3s/executable: "true"` and no
scaffold declaration.
executable-untagged no scaffold declaration and a real runnable body, but the
annotation is missing (true of the L1 render tests, which
execute a render-check.sh + golden diff, of the L0 static
checks, and of the META self-check).
Counted as executable, and flagged as an annotation gap.
scaffold anything else.
"""
@@ -299,16 +429,26 @@ def classify_test(t: dict) -> dict:
# downgrades the test to a scaffold — an annotation must never outrank the body,
# whichever file the body lives in.
extra = ""
scripts: list[tuple[str, str]] = []
for script in ("render-check.sh", "check.sh"):
sp = p.parent / script
if sp.is_file():
extra += sp.read_text()
body = sp.read_text()
extra += body
scripts.append((script, body))
declared = bool(_EXECUTABLE_RE.search(text))
has_todo = "TODO" in text or "TODO" in extra
# Each file is classified on ITS OWN structure — the yaml and the script have different
# header blocks, so they must not be concatenated before the header/body split.
reason = scaffold_marker(text)
for script, body in scripts:
if reason:
break
r = scaffold_marker(body)
reason = f"{script}: {r}" if r else ""
bm = _BLOCKED_RE.search(text)
has_body = bool(extra) or bool(re.search(r"^\s+- (script|assert|apply):", text, re.M))
if has_todo:
if reason:
state = "scaffold"
elif declared:
state = "executable"
@@ -325,7 +465,8 @@ def classify_test(t: dict) -> dict:
"name": t["name"], "path": rel, "layer": t["layer"],
"state": state,
"declared_executable": declared,
"annotation_conflict": bool(declared and has_todo),
"annotation_conflict": bool(declared and reason),
"scaffold_reason": reason,
"blocked_by": bm.group(1).strip().strip('"') if bm else "",
}
@@ -343,6 +484,169 @@ def spec_requirement_ids(spec_root: Path) -> set[str] | None:
return set(RID_RE.findall(doc.read_text()))
# ========================================================================== ledger
# PROOF axis. Everything here reads `conformance/runs/ledger.jsonl` (written ONLY by an
# explicit `lib/record_run.py` / `run.sh --record` invocation) and downgrades relentlessly.
# Nothing in this section may ever turn absent, ambiguous or aged evidence into green.
DEFAULT_LEDGER = CONF_ROOT / "runs" / "ledger.jsonl"
# Severity ranking, WORST FIRST. A requirement's proof is the MINIMUM over its verifying
# tests, so one failed/skipped/missing test drags the whole requirement down and a single
# proven test can never carry the others. `stale-proof` ranks below `proven` on purpose:
# it is a real past pass that no longer describes the current code, and it is NOT green.
PROOF_ORDER = {
"failed": 0, # a ledger row recorded a JUnit failure/error
"unknown": 1, # ran, but the log could not evidence pass vs self-skip
"not-derivable": 2, # no ledger row for this test at all (the pre-ledger default)
"junit-skipped": 3, # JUnit <skipped/> — did not execute
"skipped": 4, # SELF-SKIPPED green: exited 0 without exercising the behaviour
"stale-proof": 5, # proven at a sha the implementation/test has since moved past
"proven": 6, # the ONLY green state
}
# record_run.py's two "we cannot tell" states collapse to one reported state here; the
# per-test detail in status.json keeps the distinction for whoever has to fix it.
_PROOF_ALIASES = {
"unknown-no-sentinel-declared": "unknown",
"unknown-sentinel-absent": "unknown",
}
def read_ledger(path: Path) -> list[dict]:
"""Parse the append-only JSONL ledger. A malformed line is SKIPPED, never guessed at.
Absent or empty file -> [] , which reproduces this report's pre-ledger behaviour
exactly (every requirement `not-derivable`, headline PROVEN 0).
"""
if not path.is_file():
return []
rows: list[dict] = []
for line in path.read_text(errors="replace").splitlines():
line = line.strip()
if not line or line.startswith("#"):
continue
try:
obj = json.loads(line)
except json.JSONDecodeError:
continue
if isinstance(obj, dict) and obj.get("test"):
rows.append(obj)
return rows
def latest_by_test(rows: list[dict]) -> dict[str, dict]:
"""Most recent ledger row per test name (ts_utc, then file order as the tiebreak).
Latest wins rather than best-ever: a test that has since gone back to self-skipping
must stop reading as proven.
"""
latest: dict[str, dict] = {}
for i, r in enumerate(rows):
key = r["test"]
prev = latest.get(key)
if prev is None or (r.get("ts_utc", ""), i) >= (prev.get("ts_utc", ""),
prev["_i"]):
latest[key] = {**r, "_i": i}
return latest
class _GitOracle:
"""Small cached wrapper for the two git questions the staleness rule asks."""
def __init__(self, repo: Path):
self.repo = repo
self._last: dict[tuple[str, ...], str | None] = {}
self._anc: dict[tuple[str, str], bool] = {}
def _git(self, *args: str) -> subprocess.CompletedProcess:
return subprocess.run(["git", "-C", str(self.repo), *args],
capture_output=True, text=True)
def last_change(self, paths: tuple[str, ...]) -> str | None:
"""Commit that last touched any of `paths`; None if git knows none of them."""
if not paths:
return None
if paths in self._last:
return self._last[paths]
p = self._git("log", "-1", "--format=%H", "--", *paths)
sha = p.stdout.strip() if p.returncode == 0 else ""
self._last[paths] = sha or None
return self._last[paths]
def is_ancestor(self, older: str, newer: str) -> bool:
"""True iff `older` is an ancestor of (or equal to) `newer`.
False whenever either rev is unresolvable — an unknown or `-dirty` sha (a run
recorded from a modified working tree) can never satisfy the freshness test, which
is the intended fail-closed direction.
"""
key = (older, newer)
if key in self._anc:
return self._anc[key]
ok = self._git("merge-base", "--is-ancestor", older, newer).returncode == 0
self._anc[key] = ok
return ok
def derive_proof(git: _GitOracle, rid: str, verifiers: list[dict],
impl_files: list[dict], latest: dict[str, dict]) -> dict:
"""PROOF for one requirement, from the ledger, with the staleness rule applied.
STALENESS RULE — a proof is a claim about a specific tree, not a permanent badge:
relevant paths = every implementing artifact this report found for the requirement
+ every verifying test's directory
last_change = `git log -1 -- <relevant paths>`
A row's `proven` is honoured only if `last_change` is an ANCESTOR of the row's
`git_sha`, i.e. the run happened at or after the newest change to the code it
claims to prove. Otherwise it degrades to `stale-proof` (not green). A `-dirty`
sha, a sha this clone does not have (rebased/dropped), or a path git has never
seen (an uncommitted test) all fail the ancestry test and degrade the same way.
The requirement's state is the MINIMUM over its verifying tests: every verifying test
must be proven and fresh before the requirement reads `proven`. A test with no ledger
row contributes `not-derivable`, so a partially-recorded run cannot manufacture green.
"""
paths = sorted({f["path"] for f in impl_files}
| {os.path.dirname(v["path"]) for v in verifiers if v.get("path")})
last_change = git.last_change(tuple(paths))
per_test: list[dict] = []
for v in verifiers:
row = latest.get(v["name"])
if row is None:
per_test.append({"test": v["name"], "state": "not-derivable",
"reason": "no ledger row — this test has never been "
"recorded by run.sh --record / record_run.py"})
continue
base = _PROOF_ALIASES.get(row.get("proof", ""), row.get("proof", "unknown"))
if base not in PROOF_ORDER:
base = "unknown"
state, reason = base, row.get("note", "")
if base == "proven":
sha = row.get("git_sha", "")
if last_change and not git.is_ancestor(last_change, sha):
state = "stale-proof"
reason = (f"proven at {sha[:12]}, but the implementing manifests / test "
f"changed in {last_change[:12]}, which that run predates")
per_test.append({"test": v["name"], "state": state, "reason": reason,
"run_id": row.get("run_id"), "ts_utc": row.get("ts_utc"),
"git_sha": row.get("git_sha"), "env": row.get("env"),
"sentinel": row.get("sentinel"),
"junit_result": row.get("junit_result")})
if not per_test:
state = "not-derivable"
reason = "no requirement-tagged test exists to be proven"
else:
worst = min(per_test, key=lambda d: PROOF_ORDER[d["state"]])
state, reason = worst["state"], worst["reason"]
if state == "not-derivable" and not reason:
reason = "no committed ledger row for this requirement's verifying test(s)"
return {"state": state, "reason": reason, "last_change_sha": last_change,
"evidence_paths": paths, "per_test": per_test}
# ========================================================================== build
IMPL_ORDER = {"none": 0, "off-cluster": 1, "unwired": 2, "reconciled-partial": 3,
"reconciled": 4}
@@ -369,7 +673,13 @@ LABELS = {
def build(repo: Path, registry: list[dict], tests: list[dict], impl: dict[str, list[dict]],
applied: set[str], gitops_roots: set[str], spec_ids: set[str] | None,
env: str) -> dict:
env: str, ledger_rows: list[dict] | None = None) -> dict:
# PROOF axis inputs. `ledger_rows=None` / `[]` reproduces the pre-ledger behaviour:
# `latest` is empty, so every requirement derives `not-derivable` exactly as before.
ledger_rows = ledger_rows or []
latest = latest_by_test(ledger_rows)
git = _GitOracle(repo)
classified = [classify_test(t) for t in tests]
by_req_tests: dict[str, list[dict]] = {r["id"]: [] for r in registry}
for t, c in zip(tests, classified):
@@ -439,17 +749,19 @@ def build(repo: Path, registry: list[dict], tests: list[dict], impl: dict[str, l
"blocked_by": sorted({t["blocked_by"] for t in ts if t["blocked_by"]}),
"tests": ts,
},
# NEVER derived. See the module docstring.
"proof": {"state": "not-derivable",
"reason": "no committed, machine-readable record of a test run "
"exists in the repo (_artifacts/ and results/ are "
"git-ignored); DECISIONS-LOG.md prose is not a signal"},
# Derived ONLY from the committed ledger, and only as far as it reaches.
# With no ledger row this stays `not-derivable`, as it always was.
"proof": derive_proof(git, rid, ts, ev, latest),
"status": LABELS[(impl_state, verify_state)],
}
# --- hygiene defects this tool owns -------------------------------------------
hygiene = {
"annotation_conflicts": [t["path"] for t in classified if t["annotation_conflict"]],
# Same set, with WHY each one was called a placeholder body, so the fix is obvious
# from the report instead of requiring the reader to re-derive the rule.
"annotation_conflict_reasons": {t["path"]: t["scaffold_reason"]
for t in classified if t["annotation_conflict"]},
"executable_untagged": [t["path"] for t in classified
if t["state"] == "executable-untagged"],
"registry_not_in_spec": [],
@@ -492,8 +804,19 @@ def build(repo: Path, registry: list[dict], tests: list[dict], impl: dict[str, l
and v["verification"]["state"] == "executable"),
"safety_critical_total": count(lambda v: v["safety_critical"]),
"safety_critical_without_executable_l3": len(safety_scaffold),
"proven": 0,
"proven_derivable": False,
# PROVEN is now a real count — but only of requirements whose EVERY verifying
# test has a committed ledger row that saw its pass sentinel, at a sha the
# implementation has not moved past. `stale-proof` and `skipped` are counted
# separately and are NOT green.
"proven": count(lambda v: v["proof"]["state"] == "proven"),
"proof_stale": count(lambda v: v["proof"]["state"] == "stale-proof"),
"proof_skipped": count(lambda v: v["proof"]["state"] == "skipped"),
"proof_failed": count(lambda v: v["proof"]["state"] == "failed"),
"proof_unknown": count(lambda v: v["proof"]["state"] == "unknown"),
"proof_not_derivable": count(
lambda v: v["proof"]["state"] in ("not-derivable", "junit-skipped")),
"ledger_rows": len(ledger_rows),
"proven_derivable": bool(ledger_rows),
"coverage_style_covered": count(lambda v: v["verification"]["state"] != "none"),
}
summary["uncited_by_any_reconciled_manifest"] = count(
@@ -552,16 +875,39 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
o.append(f"| Safety-critical requirements whose L3 test is a scaffold | "
f"**{s['safety_critical_without_executable_l3']}** of "
f"{s['safety_critical_total']} |")
o.append(f"| **PROVEN (a test actually passed)** | **{s['proven']} — NOT DERIVABLE** |")
o.append("")
o.append("> **Proof is deliberately not derived.** Nothing tracked in git records a "
"test result: `conformance/_artifacts/` and `results/` are git-ignored, and "
"the only record of a green run is DECISIONS-LOG.md prose — hand-maintained, "
"which is the very thing this report replaces. Per "
"`agents/definition-of-done.md` a requirement is done only when its tagged "
"test **passes**, so the strongest claim this report can make is "
"*wired and testable*. Closing the gap needs a committed, machine-readable "
"run ledger; until then, treat every row below as unproven.\n")
if s["proven_derivable"]:
o.append(f"| **PROVEN (every verifying test passed, evidenced)** | "
f"**{s['proven']}** |")
o.append(f"| Proof exists but is STALE (code changed after the run) | "
f"{s['proof_stale']} |")
o.append(f"| Verifying test SELF-SKIPPED green (not proof) | {s['proof_skipped']} |")
o.append(f"| Verifying test FAILED in the recorded run | {s['proof_failed']} |")
o.append(f"| Ran, but the log could not evidence pass vs self-skip | "
f"{s['proof_unknown']} |")
o.append(f"| No committed ledger row yet | {s['proof_not_derivable']} |")
o.append("")
o.append(f"> **Proof comes from `conformance/runs/ledger.jsonl`** "
f"({s['ledger_rows']} row(s)), written only by an explicit "
f"`run.sh --record` / `lib/record_run.py` invocation. A row reaches "
f"`proven` ONLY when the run log carried that test's own pass sentinel "
f"token — the L2/L3 tests self-skip GREEN (Chainsaw records `passed` on "
f"`DR_RESTORE_SKIPPED_OK` exactly as on a real run, and its JUnit carries "
f"no stdout), so a skip token maps to `skipped` and can never become "
f"proof. A proof is honoured only while the implementing manifests and "
f"the test itself have not changed since the recorded `git_sha`; "
f"otherwise it degrades to `stale-proof`, which is not green.\n")
else:
o.append(f"| **PROVEN (a test actually passed)** | **{s['proven']} — NOT DERIVABLE** |")
o.append("")
o.append("> **No proof is derivable: `conformance/runs/ledger.jsonl` is absent or "
"empty.** The ledger is the committed, machine-readable run record; until "
"a run is captured with `run.sh --record`, nothing tracked in git records "
"a test result (`conformance/_artifacts/` and `results/` are git-ignored, "
"and DECISIONS-LOG.md prose is hand-maintained — the very thing this "
"report replaces). Per `agents/definition-of-done.md` a requirement is "
"done only when its tagged test **passes**, so the strongest claim this "
"report can make is *wired and testable*; treat every row below as "
"unproven.\n")
o.append(f"> For contrast, `coverage.md` counts **{s['coverage_style_covered']}** "
f"requirements as covered — that count includes the "
f"{s['verify_scaffold_only']} whose only verifying test is a TODO scaffold.\n")
@@ -612,7 +958,15 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
if ver["blocked_by"]:
vcell += f" · blocked-by: {'; '.join(ver['blocked_by'])}"
sc = " ⚠safety" if v["safety_critical"] else ""
o.append(f"| {rid}{sc} | **{v['status']}** | {icell} | {vcell} | not-derivable |")
pr = v["proof"]
if pr["state"] == "not-derivable":
pcell = "not-derivable"
else:
runs = sorted({d["run_id"] for d in pr["per_test"] if d.get("run_id")})
pcell = f"**{pr['state']}**" if pr["state"] == "proven" else pr["state"]
if runs:
pcell += f" ({', '.join(runs)})"
o.append(f"| {rid}{sc} | **{v['status']}** | {icell} | {vcell} | {pcell} |")
if rep["safety_critical_without_executable_l3"]:
o.append("\n## Safety-critical requirements whose L3 fault test is a SCAFFOLD "
@@ -678,8 +1032,10 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
else:
o.append("- registry ↔ spec parity: **NOT CHECKED** (spec tree not available; "
"this is a skip, not a pass)")
o.append(f"- tests annotated executable but containing TODO bodies: "
f"{h['annotation_conflicts'] or 'none'}")
o.append("- tests annotated executable but declaring a placeholder body: "
+ (", ".join(f"`{k}` ({v})"
for k, v in sorted(h["annotation_conflict_reasons"].items()))
if h["annotation_conflicts"] else "none"))
o.append(f"- executable tests missing the `{LABEL_PREFIX}/executable` annotation: "
f"{h['executable_untagged'] or 'none'}")
@@ -698,25 +1054,82 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
"reconciled |")
o.append("| verifying test | `conformance.home.k3s/R-*` label on a Chainsaw test "
"(the authoritative tag, same source as `coverage.md`) | exact |")
o.append("| executable vs scaffold | TODO markers in the test body outrank the "
"`executable` annotation; a real body without the annotation still counts "
"and is flagged | conservative — ambiguity never upgrades |")
o.append("| **test actually passed** | **NOT DERIVED** — nothing in git records a "
"run result | refused |")
o.append("| executable vs scaffold | a `# STATUS: scaffold` header line, or a "
"TODO/FIXME marker CARRYING A MESSAGE in the body (a marker held as a bare "
"quoted datum — a placeholder detector's own vocabulary — is not one); "
"either outranks the `executable` annotation, and a real body without the "
"annotation still counts and is flagged | conservative — ambiguity never "
"upgrades |")
o.append("| **test actually passed** | the committed `runs/ledger.jsonl` row for "
"every verifying test: JUnit result **plus** that test's own pass sentinel "
"token grepped from the run log | exact where a run was recorded, refused "
"everywhere else — a self-skip token or a missing/unrecognised sentinel "
"NEVER yields `proven` |")
o.append("| proof still current | the row's `git_sha` must be a descendant of the "
"last commit touching the requirement's implementing manifests or its test "
"| conservative — an unresolvable or `-dirty` sha degrades to `stale-proof` |")
o.append("| cluster liveness (pods/CRs healthy) | **NOT DERIVED** — no cluster is "
"contacted; `reconciled` means Flux *would* apply it, not that it is "
"healthy | refused |")
o.append("| phase/board status | **NOT DERIVED** — `agents/task-board.md` is stale "
"and is not a signal | refused |")
o.append("\n## Run ledger — the committed proof-of-pass evidence\n")
if not s["proven_derivable"]:
o.append("- `conformance/runs/ledger.jsonl` holds no rows. Capture a run with "
"`./run.sh --record` (recording is never implicit: a run that silently "
"records itself is how a bad run becomes a permanent false proof), then "
"commit `conformance/runs/<run-id>/` **with its `run.log`** and the "
"ledger. The log is the load-bearing half — Chainsaw's JUnit carries no "
"stdout, so without it a self-skip is indistinguishable from a pass.")
else:
o.append(f"- {s['ledger_rows']} row(s) in `conformance/runs/ledger.jsonl`. "
f"Per-test evidence for every non-`not-derivable` requirement below; "
f"full rows (env, chainsaw/kube versions, sentinel, note) are in "
f"`status.json` under `by_requirement.<R-ID>.proof.per_test`.\n")
o.append("| Requirement | Proof | Test | Ledger state | Sentinel | Run |")
o.append("|---|---|---|---|---|---|")
for rid, v in rep["by_requirement"].items():
pr = v["proof"]
if pr["state"] == "not-derivable":
continue
for d in pr["per_test"]:
o.append(f"| {rid} | {pr['state']} | `{d['test']}` | {d['state']} | "
f"`{d.get('sentinel') or '—'}` | {d.get('run_id') or '—'} |")
stale = [(rid, v) for rid, v in rep["by_requirement"].items()
if v["proof"]["state"] == "stale-proof"]
if stale:
o.append("\n### Stale proofs — a real pass that no longer describes this tree\n")
o.append("These requirements DID pass, but their implementing manifests or "
"their verifying test changed in a commit the recorded run predates. "
"A stale proof is not green; re-run with `--record`.\n")
for rid, v in stale:
pr = v["proof"]
o.append(f"- **{rid}** — {pr['reason']}")
skipped = [(rid, v) for rid, v in rep["by_requirement"].items()
if v["proof"]["state"] == "skipped"]
if skipped:
o.append("\n### SELF-SKIPPED green — recorded, and deliberately NOT proof\n")
o.append("Chainsaw reported `passed` for these, because the test exited 0 "
"after emitting its skip token: a precondition (Longhorn "
"StorageClass, Velero CRDs, >=3 Ready nodes) was absent, so the "
"behaviour was never exercised. This is exactly the row a "
"JUnit-only ledger would have recorded as green.\n")
for rid, v in skipped:
toks = ", ".join(sorted({d.get("sentinel") or "?"
for d in v["proof"]["per_test"]
if d["state"] == "skipped"}))
o.append(f"- **{rid}** — skip token(s): `{toks}`")
o.append("\n## Advisory — untracked local run artifact (NOT counted)\n")
if advisory:
o.append(f"- `{advisory['path']}` (untracked, git-ignored), last written "
f"{advisory['mtime']}: {advisory['tests']} testcase(s), "
f"{advisory['failures']} failure(s), {advisory['skipped']} skipped.")
o.append(f"- requirement IDs it names: {', '.join(advisory['requirements']) or '—'}")
o.append("- This is one developer's local workspace, not repo state. It is shown "
"for orientation only and contributes to NO status above.")
o.append("- This is one developer's local workspace, not repo state, and it is NOT "
"the ledger. It is shown for orientation only and contributes to NO "
"status above — a JUnit alone cannot tell a self-skip from a pass.")
else:
o.append("- none present.")
return "\n".join(o) + "\n"
@@ -841,6 +1254,186 @@ def _selftest() -> int:
classify_test({"name": "s", "path": str(f), "layer": "L2"})["state"]
== "scaffold")
# === THE BUG THIS DETECTOR WAS REWRITTEN FOR ==============================
# A whole-file `"TODO" in text` mis-classified `l0-static/placeholder-guard-
# coupling` — a 32-assertion executable check whose SUBJECT is placeholder
# detection, so it must name the idioms it hunts — as a scaffold, and through it
# reported the SAFETY-CRITICAL R-STATE-03 as `exec=1 scaffold=1`. These
# assertions pin BOTH directions: a marker held as DATA must not downgrade, and a
# real placeholder body must still be caught. Both halves are load-bearing; the
# second is what stops this fix from degrading into "always executable".
yml = ("metadata:\n annotations:\n"
" conformance.home.k3s/executable: \"true\"\n"
"spec:\n steps:\n - script:\n"
" content: bash ./check.sh\n")
f.write_text(yml)
detector = td2 / "check.sh"
# Shaped exactly like the real check: a prose header that discusses the idioms,
# then executable code holding them as a vocabulary list.
detector.write_text(
"#!/usr/bin/env bash\n"
"# L0 static check — placeholder vocabulary\n"
"# * SEED — loud-placeholder idioms (TBD, TODO, FIXME, PLACEHOLDER)\n"
"# STATUS: EXECUTABLE. A pure offline assertion script.\n"
"set -euo pipefail\n"
"python3 - <<'EOF'\n"
"SEED = [\"TBD\", \"TODO\", \"FIXME\", \"PLACEHOLDER\"]\n"
"assert any(w in open('/etc/hostname').read().upper() for w in SEED) is False\n"
"EOF\n")
c = classify_test({"name": "s", "path": str(f), "layer": "L0"})
check("a marker in HEADER PROSE does not downgrade an executable check",
c["state"] == "executable" and not c["annotation_conflict"])
check("a marker held as a bare quoted DATUM does not downgrade (the R-STATE-03 bug)",
c["scaffold_reason"] == "")
# ... and the protection is intact: a real placeholder body is still caught, in
# each of the three shapes the corpus actually uses.
detector.write_text("#!/usr/bin/env bash\nset -euo pipefail\n"
"echo \"TODO: assert the PVC is bound\"\n")
c = classify_test({"name": "s", "path": str(f), "layer": "L0"})
check("a TODO carrying a message in check.sh IS still a scaffold",
c["state"] == "scaffold" and c["annotation_conflict"])
check("the scaffold reason names the file and the offending line",
c["scaffold_reason"].startswith("check.sh:")
and "TODO: assert the PVC is bound" in c["scaffold_reason"])
detector.write_text("#!/usr/bin/env bash\nset -euo pipefail\n"
"# TODO(T-4.1) write the hermetic sink fixture\ntrue\n")
check("a TODO(T-x.y) roadmap marker in an inline body comment IS still a scaffold",
classify_test({"name": "s", "path": str(f), "layer": "L0"})["state"]
== "scaffold")
detector.write_text("#!/usr/bin/env bash\nset -euo pipefail\n"
"echo \"FIXME the extraction before trusting this\"\n")
check("an unpunctuated FIXME carrying trailing prose IS still a scaffold",
classify_test({"name": "s", "path": str(f), "layer": "L0"})["state"]
== "scaffold")
# The dedicated declaration slot outranks a body that would otherwise look real —
# this is how all three genuine scaffolds in the tree declare themselves.
detector.unlink()
f.write_text("# L3 FAULT-INJECTION SCAFFOLD — x\n"
"# STATUS: scaffold — needs T-1.4 (Flux bootstrapped).\n" + yml)
c = classify_test({"name": "s", "path": str(f), "layer": "L3"})
check("a `# STATUS: scaffold` header declaration IS a scaffold, body notwithstanding",
c["state"] == "scaffold" and c["annotation_conflict"])
check("`# STATUS: EXECUTABLE` is not read as a scaffold declaration",
scaffold_marker("# STATUS: EXECUTABLE. `check.sh` is a pure offline "
"assertion script; covers no scaffold.\nset -e\n") == "")
# ---------------------------------------------------------------- PROOF / ledger
# These are the assertions that stand between this report and a fake green.
check("empty ledger parses to no rows (pre-ledger behaviour is preserved)",
read_ledger(Path("/nonexistent/ledger.jsonl")) == [])
def row(**kw):
base = {"ts_utc": "2026-09-01T12:00:00Z", "git_sha": "deadbeef", "env": "dev",
"chainsaw_version": "v0.2.12", "kube_version": "v1.36.4", "layer": "L3",
"test": "dr-restore", "requirements": ["R-STATE-03"],
"junit_result": "passed", "sentinel": None, "proof": "skipped",
"run_id": "20260901T120000Z-deadbee", "note": ""}
base.update(kw)
return base
verifiers = [{"name": "dr-restore",
"path": "conformance/tests/l3-fault-injection/dr-restore/"
"chainsaw-test.yaml"}]
with tempfile.TemporaryDirectory() as td:
gr = Path(td)
env = {**os.environ, "GIT_AUTHOR_NAME": "t", "GIT_AUTHOR_EMAIL": "t@t",
"GIT_COMMITTER_NAME": "t", "GIT_COMMITTER_EMAIL": "t@t"}
def git(*a):
return subprocess.run(["git", "-C", str(gr), *a], capture_output=True,
text=True, env=env)
git("init", "-q", "-b", "main")
(gr / "platform").mkdir()
(gr / "platform" / "velero.yaml").write_text("kind: X # R-STATE-03\n")
git("add", "-A"); git("commit", "-qm", "one")
sha_a = git("rev-parse", "HEAD").stdout.strip()
(gr / "platform" / "velero.yaml").write_text("kind: X # R-STATE-03 v2\n")
git("add", "-A"); git("commit", "-qm", "two")
sha_b = git("rev-parse", "HEAD").stdout.strip()
oracle = _GitOracle(gr)
impl = [{"path": "platform/velero.yaml"}]
# *** THE CASE THIS WHOLE FEATURE EXISTS FOR ***
pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
latest_by_test([row(proof="skipped", junit_result="passed",
sentinel="DR_RESTORE_SKIPPED_OK",
git_sha=sha_b)]))
check("a SELF-SKIP row on a GREEN JUnit is 'skipped', NEVER 'proven'",
pr["state"] == "skipped")
pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
latest_by_test([row(proof="proven", git_sha=sha_b,
sentinel="DR_RESTORED_OK")]))
check("a proven row at the CURRENT sha is honoured", pr["state"] == "proven")
pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
latest_by_test([row(proof="proven", git_sha=sha_a,
sentinel="DR_RESTORED_OK")]))
check("STALENESS: proven at a sha the implementation has moved past degrades "
"to stale-proof", pr["state"] == "stale-proof")
pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
latest_by_test([row(proof="proven", git_sha=f"{sha_b}-dirty",
sentinel="DR_RESTORED_OK")]))
check("a run recorded from a DIRTY tree is never honoured as proof",
pr["state"] == "stale-proof")
pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
latest_by_test([row(proof="proven", git_sha="0" * 40)]))
check("a git_sha this clone cannot resolve degrades, never upgrades",
pr["state"] == "stale-proof")
for bad, expect in (("failed", "failed"),
("junit-skipped", "junit-skipped"),
("unknown-sentinel-absent", "unknown"),
("unknown-no-sentinel-declared", "unknown"),
("something-new-from-the-future", "unknown")):
pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
latest_by_test([row(proof=bad, git_sha=sha_b)]))
check(f"ledger state {bad!r} maps to {expect!r}, never to proven",
pr["state"] == expect)
pr = derive_proof(oracle, "R-STATE-03", verifiers, impl, {})
check("no ledger row at all stays not-derivable",
pr["state"] == "not-derivable")
# Two verifying tests, only one proven: the requirement must NOT read proven.
two = verifiers + [{"name": "data-survival",
"path": "conformance/tests/l3-fault-injection/"
"data-survival/chainsaw-test.yaml"}]
pr = derive_proof(oracle, "R-STATE-03", two, impl,
latest_by_test([row(proof="proven", git_sha=sha_b)]))
check("a requirement is proven only when EVERY verifying test is",
pr["state"] == "not-derivable")
pr = derive_proof(oracle, "R-STATE-03", two, impl,
latest_by_test([row(proof="proven", git_sha=sha_b),
row(test="data-survival", proof="skipped",
git_sha=sha_b)]))
check("worst-of wins across verifying tests (skip drags proven down)",
pr["state"] == "skipped")
# Latest row wins, not best-ever: a test that regressed to self-skipping must
# stop reading as proven.
pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
latest_by_test([row(ts_utc="2026-09-01T10:00:00Z",
proof="proven", git_sha=sha_b),
row(ts_utc="2026-09-02T10:00:00Z",
proof="skipped", git_sha=sha_b)]))
check("the LATEST ledger row wins, not the best one ever recorded",
pr["state"] == "skipped")
# A malformed ledger line is skipped, never guessed at.
lf = gr / "ledger.jsonl"
lf.write_text('{"test":"a","proof":"proven"}\nnot json at all\n\n'
'{"proof":"proven"}\n')
rows = read_ledger(lf)
check("malformed / test-less ledger lines are dropped, not guessed at",
len(rows) == 1 and rows[0]["test"] == "a")
if not ok:
return 1
sys.stderr.write("status.py selftest: PASS\n")
@@ -865,6 +1458,10 @@ def main() -> int:
help="exit 1 on defects this tool owns: registry/spec drift, "
"executable-annotation conflicts, or a safety-critical "
"requirement whose only L3 test is a scaffold")
ap.add_argument("--ledger", type=Path, default=DEFAULT_LEDGER,
help=f"committed proof-of-pass ledger written by lib/record_run.py "
f"(default {DEFAULT_LEDGER}). Absent or empty => every "
f"requirement stays `not-derivable`, as before the ledger existed")
ap.add_argument("--selftest", action="store_true", help="run the built-in self-test")
args = ap.parse_args()
@@ -895,7 +1492,12 @@ def main() -> int:
applied, nodes = reconciled_files(repo, entry)
gitops_roots = {n["resolved"].split("/", 1)[0] for n in nodes if n["exists"]}
impl = scan_implementation(repo, tracked_files(repo))
rep = build(repo, registry, tests, impl, applied, gitops_roots, spec_ids, args.env)
ledger_rows = read_ledger(args.ledger)
rep = build(repo, registry, tests, impl, applied, gitops_roots, spec_ids, args.env,
ledger_rows)
rep["ledger"] = {"path": os.path.relpath(args.ledger, REPO_ROOT),
"rows": len(ledger_rows),
"present": args.ledger.is_file()}
rep["flux_graph"] = nodes
rep["gitops_roots"] = sorted(gitops_roots)
rep["reconciled_file_count"] = len(applied)
@@ -915,7 +1517,11 @@ def main() -> int:
f"(+{s['wired_or_partial_and_testable'] - s['wired_and_testable']} partial) | "
f"scaffold-only {s['verify_scaffold_only']} | "
f"safety-critical w/o executable L3 "
f"{s['safety_critical_without_executable_l3']} | PROVEN: not derivable")
f"{s['safety_critical_without_executable_l3']} | "
+ (f"PROVEN: {s['proven']} (stale {s['proof_stale']}, self-skipped "
f"{s['proof_skipped']}, failed {s['proof_failed']}, unknown "
f"{s['proof_unknown']}) from {s['ledger_rows']} ledger row(s)"
if s["proven_derivable"] else "PROVEN: not derivable (empty run ledger)"))
print(f"[status] wrote {args.out/'status.md'} and {args.out/'status.json'}")
rc = 0
@@ -926,8 +1532,11 @@ def main() -> int:
f"{h['registry_not_in_spec']} {h['spec_not_in_registry']}\n")
rc = 1
if h["annotation_conflicts"]:
sys.stderr.write(f"[status] STRICT: tests annotated executable with TODO "
f"bodies: {h['annotation_conflicts']}\n")
sys.stderr.write(
"[status] STRICT: tests annotated executable that declare a placeholder "
"body: " + "; ".join(f"{k} ({v})" for k, v in
sorted(h["annotation_conflict_reasons"].items()))
+ "\n")
rc = 1
if rep["safety_critical_without_executable_l3"]:
sys.stderr.write(f"[status] STRICT: safety-critical requirement(s) whose L3 "
+175 -13
View File
@@ -10,16 +10,48 @@
# ./run.sh --requirements R-NET-01,R-NET-06 # merge-gate subset (tagged)
# ./run.sh --layer l2 # one layer (l0|l1|l2|l3|meta|all; l0/l1 need no cluster)
# ./run.sh --requirements R-NET-01,R-NET-02 --layer l2 # intersect: tagged AND in layer
# ./run.sh --layer l0 --requirements R-X --include-scope whole-repo
# # UNION: the tagged subset PLUS every test that
# # DECLARES `conformance.home.k3s/scope: whole-repo`.
# # A whole-repo invariant must run on EVERY gate
# # invocation — see lib/common.sh's scope block for
# # why, and why a SKIP of one is exit 3, not green.
# ./run.sh --coverage-only # regenerate coverage report only (no cluster)
# ./run.sh --status # DERIVED STATUS report only (no cluster): per
# # requirement, is it implemented, is it actually
# # reconciled by Flux, is its test executable or a
# # scaffold. Proof-of-pass is deliberately NOT
# # derived — see lib/status.py.
# # scaffold, and whether the committed run ledger
# # (runs/ledger.jsonl) actually proves it — see
# # lib/status.py. No ledger row => not-derivable.
# ./run.sh --lint # structural-validate test defs (no cluster)
# ./run.sh --print-selection # print the test dirs the CURRENT selection resolves
# # to, one per line, and exit. Runs no check. This is
# # how a test can PROVE the selection includes it
# # (tests/l0-static/whole-repo-invariants-wired) without
# # executing the suite and recursing into itself.
# ./run.sh --strict-coverage # hygiene gate: fail on tag/annotation drift
# ./run.sh --requirements R-NET-01 # ALSO asserts those IDs are covered (merge gate)
# ./run.sh --require-all # phase-exit: every requirement must be covered
# ./run.sh --record # RUN AND CAPTURE: tee the whole run to
# # runs/<UTC>-<short-sha>/run.log, copy the JUnit
# # beside it, and append one ledger row per
# # testcase to runs/ledger.jsonl (lib/record_run.py)
#
# --record — PROOF-OF-PASS, CAPTURED AT RUN TIME (why it is a separate flag)
# --------------------------------------------------------------------------
# Without it, nothing about a green run ever reaches repo state: _artifacts/ is gitignored
# and the root .gitignore kills *.junit.xml globally, which is why `--status` reports
# PROVEN 0 for all 51 requirements. `--record` writes the one tracked thing that fixes
# that — `conformance/runs/` (see conformance/.gitignore for the negations).
#
# RECORDING IS DELIBERATELY EXPLICIT. A run that silently records itself is how a bad run
# becomes a permanent false proof, so the plain `./run.sh` writes no ledger row at all.
#
# BOTH FILES ARE KEPT, and the LOG is the load-bearing one. The L2/L3 tests self-skip
# GREEN: when a precondition is missing they echo e.g. `DR_RESTORE_SKIPPED_OK` and exit 0,
# and Chainsaw records `passed` exactly as on a real run. Chainsaw's JUnit carries no
# stdout, so the sentinel token in run.log is the ONLY thing that separates proven from
# skipped. A JUnit committed on its own would record a skip as proof of R-STATE-03.
#
# Artifacts (always written under conformance/_artifacts/, gitignored):
# _artifacts/report.junit.xml JUnit (Chainsaw output, or skipped-scaffold shim)
@@ -28,6 +60,11 @@
# _artifacts/status/status.md human requirement-status report (--status)
# _artifacts/status/status.json machine requirement-status report (--status)
#
# Run ledger (TRACKED — commit these; --record only):
# runs/<UTC>-<short-sha>/run.log full tee'd run output (carries the sentinels)
# runs/<UTC>-<short-sha>/report.junit.xml the JUnit for that same run
# runs/ledger.jsonl append-only, one row per testcase per run
#
# Phase 0: no cluster exists. The behavior tests are valid scaffolds that cannot
# execute yet, so run.sh emits a *skipped* JUnit and a coverage report, and exits
# green — the harness is proven reproducible while later phases fill in behavior.
@@ -47,12 +84,18 @@ STATUS_OUT="${ARTIFACTS}/status"
JUNIT_OUT="${ARTIFACTS}/report.junit.xml"
PYTHON="${PYTHON:-python3}"
# Every argument as given, so `--record` can re-exec this script unchanged (minus itself)
# under `tee`. Captured BEFORE parsing so nothing is normalized away.
ORIG_ARGS=("$@")
LAYER="all"
REQS=""
INCLUDE_SCOPES="" # --include-scope: scopes unioned into the selection (see lib/common.sh)
MODE="run" # run | coverage-only | status | lint
STRICT_COVERAGE=0
REQUIRE_ALL=0
FORCE_NO_CLUSTER=0
RECORD=0
# Documented, out-of-scope coverage gaps exempt from the completeness gate: requirements
# owned by another workstream that WS-TEST does not yet scaffold. Listed here so the
@@ -83,18 +126,23 @@ FORCE_NO_CLUSTER=0
# longer a gap and has been removed from this list; the coverage report counts it as covered.
DOCUMENTED_GAPS=""
usage() { sed -n '2,40p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; }
# Print the header block (line 2 through the closing `####` rule, which is dropped) as
# help. Bounded by the rule rather than a fixed line number so the header can grow.
usage() { sed -n '2,/^#####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; }
while [ $# -gt 0 ]; do
case "$1" in
--layer) LAYER="$2"; shift 2 ;;
--requirements) REQS="$(conf_normalize_reqs "$2")"; shift 2 ;;
--include-scope) INCLUDE_SCOPES="${INCLUDE_SCOPES:+${INCLUDE_SCOPES},}$2"; shift 2 ;;
--coverage-only) MODE="coverage-only"; shift ;;
--status) MODE="status"; shift ;;
--lint) MODE="lint"; shift ;;
--print-selection) MODE="print-selection"; shift ;;
--strict-coverage) STRICT_COVERAGE=1; shift ;;
--require-all) REQUIRE_ALL=1; shift ;;
--no-cluster) FORCE_NO_CLUSTER=1; shift ;;
--record) RECORD=1; shift ;;
-h|--help) usage; exit 0 ;;
*) echo "unknown arg: $1" >&2; usage; exit 2 ;;
esac
@@ -104,6 +152,62 @@ mkdir -p "${ARTIFACTS}" "${COVERAGE_OUT}" "${STATUS_OUT}"
log() { printf '\033[1m[run.sh]\033[0m %s\n' "$*"; }
###############################################################################
# --record: capture this run as committed evidence, then hand it to the ledger.
#
# Implemented as a ONE-LEVEL re-exec of this same script under `tee`, so the run itself is
# byte-for-byte the ordinary run (same code path, same artifacts) and the log is the exact
# operator-visible output — no second, differently-behaving "recording mode" to drift.
# CONF_RECORD_CHILD guards against recursion.
#
# The recorder runs REGARDLESS of the run's exit status: a red run must be recorded as red
# (that is evidence too), and run.sh still exits with the child's real rc, so no gate is
# softened by recording. `${PIPESTATUS[0]}` is what carries that rc past `tee`.
###############################################################################
if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then
case "${MODE}" in
run) ;;
# --record is about EVIDENCE OF A RUN. --coverage-only/--status/--lint execute no
# test, so a ledger row from one of them would be a row about nothing.
*) echo "--record captures a TEST RUN; --${MODE} executes none." >&2; exit 2 ;;
esac
RUN_SHA="$(git -C "${CONF_ROOT}" rev-parse --short HEAD 2>/dev/null || echo nogit)"
RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)-${RUN_SHA}"
RUN_DIR="${CONF_ROOT}/runs/${RUN_ID}"
mkdir -p "${RUN_DIR}"
# Drop any JUnit left behind by a PREVIOUS run before starting, so this run dir can only
# ever capture a report THIS run produced. Copying a stale artifact would pair someone
# else's result with this run's log — a fabricated pairing, which is precisely the class
# of false proof the ledger exists to prevent.
rm -f "${JUNIT_OUT}"
# Strip --record from the argv handed to the child; everything else passes through.
CHILD_ARGS=()
for a in ${ORIG_ARGS[@]+"${ORIG_ARGS[@]}"}; do
[ "$a" = "--record" ] || CHILD_ARGS+=("$a")
done
log "RECORD: capturing this run to runs/${RUN_ID}/ (run.log + report.junit.xml)"
set +e
CONF_RECORD_CHILD=1 "${BASH_SOURCE[0]}" ${CHILD_ARGS[@]+"${CHILD_ARGS[@]}"} 2>&1 \
| tee "${RUN_DIR}/run.log"
RUN_RC=${PIPESTATUS[0]}
set -e
if [ -f "${JUNIT_OUT}" ]; then
cp "${JUNIT_OUT}" "${RUN_DIR}/report.junit.xml"
else
log "RECORD: no ${JUNIT_OUT} was produced — the ledger step will SKIP (never green)."
fi
# Explicit, separate step: read the JUnit + the log and append one row per testcase.
# rc 3 = inputs absent (nothing recorded, not green); it must not mask the run's own rc.
# CONF_LEDGER exists so the harness's own tests can exercise this path without writing
# to the real ledger; operators should never set it.
"${PYTHON}" "${CONF_ROOT}/lib/record_run.py" \
--run-dir "${RUN_DIR}" --env "${CONF_ENV:-dev}" \
--ledger "${CONF_LEDGER:-${CONF_ROOT}/runs/ledger.jsonl}" || true
log "RECORD: run rc=${RUN_RC}. Commit conformance/runs/${RUN_ID}/ WITH its run.log and"
log "RECORD: conformance/runs/ledger.jsonl — an uncommitted ledger proves nothing."
exit "${RUN_RC}"
fi
# --- Coverage report is always produced (cluster-independent, R-TEST-04) --------------
gen_coverage() {
# Pass --allow-gap only when there IS an exemption: an empty DOCUMENTED_GAPS means no
@@ -161,11 +265,38 @@ lint_tests() {
}
# --- Select which test directories Chainsaw should run --------------------------------
# Requirement-scoped selection, then the DECLARED-SCOPE union on top:
#
# (--requirements ∩ --layer) ∪ (--include-scope ∩ --layer)
#
# The left half is R-TEST-03's touched subset. The right half is the fix for its blind
# spot: an invariant that quantifies over the whole tracked tree cannot be selected BY the
# requirements a change touches, because the change most likely to break it touches none of
# them. See lib/common.sh's scope block for the incidents. The union is fail-closed by
# construction — it can only ADD tests to a run, never remove one.
select_dirs() {
# --requirements AND --layer together => intersection (tests carrying the tag
# AND living in that layer subtree). Either alone keeps its prior meaning, so
# this is backward compatible (LAYER defaults to "all"). Both helpers emit
# sorted, de-duplicated dirs, so `comm -12` yields the intersection.
{
requirement_scoped_dirs
# The scope union is still LAYER-bounded: `--layer l0` must not pull a whole-repo-scoped
# L2 test into an offline run. With no --layer (all), nothing is bounded away.
if [ -n "${INCLUDE_SCOPES}" ]; then
if [ "${LAYER}" != all ]; then
comm -12 \
<(conf_dirs_for_scopes "${INCLUDE_SCOPES}") \
<(conf_dirs_for_layer "${LAYER}")
else
conf_dirs_for_scopes "${INCLUDE_SCOPES}"
fi
fi
} | sort -u
}
# The pre-existing requirement/layer selection, unchanged:
# --requirements AND --layer together => intersection (tests carrying the tag
# AND living in that layer subtree). Either alone keeps its prior meaning, so
# this is backward compatible (LAYER defaults to "all"). Both helpers emit
# sorted, de-duplicated dirs, so `comm -12` yields the intersection.
requirement_scoped_dirs() {
if [ -n "${REQS}" ] && [ "${LAYER}" != all ]; then
comm -12 \
<(conf_dirs_for_requirements "${REQS}") \
@@ -306,6 +437,12 @@ layers_are_offline_only() {
return 0
}
# Markdown one-liner per executed offline check, for the CI step summary. Written next to
# the JUnit; ci/l0.sh + ci/l1.sh splice it into $GITHUB_STEP_SUMMARY so the gate output still
# NAMES every check that ran (that is what the deleted per-check gate stages did, one hard-coded
# `summary` line each — now generated for all of them, whole-repo-scoped ones marked as such).
: "${OFFLINE_SUMMARY_OUT:=${ARTIFACTS}/offline-summary.md}"
run_offline_checks() {
local -a dirs
mapfile -t dirs < <(offline_selected_dirs)
@@ -313,14 +450,30 @@ run_offline_checks() {
log "No L0/L1 offline checks match the selection (layer=${LAYER} reqs='${REQS}')."
return 0
fi
# Declared scopes are passed through so the runner knows WHICH cases must not be allowed to
# report a SKIP as green (rc 3). See lib/common.sh's scope block and run_l1_offline.py.
local -a scope_args=() s
for s in ${INCLUDE_SCOPES//,/ }; do scope_args+=(--unwired-scope "$s"); done
log "Running ${#dirs[@]} offline L0/L1 check(s) (no cluster; docs/14)."
[ -n "${INCLUDE_SCOPES}" ] && log " unconditional (declared scope ${INCLUDE_SCOPES}): $(conf_dirs_for_scopes "${INCLUDE_SCOPES}" | xargs -n1 basename 2>/dev/null | tr '\n' ' ')"
"${PYTHON}" "${CONF_ROOT}/lib/run_l1_offline.py" \
--junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" "${dirs[@]}"
--junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" \
--scope-annotation "${CONF_SCOPE_ANNOTATION}" \
--summary-out "${OFFLINE_SUMMARY_OUT}" \
${scope_args[@]+"${scope_args[@]}"} "${dirs[@]}"
}
###############################################################################
# Main
###############################################################################
# --print-selection is deliberately handled FIRST: it resolves the selection and exits,
# writing nothing and running no check. A test that asserts "the gate's selection cannot
# filter me out" needs to ask that question without executing the suite it is part of.
if [ "${MODE}" = "print-selection" ]; then
select_dirs
exit 0
fi
log "Coverage report first (always)"
gen_coverage
@@ -350,17 +503,26 @@ esac
# L0 static + L1 rendering are cluster-independent (docs/14): run the selected offline
# checks and treat their result as authoritative REGARDLESS of cluster availability. An
# offline check failure is a real failure (nothing cluster-gated about it), so it always blocks.
# rc 3 is the harness-wide SKIPPED/not-run code (ci/README.md): NOT a pass, NOT a fail. The
# offline runner returns it when a test carrying a --unwired-scope scope SKIPPED — a whole-repo
# invariant reporting "nothing to audit" means the subtree it audits is gone, which the gate
# must see as UNWIRED. It is propagated verbatim so ci/l0.sh can set the gate's own rc=3,
# exactly as the deleted per-check stages did.
OFFLINE_RC=0
run_offline_checks || OFFLINE_RC=$?
if layers_are_offline_only; then
# Offline-only run (the ci/l0.sh + ci/l1.sh merge-gate paths): the offline result IS the result.
if [ "${OFFLINE_RC}" -eq 0 ]; then
log "Done (offline ${LAYER}). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md"
else
log "Offline ${LAYER} checks FAILED (rc=${OFFLINE_RC}). JUnit: ${JUNIT_OUT}"
fi
case "${OFFLINE_RC}" in
0) log "Done (offline ${LAYER}). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md" ;;
3) log "Offline ${LAYER}: a scope-declared check SKIPPED — UNWIRED, not green (rc=3). JUnit: ${JUNIT_OUT}" ;;
*) log "Offline ${LAYER} checks FAILED (rc=${OFFLINE_RC}). JUnit: ${JUNIT_OUT}" ;;
esac
exit "${OFFLINE_RC}"
fi
if [ "${OFFLINE_RC}" -eq 3 ]; then
log "Offline L0/L1: a scope-declared check SKIPPED — UNWIRED, not green (rc=3)."
exit 3
fi
if [ "${OFFLINE_RC}" -ne 0 ]; then
log "Offline L0/L1 checks FAILED (rc=${OFFLINE_RC}) — blocking (not cluster-gated)."
exit "${OFFLINE_RC}"
@@ -0,0 +1,4 @@
<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/node-loss" tests="1" failures="0" errors="0" id="0" time="">
<testcase name="node-loss" classname="" time="131.090121" />
</testsuite>
</testsuites>
@@ -0,0 +1,232 @@
[run.sh] Coverage report first (always)
[coverage] 42/51 requirements have a scaffolded test (82.4%); 9 gap(s), 0 safety-critical gap(s).
[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
[run.sh] LIGHT pass: no tests selected.
[run.sh] HEAVY pass: 1 test(s) SERIAL (parallel=1) — node-loss
Version: 0.2.12
Loading config (/work/conformance/config/chainsaw-config.yaml)...
- Using test file: chainsaw-test
- TestDirs [/work/conformance/tests/l3-fault-injection/node-loss]
- SkipDelete false
- FailFast false
- ReportFormat 'JUNIT-TEST'
- ReportName 'chainsaw-report-heavy'
- ReportPath '/work/conformance/_artifacts'
- Namespace ''
- FullName false
- IncludeTestRegex ''
- ExcludeTestRegex ''
- ApplyTimeout 1m30s
- AssertTimeout 3m0s
- CleanupTimeout 2m0s
- DeleteTimeout 2m0s
- ErrorTimeout 1m0s
- ExecTimeout 3m0s
- DeletionPropagationPolicy Background
- Parallel 1
- Template true
- NoCluster false
- PauseOnFailure false
Loading tests...
- node-loss (/work/conformance/tests/l3-fault-injection/node-loss)
Loading values...
Running tests...
=== RUN chainsaw
=== PAUSE chainsaw
=== CONT chainsaw
=== RUN chainsaw/node-loss
=== PAUSE chainsaw/node-loss
=== CONT chainsaw/node-loss
l.go:52: | 23:51:18 | node-loss | @chainsaw  | CREATE | OK | v1/Namespace @ chainsaw-fond-marlin
l.go:52: | 23:51:18 | node-loss | volume-stays-available-across-node-loss | TRY | BEGIN |
l.go:52: | 23:51:18 | node-loss | volume-stays-available-across-node-loss | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -euo pipefail
NS="$NAMESPACE"
# -- Gate 1: skip cleanly (green) if the longhorn tier is not present yet. --
if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
echo "SKIP: 'longhorn' StorageClass absent — node-loss not applicable on this cluster yet"
echo NODE_LOSS_SKIPPED_OK
exit 0
fi
# -- Gate 2: a 3-replica volume + single-node-loss headroom needs >= 3 Ready nodes. --
READY="$(kubectl get nodes \
-o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
| grep -c '^True$' || true)"
echo "ready nodes: ${READY}"
if [ "${READY:-0}" -lt 3 ]; then
echo "SKIP: only ${READY:-0} Ready node(s) (<3) — node-loss resilience not meaningful yet"
echo NODE_LOSS_SKIPPED_OK
exit 0
fi
CANARY="node-loss-canary-$(date +%s%N)"
# -- GIVEN: a Longhorn PVC (platform default SC = 3 replicas across nodes) + a
# writer pod that writes the canary and flushes it to the replicated volume. --
kubectl -n "$NS" apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: survivor
spec:
accessModes: [ReadWriteOnce]
storageClassName: longhorn
resources:
requests:
storage: 1Gi
EOF
apply_pod() {
kubectl -n "$NS" apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: $1
spec:
restartPolicy: Never
containers:
- name: app
image: busybox:1.36.1
command: ["sh", "-c", "sleep 3600"]
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: survivor
EOF
}
apply_pod writer
kubectl -n "$NS" wait --for=condition=Ready pod/writer --timeout=240s
printf '%s' "$CANARY" | kubectl -n "$NS" exec -i writer -- sh -c 'cat > /data/canary && sync'
echo "wrote canary: $(kubectl -n "$NS" exec writer -- cat /data/canary)"
# The victim node is the one the attached volume currently lives on.
VICTIM="$(kubectl -n "$NS" get pod writer -o jsonpath='{.spec.nodeName}')"
[ -n "$VICTIM" ] || { echo "FAIL: could not determine writer's node" >&2; exit 1; }
echo "victim node (currently hosts the attached volume): $VICTIM"
# Best-effort precondition log: if the Longhorn Volume CR is readable, show the
# volume is genuinely 3-replica/healthy before we take a node. Non-fatal (the
# runner is least-privilege and may not have longhorn.io read rights) — the
# data-availability assertion below is the authoritative proof, not this.
PVNAME="$(kubectl -n "$NS" get pvc survivor -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
if [ -n "$PVNAME" ]; then
kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
-o jsonpath='pre-fault robustness={.status.robustness} replicas={.spec.numberOfReplicas}{"\n"}' \
2>/dev/null || echo "(Longhorn Volume CR not readable — continuing; data read is the proof)"
fi
# -- WHEN: FAULT INJECTION — remove the victim node FROM SERVICE. --
# Label it with THIS test's unique namespace so `finally` can always find and
# uncordon exactly this node; then cordon it; then evict ONLY our own writer
# pod, forcing the volume to detach from the victim. (No `drain` — unrelated
# pods on the node are never touched.)
kubectl label node "$VICTIM" "conformance.home.k3s/cordoned-by=$NS" --overwrite
kubectl cordon "$VICTIM"
echo "cordoned + labelled victim node: $VICTIM"
kubectl -n "$NS" delete pod writer --wait=true --timeout=240s
echo "evicted writer off the victim node; volume detached from $VICTIM"
# -- THEN: a fresh workload re-binds the SAME volume. The victim is unschedulable,
# so the scheduler MUST place the reader on a SURVIVING node; Longhorn re-attaches
# the volume there from its replicas on the other nodes. --
apply_pod reader
kubectl -n "$NS" wait --for=condition=Ready pod/reader --timeout=360s
READER_NODE="$(kubectl -n "$NS" get pod reader -o jsonpath='{.spec.nodeName}')"
echo "reader scheduled on surviving node: $READER_NODE (victim was $VICTIM)"
if [ "$READER_NODE" = "$VICTIM" ]; then
echo "FAIL: reader landed back on the cordoned victim node — node was not removed from service" >&2
exit 1
fi
GOT="$(kubectl -n "$NS" exec reader -- cat /data/canary)"
echo "recovered='$GOT' expected='$CANARY'"
[ "$GOT" = "$CANARY" ] || { echo "FAIL: data not available after node loss" >&2; exit 1; }
echo "volume available + data intact from a surviving node"
# Best-effort reconvergence log: if the Longhorn Volume CR is readable, wait
# (bounded) for robustness to settle back to available (healthy|degraded — both
# serve I/O; a healthy 3-replica volume rebuilds a replica once headroom exists).
# Non-fatal for the same least-privilege reason as the pre-fault log.
if [ -n "$PVNAME" ]; then
for i in $(seq 1 30); do
ROB="$(kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
-o jsonpath='{.status.robustness}' 2>/dev/null || true)"
[ -z "$ROB" ] && { echo "(Longhorn Volume CR not readable — skipping reconvergence log)"; break; }
echo "post-fault robustness[$i]=$ROB"
case "$ROB" in healthy|degraded) break ;; esac
sleep 10
done
fi
echo NODE_LOSS_VOLUME_AVAILABLE_OK
l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | SCRIPT | LOG |
=== STDOUT
ready nodes: 3
persistentvolumeclaim/survivor created
pod/writer created
pod/writer condition met
wrote canary: node-loss-canary-1788306678898809444
victim node (currently hosts the attached volume): k3s-dev-2
pre-fault robustness=healthy replicas=3
node/k3s-dev-2 labeled
node/k3s-dev-2 cordoned
cordoned + labelled victim node: k3s-dev-2
pod "writer" deleted from chainsaw-fond-marlin namespace
evicted writer off the victim node; volume detached from k3s-dev-2
pod/reader created
pod/reader condition met
reader scheduled on surviving node: k3s-dev-1 (victim was k3s-dev-2)
recovered='node-loss-canary-1788306678898809444' expected='node-loss-canary-1788306678898809444'
volume available + data intact from a surviving node
post-fault robustness[1]=healthy
NODE_LOSS_VOLUME_AVAILABLE_OK
l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | SCRIPT | DONE |
l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | TRY | END |
l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | FINALLY | BEGIN |
l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -uo pipefail
for n in $(kubectl get nodes -l "conformance.home.k3s/cordoned-by=$NAMESPACE" -o name 2>/dev/null); do
kubectl uncordon "${n#node/}" || true
kubectl label "$n" "conformance.home.k3s/cordoned-by-" || true
echo "restored node to service: ${n#node/}"
done
l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | SCRIPT | LOG |
=== STDOUT
node/k3s-dev-2 uncordoned
node/k3s-dev-2 unlabeled
restored node to service: k3s-dev-2
l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | SCRIPT | DONE |
l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | CMD | RUN |
=== COMMAND
/usr/bin/sh -c kubectl -n "$NAMESPACE" delete pod writer reader --ignore-not-found --wait=false || true
kubectl -n "$NAMESPACE" delete pvc survivor --ignore-not-found --wait=false || true
echo "cleanup best-effort; chainsaw namespace teardown finalizes"
l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | SCRIPT | LOG |
=== STDOUT
pod "reader" deleted from chainsaw-fond-marlin namespace
persistentvolumeclaim "survivor" deleted from chainsaw-fond-marlin namespace
cleanup best-effort; chainsaw namespace teardown finalizes
l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | SCRIPT | DONE |
l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | FINALLY | END |
l.go:52: | 23:52:43 | node-loss | @chainsaw  | CLEANUP | BEGIN |
l.go:52: | 23:52:43 | node-loss | @chainsaw  | DELETE | OK | v1/Namespace @ chainsaw-fond-marlin
l.go:52: | 23:53:29 | node-loss | @chainsaw  | CLEANUP | END |
--- PASS: chainsaw (0.00s)
--- PASS: chainsaw/node-loss (131.09s)
PASS
Tests Summary...
- Passed tests 1
- Failed tests 0
- Skipped tests 0
Done.
merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
[run.sh] Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md
@@ -0,0 +1,4 @@
<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/dr-restore" tests="1" failures="0" errors="0" id="0" time="">
<testcase name="dr-restore" classname="" time="149.053701" />
</testsuite>
</testsuites>
@@ -0,0 +1,671 @@
[run.sh] Coverage report first (always)
[coverage] 42/51 requirements have a scaffolded test (82.4%); 9 gap(s), 0 safety-critical gap(s).
[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-03').
[run.sh] LIGHT pass: no tests selected.
[run.sh] HEAVY pass: 1 test(s) SERIAL (parallel=1) — dr-restore
Version: 0.2.12
Loading config (/work/conformance/config/chainsaw-config.yaml)...
- Using test file: chainsaw-test
- TestDirs [/work/conformance/tests/l3-fault-injection/dr-restore]
- SkipDelete false
- FailFast false
- ReportFormat 'JUNIT-TEST'
- ReportName 'chainsaw-report-heavy'
- ReportPath '/work/conformance/_artifacts'
- Namespace ''
- FullName false
- IncludeTestRegex ''
- ExcludeTestRegex ''
- ApplyTimeout 1m30s
- AssertTimeout 3m0s
- CleanupTimeout 2m0s
- DeleteTimeout 2m0s
- ErrorTimeout 1m0s
- ExecTimeout 3m0s
- DeletionPropagationPolicy Background
- Parallel 1
- Template true
- NoCluster false
- PauseOnFailure false
Loading tests...
- dr-restore (/work/conformance/tests/l3-fault-injection/dr-restore)
Loading values...
Running tests...
=== RUN chainsaw
=== PAUSE chainsaw
=== CONT chainsaw
=== RUN chainsaw/dr-restore
=== PAUSE chainsaw/dr-restore
=== CONT chainsaw/dr-restore
l.go:52: | 00:19:07 | dr-restore | @chainsaw  | CREATE | OK | v1/Namespace @ chainsaw-cunning-hen
l.go:52: | 00:19:07 | dr-restore | assert-backup-target-is-off-cluster  | TRY | BEGIN |
l.go:52: | 00:19:07 | dr-restore | assert-backup-target-is-off-cluster  | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -eu
# ---- helpers -------------------------------------------------------
# Classify a kubectl failure: an RBAC denial is a HARNESS DEFECT and must
# fail loudly; an absent API is a genuine precondition gap and may skip.
die_or_skip() {
case "$2" in
*orbidden*|*Unauthorized*|*cannot\ list*|*cannot\ get*)
echo "FAIL: conformance runner lacks RBAC for $1 -- fix conformance/incluster/rbac.yaml" >&2
echo "$2" >&2
exit 1 ;;
*)
echo "SKIP: $1 not available on this cluster -- $2"
echo DR_RESTORE_SKIPPED_OK
exit 0 ;;
esac
}
ip2int() {
_q="$(echo "$1" | tr '.' ' ')"
set -- $_q
[ $# -eq 4 ] || { echo -1; return 0; }
echo $(( ($1 * 16777216) + ($2 * 65536) + ($3 * 256) + $4 ))
}
# in_cidr <ip> <a.b.c.d/len> -> 0 when the ip is inside the prefix
in_cidr() {
case "$2" in *"/"*) ;; *) return 1 ;; esac
_net="${2%/*}"; _len="${2#*/}"
case "$_len" in ''|*[!0-9]*) return 1 ;; esac
[ "$_len" -ge 0 ] && [ "$_len" -le 32 ] || return 1
_i="$(ip2int "$1")"; _n="$(ip2int "$_net")"
[ "$_i" -ge 0 ] && [ "$_n" -ge 0 ] || return 1
[ "$_len" -eq 0 ] && return 0
_d=1; _s=0
while [ "$_s" -lt $((32 - _len)) ]; do _d=$((_d * 2)); _s=$((_s + 1)); done
[ $(( _i / _d )) -eq $(( _n / _d )) ]
}
# ---- Gate 1: the replicated storage tier the DR-critical data lives on --
if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
echo "SKIP: 'longhorn' StorageClass absent -- no DR-critical data tier on this cluster yet"
echo DR_RESTORE_SKIPPED_OK
exit 0
fi
# ---- Gate 2: the Velero API + a configured BackupStorageLocation --------
BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' \
2>&1)" || die_or_skip "velero.io/backupstoragelocations" "$BSL_LINES"
if [ -z "$(echo "$BSL_LINES" | tr -d '[:space:]')" ]; then
echo "SKIP: Velero API present but NO BackupStorageLocation is configured -- no off-cluster backup target yet"
echo DR_RESTORE_SKIPPED_OK
exit 0
fi
echo "backup storage locations:"
echo "$BSL_LINES"
VNS="$(echo "$BSL_LINES" | head -1 | cut -d' ' -f1)"
# ---- Gate 3: Velero server + node-agent (kopia FSB) actually ready -------
# File System Backup is the volume-DATA path (platform/velero/helmrelease.yaml:
# deployNodeAgent=true, uploaderType=kopia, snapshotsEnabled=false). Without a
# ready node-agent no volume data can be captured, so the drill is not runnable.
VREADY=0; NAREADY=0
for i in $(seq 1 12); do
VREADY="$(kubectl -n "$VNS" get deploy velero -o jsonpath='{.status.readyReplicas}' 2>/dev/null || true)"
NAREADY="$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || true)"
echo "poll $i: velero readyReplicas=${VREADY:-0} node-agent numberReady=${NAREADY:-0}"
if [ "${VREADY:-0}" -ge 1 ] && [ "${NAREADY:-0}" -ge 1 ]; then break; fi
sleep 10
done
if [ "${VREADY:-0}" -lt 1 ] || [ "${NAREADY:-0}" -lt 1 ]; then
echo "SKIP: Velero server and/or its node-agent DaemonSet are not ready in ns '$VNS' -- backup/restore path not available"
echo DR_RESTORE_SKIPPED_OK
exit 0
fi
# ---- Pick the BSL this drill will use: prefer the default, and it MUST be
# Available. A CONFIGURED-BUT-UNAVAILABLE target is NOT a precondition
# gap -- it is a live R-STATE-03 defect, so it FAILS (after a bounded
# wait for first-validation on a cold start), it does not skip. --------
BSL_NS=""; BSL_NAME=""; BSL_URL=""; BSL_BUCKET=""
for i in $(seq 1 12); do
BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' 2>/dev/null || true)"
CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true ' | head -1 || true)"
[ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
if [ -n "$CAND" ]; then
BSL_NS="$(echo "$CAND" | cut -d' ' -f1)"
BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
BSL_URL="$(echo "$CAND" | cut -d' ' -f6)"
BSL_BUCKET="$(echo "$CAND" | cut -d' ' -f7)"
break
fi
echo "poll $i: no Available BackupStorageLocation yet"
sleep 10
done
if [ -z "$BSL_NAME" ]; then
echo "FAIL: a BackupStorageLocation is configured but NONE reached phase 'Available'." >&2
echo " The off-cluster backup target is unreachable -- R-STATE-03 is BROKEN, not merely untested." >&2
echo "$BSL_LINES" >&2
exit 1
fi
echo "using BSL ${BSL_NS}/${BSL_NAME} bucket=${BSL_BUCKET} endpoint=${BSL_URL}"
# ---- OFF-CLUSTER PROOF (1): an endpoint we can evaluate ------------------
if [ -z "$BSL_URL" ]; then
echo "FAIL: BackupStorageLocation ${BSL_NS}/${BSL_NAME} exposes no spec.config.s3Url;" >&2
echo " the OFF-CLUSTER property of the backup target cannot be established, so this" >&2
echo " test refuses to claim it. (Extend this check for a non-S3 provider.)" >&2
exit 1
fi
HOST="${BSL_URL#*://}" # strip scheme
HOST="${HOST%%/*}" # strip path
HOST="${HOST%%:*}" # strip port
echo "backup endpoint host: $HOST"
case "$HOST" in
localhost|*.svc|*.svc.cluster.local|*.cluster.local)
echo "FAIL: backup endpoint '$HOST' is a CLUSTER-INTERNAL name -- the backup target is IN the cluster." >&2
echo " An in-cluster backup store cannot satisfy R-STATE-03 (off-cluster backup)." >&2
exit 1 ;;
esac
# Resolve to an IPv4 address. If it cannot be resolved we CANNOT establish the
# off-cluster property -- that is a failure, never a silent pass.
case "$HOST" in
*[!0-9.]*)
# Endpoint given as a name. Try the resolvers that may exist in the
# runner image, in order; if none is available or none resolves, the
# test FAILS below rather than guessing.
EPIP="$(getent ahostsv4 "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
[ -n "$EPIP" ] || EPIP="$(getent hosts "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
[ -n "$EPIP" ] || EPIP="$(nslookup "$HOST" 2>/dev/null | grep -A2 '^Name:' | grep '^Address' | head -1 | sed 's/.*: *//' || true)" ;;
*) EPIP="$HOST" ;;
esac
if [ -z "$EPIP" ]; then
echo "FAIL: could not resolve backup endpoint host '$HOST' to an IPv4 address;" >&2
echo " the OFF-CLUSTER property cannot be asserted, so this test does not pass." >&2
exit 1
fi
echo "backup endpoint resolves to: $EPIP"
# ---- OFF-CLUSTER PROOF (2): the endpoint is outside every cluster address --
case "$EPIP" in 127.*|0.0.0.0) echo "FAIL: backup endpoint $EPIP is loopback -- not off-cluster" >&2; exit 1 ;; esac
NODE_IPS="$(kubectl get nodes -o jsonpath='{range .items[*]}{range .status.addresses[*]}{.address}{"\n"}{end}{end}' 2>/dev/null || true)"
for a in $NODE_IPS; do
if [ "$a" = "$EPIP" ]; then
echo "FAIL: backup endpoint $EPIP is a NODE address of this cluster -- the backup is not off-cluster." >&2
exit 1
fi
done
echo "endpoint is not any node address (checked $(echo "$NODE_IPS" | grep -c . || true) node addresses)"
SVC_IPS="$(kubectl get svc -A -o jsonpath='{range .items[*]}{.spec.clusterIP}{"\n"}{end}' 2>/dev/null || true)"
for a in $SVC_IPS; do
if [ "$a" = "$EPIP" ]; then
echo "FAIL: backup endpoint $EPIP is a Service ClusterIP -- the backup target lives INSIDE the cluster." >&2
exit 1
fi
done
echo "endpoint is not any Service ClusterIP (checked $(echo "$SVC_IPS" | grep -c . || true))"
POD_IPS="$(kubectl get pods -A -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}' 2>/dev/null || true)"
for a in $POD_IPS; do
if [ "$a" = "$EPIP" ]; then
echo "FAIL: backup endpoint $EPIP is a POD IP -- the backup target is an in-cluster pod." >&2
exit 1
fi
done
echo "endpoint is not any Pod IP (checked $(echo "$POD_IPS" | grep -c . || true))"
POD_CIDRS="$(kubectl get nodes -o jsonpath='{range .items[*]}{.spec.podCIDR}{"\n"}{range .spec.podCIDRs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
for c in $POD_CIDRS; do
case "$c" in *:*) continue ;; esac # IPv6 prefixes are out of scope here
if in_cidr "$EPIP" "$c"; then
echo "FAIL: backup endpoint $EPIP falls inside node podCIDR $c -- it is in the cluster pod network." >&2
exit 1
fi
done
echo "endpoint is outside every node podCIDR (checked: $(echo "$POD_CIDRS" | tr '\n' ' '))"
# Best-effort (needs networking.k8s.io/ServiceCIDR, which the least-privilege
# runner may not read): the checks above are the authoritative ones.
SVC_CIDRS="$(kubectl get servicecidrs.networking.k8s.io -o jsonpath='{range .items[*]}{range .spec.cidrs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
if [ -n "$SVC_CIDRS" ]; then
for c in $SVC_CIDRS; do
case "$c" in *:*) continue ;; esac
if in_cidr "$EPIP" "$c"; then
echo "FAIL: backup endpoint $EPIP falls inside the Service CIDR $c -- it is in the cluster." >&2
exit 1
fi
done
echo "endpoint is outside every ServiceCIDR ($(echo "$SVC_CIDRS" | tr '\n' ' '))"
else
echo "(ServiceCIDR objects unreadable -- skipping that best-effort check; ClusterIP/PodIP/podCIDR checks stand)"
fi
echo "OFF_CLUSTER_TARGET_OK bsl=${BSL_NS}/${BSL_NAME} endpoint=${EPIP} bucket=${BSL_BUCKET}"
l.go:52: | 00:19:11 | dr-restore | assert-backup-target-is-off-cluster  | SCRIPT | LOG |
=== STDOUT
backup storage locations:
velero default Available aws true http://192.168.1.9:9000 velero
poll 1: velero readyReplicas=1 node-agent numberReady=3
using BSL velero/default bucket=velero endpoint=http://192.168.1.9:9000
backup endpoint host: 192.168.1.9
backup endpoint resolves to: 192.168.1.9
endpoint is not any node address (checked 6 node addresses)
endpoint is not any Service ClusterIP (checked 81)
endpoint is not any Pod IP (checked 186)
endpoint is outside every node podCIDR (checked: 10.42.0.0/24 10.42.0.0/24 10.42.1.0/24 10.42.1.0/24 10.42.2.0/24 10.42.2.0/24 )
endpoint is outside every ServiceCIDR (10.43.0.0/16 )
OFF_CLUSTER_TARGET_OK bsl=velero/default endpoint=192.168.1.9 bucket=velero
l.go:52: | 00:19:11 | dr-restore | assert-backup-target-is-off-cluster  | SCRIPT | DONE |
l.go:52: | 00:19:11 | dr-restore | assert-backup-target-is-off-cluster  | TRY | END |
l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled  | TRY | BEGIN |
l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled  | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -eu
SCHED="$(kubectl get schedules.velero.io -A \
-o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" phase="}{.status.phase}{" cron="}{.spec.schedule}{" lastBackup="}{.status.lastBackup}{"\n"}{end}' 2>/dev/null || true)"
if [ -z "$(echo "$SCHED" | tr -d '[:space:]')" ]; then
# Distinguish "Velero absent" (skip, in lock-step with step 1) from
# "Velero present but nothing scheduled" (a real R-STATE-03 defect).
if kubectl get backupstoragelocations.velero.io -A >/dev/null 2>&1; then
if [ -z "$(kubectl get backupstoragelocations.velero.io -A -o name 2>/dev/null || true)" ]; then
echo "SKIP: no BackupStorageLocation configured -- scheduling not applicable yet"
echo DR_SCHEDULE_SKIPPED_OK
exit 0
fi
echo "FAIL: Velero is configured with a backup target but declares NO Schedule --" >&2
echo " R-STATE-03 requires SCHEDULED off-cluster backups, not manual ones." >&2
exit 1
fi
echo "SKIP: Velero API absent -- backup scheduling not applicable on this cluster yet"
echo DR_SCHEDULE_SKIPPED_OK
exit 0
fi
echo "velero schedules:"
echo "$SCHED"
# A Schedule that exists but is paused/disabled backs nothing up.
if echo "$SCHED" | grep -q 'phase=Enabled'; then
echo BACKUPS_ARE_SCHEDULED_OK
else
echo "FAIL: no Velero Schedule is in phase 'Enabled' -- scheduled off-cluster backups are not running." >&2
exit 1
fi
l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled  | SCRIPT | LOG |
=== STDOUT
velero schedules:
velero/velero-daily phase=Enabled cron=0 3 * * * lastBackup=2026-09-01T03:00:27Z
velero/velero-daily-resources-only phase=Enabled cron=0 3 * * * lastBackup=2026-09-01T03:00:27Z
BACKUPS_ARE_SCHEDULED_OK
l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled  | SCRIPT | DONE |
l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled  | TRY | END |
l.go:52: | 00:19:11 | dr-restore | data-restores-from-the-off-cluster-backup | TRY | BEGIN |
l.go:52: | 00:19:11 | dr-restore | data-restores-from-the-off-cluster-backup | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -eu
NS="$NAMESPACE"
SRC="${NS}-dr-src" # scratch source namespace (destroyed by design)
DST="${NS}-dr-dst" # clean restore target (must not pre-exist)
BK="${NS}-dr-backup"
RS="${NS}-dr-restore"
OWNER_LABEL="conformance.home.k3s/created-by=${NS}"
# ---- Re-derive the preconditions (steps do not share shell state). Every
# skip below mirrors step 1 exactly, so the whole test skips or runs
# as a unit -- it can never half-run. -----------------------------
if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
echo "SKIP: 'longhorn' StorageClass absent"; echo DR_RESTORE_SKIPPED_OK; exit 0
fi
BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.default}{"\n"}{end}' 2>/dev/null || true)"
CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true$' | head -1 || true)"
[ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
if [ -z "$CAND" ]; then
echo "SKIP: no Available BackupStorageLocation (see the off-cluster step for the authoritative verdict)"
echo DR_RESTORE_SKIPPED_OK; exit 0
fi
VNS="$(echo "$CAND" | cut -d' ' -f1)"
BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
if [ "$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || echo 0)" -lt 1 ]; then
echo "SKIP: Velero node-agent (kopia FSB) not ready -- volume data cannot be backed up"
echo DR_RESTORE_SKIPPED_OK; exit 0
fi
echo "velero ns=$VNS bsl=$BSL_NAME src=$SRC dst=$DST"
# A clean restore target means PREVIOUSLY NONEXISTENT, and the source name must
# be free. If either is lying around from an aborted run, remove it before we
# claim "clean" -- and fail loudly if it will not go, because a half-deleted
# namespace would silently invalidate the destroy proof below.
kubectl delete namespace "$DST" "$SRC" --ignore-not-found --wait=true --timeout=300s
# ============================ GIVEN ==============================
# A DR-critical stateful workload with known data on the replicated tier.
CANARY="dr-canary-$(date +%s%N)"
kubectl create namespace "$SRC"
kubectl label namespace "$SRC" "$OWNER_LABEL" --overwrite
kubectl -n "$SRC" apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: dr-data
labels:
conformance.home.k3s/created-by: ${NS}
spec:
accessModes: [ReadWriteOnce]
storageClassName: longhorn
resources:
requests:
storage: 1Gi
EOF
kubectl -n "$SRC" apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: dr-workload
labels:
app: dr-workload
conformance.home.k3s/created-by: ${NS}
spec:
restartPolicy: Never
containers:
- name: app
image: busybox:1.36.1
command: ["sh", "-c", "sleep 7200"]
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: dr-data
EOF
kubectl -n "$SRC" wait --for=condition=Ready pod/dr-workload --timeout=300s
printf '%s' "$CANARY" | kubectl -n "$SRC" exec -i dr-workload -- sh -c 'cat > /data/canary && sync'
echo "seeded canary: $(kubectl -n "$SRC" exec dr-workload -- cat /data/canary)"
ORIG_PV="$(kubectl -n "$SRC" get pvc dr-data -o jsonpath='{.spec.volumeName}')"
[ -n "$ORIG_PV" ] || { echo "FAIL: source PVC never bound to a PV" >&2; exit 1; }
echo "original in-cluster PersistentVolume: $ORIG_PV"
# ======================= WHEN (part 1): back up off-cluster =======
# Created as a Velero Backup CR -- exactly what `velero backup create` does.
# defaultVolumesToFsBackup:true selects the kopia File System Backup path, so
# the VOLUME DATA (not just the resource JSON) is uploaded to the object store.
# ttl is short: if the finally-step DeleteBackupRequest is ever lost, Velero's
# own GC still removes this drill's objects from the bucket.
kubectl apply -f - <<EOF
apiVersion: velero.io/v1
kind: Backup
metadata:
name: ${BK}
namespace: ${VNS}
labels:
conformance.home.k3s/created-by: ${NS}
spec:
includedNamespaces: [${SRC}]
storageLocation: ${BSL_NAME}
snapshotVolumes: false
defaultVolumesToFsBackup: true
ttl: 2h0m0s
EOF
BPHASE=""
for i in $(seq 1 90); do
BPHASE="$(kubectl -n "$VNS" get backups.velero.io "$BK" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
echo "poll $i: backup phase=${BPHASE:-<pending>}"
case "${BPHASE:-}" in
Completed|Failed|PartiallyFailed|FailedValidation) break ;;
esac
sleep 10
done
if [ "${BPHASE:-}" != "Completed" ]; then
echo "FAIL: off-cluster backup did not complete (phase=${BPHASE:-<timeout>})" >&2
kubectl -n "$VNS" get backups.velero.io "$BK" -o yaml >&2 || true
exit 1
fi
echo "backup $BK Completed to ${BSL_NAME}"
# ---- OFF-CLUSTER PROOF (3): the VOLUME DATA really landed in the remote
# kopia repository -- a PodVolumeBackup Completed with a snapshot ID.
PVB="$(kubectl -n "$VNS" get podvolumebackups.velero.io \
-l velero.io/backup-name="$BK" \
-o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.phase}{" "}{.status.snapshotID}{"\n"}{end}' 2>/dev/null || true)"
echo "pod volume backups:"
echo "$PVB"
if [ -z "$(echo "$PVB" | tr -d '[:space:]')" ]; then
echo "FAIL: the backup captured NO pod volume data (no PodVolumeBackup)." >&2
echo " Resource-only backups cannot restore DR-critical DATA." >&2
exit 1
fi
# A line is "<name> <phase> <snapshotID>"; require Completed AND a
# non-empty snapshot ID (an empty ID means nothing reached the repository).
GOOD_PVB="$(echo "$PVB" | grep -cE ' Completed [^[:space:]]+$' || true)"
if [ "${GOOD_PVB:-0}" -lt 1 ]; then
echo "FAIL: no PodVolumeBackup reached phase Completed with a snapshot ID --" >&2
echo " the volume data did not reach the off-cluster repository." >&2
exit 1
fi
echo "volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)"
# ============ WHEN (part 2): FAULT INJECTION -- annihilate the source ====
# Destroy EVERY in-cluster copy: the pod, the PVC, the namespace, and the
# underlying PersistentVolume (and with it the Longhorn replicas). After this
# there is no in-cluster source the restore could quietly read from.
kubectl delete namespace "$SRC" --wait=true --timeout=420s
for i in $(seq 1 30); do
if ! kubectl get namespace "$SRC" >/dev/null 2>&1; then break; fi
echo "poll $i: waiting for namespace $SRC to disappear"
sleep 10
done
if kubectl get namespace "$SRC" >/dev/null 2>&1; then
echo "FAIL: source namespace $SRC did not terminate; the destroy step is not convincing" >&2
exit 1
fi
echo "source namespace destroyed"
# The longhorn StorageClass reclaims Delete, so the PV should already be gone;
# force it if a Retain policy (or a stuck finalizer) left it behind, because a
# surviving PV would make the restore prove nothing about off-cluster recovery.
kubectl delete pv "$ORIG_PV" --ignore-not-found --wait=false || true
for i in $(seq 1 30); do
if ! kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then break; fi
echo "poll $i: waiting for PersistentVolume $ORIG_PV to disappear"
sleep 10
done
if kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then
echo "FAIL: the original PersistentVolume $ORIG_PV SURVIVED the destroy step." >&2
echo " An in-cluster copy of the data still exists, so a successful restore" >&2
echo " would NOT prove recovery from the off-cluster backup. Refusing to pass." >&2
exit 1
fi
echo "original PersistentVolume $ORIG_PV is GONE -- no in-cluster copy of the data remains"
# Best-effort corroboration (the least-privilege runner may not read longhorn.io):
# the Longhorn Volume CR and its replicas should be gone too.
if kubectl -n longhorn-system get volumes.longhorn.io "$ORIG_PV" >/dev/null 2>&1; then
echo "FAIL: Longhorn Volume $ORIG_PV still exists after PV deletion -- replicas survive in-cluster" >&2
exit 1
fi
echo "(longhorn Volume CR $ORIG_PV not present -- either deleted or not readable by this identity)"
# ============================ RESTORE ============================
# Into a CLEAN, previously-nonexistent namespace (Velero creates it), which is
# the catalog's "restored into a clean state".
kubectl apply -f - <<EOF
apiVersion: velero.io/v1
kind: Restore
metadata:
name: ${RS}
namespace: ${VNS}
labels:
conformance.home.k3s/created-by: ${NS}
spec:
backupName: ${BK}
includedNamespaces: [${SRC}]
namespaceMapping:
${SRC}: ${DST}
existingResourcePolicy: none
EOF
RPHASE=""
for i in $(seq 1 90); do
RPHASE="$(kubectl -n "$VNS" get restores.velero.io "$RS" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
echo "poll $i: restore phase=${RPHASE:-<pending>}"
case "${RPHASE:-}" in
Completed|Failed|PartiallyFailed|FailedValidation) break ;;
esac
sleep 10
done
if [ "${RPHASE:-}" != "Completed" ]; then
echo "FAIL: restore did not complete (phase=${RPHASE:-<timeout>})" >&2
kubectl -n "$VNS" get restores.velero.io "$RS" -o yaml >&2 || true
kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
exit 1
fi
echo "restore $RS Completed into $DST"
# ============================= THEN ==============================
# The service returns WITH ITS DATA. With kopia FSB the restored pod carries a
# `restore-wait` init container that blocks until the node-agent has written the
# volume contents back, so Ready here already implies the data landed.
if ! kubectl -n "$DST" wait --for=condition=Ready pod/dr-workload --timeout=600s; then
echo "FAIL: restored workload never became Ready in the clean namespace $DST" >&2
kubectl -n "$DST" describe pod dr-workload >&2 || true
kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
exit 1
fi
NEW_PV="$(kubectl -n "$DST" get pvc dr-data -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
echo "restored PVC bound to PersistentVolume: ${NEW_PV:-<none>}"
[ -n "$NEW_PV" ] || { echo "FAIL: restored PVC did not bind a PV" >&2; exit 1; }
if [ "$NEW_PV" = "$ORIG_PV" ]; then
echo "FAIL: the restored PVC rebound the ORIGINAL PersistentVolume ($ORIG_PV)." >&2
echo " That is an in-cluster survival, not an off-cluster restore." >&2
exit 1
fi
echo "restored onto a NEW volume ($NEW_PV != $ORIG_PV) -- the data came from outside the cluster"
GOT="$(kubectl -n "$DST" exec dr-workload -- cat /data/canary)"
echo "recovered='$GOT' expected='$CANARY'"
if [ "$GOT" != "$CANARY" ]; then
echo "FAIL: restored data does not match what was backed up -- DR restore lost/corrupted data" >&2
exit 1
fi
echo DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT | LOG |
=== STDOUT
velero ns=velero bsl=default src=chainsaw-cunning-hen-dr-src dst=chainsaw-cunning-hen-dr-dst
namespace/chainsaw-cunning-hen-dr-src created
namespace/chainsaw-cunning-hen-dr-src labeled
persistentvolumeclaim/dr-data created
pod/dr-workload created
pod/dr-workload condition met
seeded canary: dr-canary-1788308352223263220
original in-cluster PersistentVolume: pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745
backup.velero.io/chainsaw-cunning-hen-dr-backup created
poll 1: backup phase=InProgress
poll 2: backup phase=InProgress
poll 3: backup phase=Completed
backup chainsaw-cunning-hen-dr-backup Completed to default
pod volume backups:
chainsaw-cunning-hen-dr-backup-l95lw Completed 47fa2737115e17142fb41857cef4e779
volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)
namespace "chainsaw-cunning-hen-dr-src" deleted
source namespace destroyed
original PersistentVolume pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745 is GONE -- no in-cluster copy of the data remains
(longhorn Volume CR pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745 not present -- either deleted or not readable by this identity)
restore.velero.io/chainsaw-cunning-hen-dr-restore created
poll 1: restore phase=InProgress
poll 2: restore phase=InProgress
poll 3: restore phase=InProgress
poll 4: restore phase=Completed
restore chainsaw-cunning-hen-dr-restore Completed into chainsaw-cunning-hen-dr-dst
pod/dr-workload condition met
restored PVC bound to PersistentVolume: pvc-63c47973-9c7f-4423-a27e-caa008026e49
restored onto a NEW volume (pvc-63c47973-9c7f-4423-a27e-caa008026e49 != pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745) -- the data came from outside the cluster
recovered='dr-canary-1788308352223263220' expected='dr-canary-1788308352223263220'
DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
=== STDERR
Defaulted container "app" out of: app, restore-wait (init)
l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT | DONE |
l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | TRY | END |
l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | FINALLY | BEGIN |
l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -u
NS="$NAMESPACE"
SRC="${NS}-dr-src"; DST="${NS}-dr-dst"; BK="${NS}-dr-backup"; RS="${NS}-dr-restore"
VNS="$(kubectl get backupstoragelocations.velero.io -A -o jsonpath='{.items[0].metadata.namespace}' 2>/dev/null || true)"
# Velero objects first: a DeleteBackupRequest makes Velero remove the backup
# DATA from the off-cluster bucket, not just the CR (self-cleaning contract).
if [ -n "$VNS" ]; then
kubectl -n "$VNS" delete restores.velero.io "$RS" --ignore-not-found --wait=false || true
if kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1; then
kubectl apply -f - <<EOF || true
apiVersion: velero.io/v1
kind: DeleteBackupRequest
metadata:
name: ${BK}-delete
namespace: ${VNS}
labels:
conformance.home.k3s/created-by: ${NS}
spec:
backupName: ${BK}
EOF
for i in $(seq 1 18); do
kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1 || break
sleep 10
done
# Backstop if Velero could not process the request (the 2h ttl also GCs it).
kubectl -n "$VNS" delete backups.velero.io "$BK" --ignore-not-found --wait=false || true
kubectl -n "$VNS" delete deletebackuprequests.velero.io "${BK}-delete" --ignore-not-found --wait=false || true
fi
echo "velero objects for $NS cleaned up (ns $VNS)"
fi
# Scratch namespaces (both, in case the run aborted at any point).
for n in "$DST" "$SRC"; do
kubectl delete namespace "$n" --ignore-not-found --wait=false || true
done
# Anything left labelled as ours (defensive; the two names above are the only
# namespaces this test ever creates).
kubectl delete namespace -l "conformance.home.k3s/created-by=${NS}" --ignore-not-found --wait=false || true
# Released PVs from either namespace's PVCs would otherwise leak storage.
for pv in $(kubectl get pv -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.claimRef.namespace}{"\n"}{end}' 2>/dev/null | grep -E " (${SRC}|${DST})$" | cut -d' ' -f1 || true); do
kubectl delete pv "$pv" --ignore-not-found --wait=false || true
echo "released leftover PersistentVolume: $pv"
done
echo "dr-restore teardown complete (best-effort; every object was created by this test)"
l.go:52: | 00:21:28 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT | LOG |
=== STDOUT
restore.velero.io "chainsaw-cunning-hen-dr-restore" deleted from velero namespace
deletebackuprequest.velero.io/chainsaw-cunning-hen-dr-backup-delete created
velero objects for chainsaw-cunning-hen cleaned up (ns velero)
namespace "chainsaw-cunning-hen-dr-dst" deleted
namespace "chainsaw-cunning-hen-dr-dst" deleted
persistentvolume "pvc-63c47973-9c7f-4423-a27e-caa008026e49" deleted
released leftover PersistentVolume: pvc-63c47973-9c7f-4423-a27e-caa008026e49
dr-restore teardown complete (best-effort; every object was created by this test)
l.go:52: | 00:21:28 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT | DONE |
l.go:52: | 00:21:28 | dr-restore | data-restores-from-the-off-cluster-backup | FINALLY | END |
l.go:52: | 00:21:28 | dr-restore | @chainsaw  | CLEANUP | BEGIN |
l.go:52: | 00:21:28 | dr-restore | @chainsaw  | DELETE | OK | v1/Namespace @ chainsaw-cunning-hen
l.go:52: | 00:21:36 | dr-restore | @chainsaw  | CLEANUP | END |
--- PASS: chainsaw (0.00s)
--- PASS: chainsaw/dr-restore (149.05s)
PASS
Tests Summary...
- Passed tests 1
- Failed tests 0
- Skipped tests 0
Done.
merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
[run.sh] Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md
@@ -0,0 +1,12 @@
<?xml version="1.0" ?>
<testsuites name="home-k3s-conformance" tests="1" failures="0" errors="0" skipped="0">
<testsuite name="home-k3s-conformance L0/L1 (offline)" tests="1" failures="0" errors="0" skipped="0">
<testcase classname="L1:R-NET-05,R-NET-06,R-NET-07" name="egress-class-render">
<properties>
<property name="requirements" value="R-NET-05,R-NET-06,R-NET-07"/>
<property name="layer" value="L1"/>
<property name="path" value="/work/.claude/conf-clone/conformance/tests/l1-rendering/egress-class-render"/>
</properties>
</testcase>
</testsuite>
</testsuites>
@@ -0,0 +1,7 @@
[run.sh] Coverage report first (always)
[coverage] 49/51 requirements have a scaffolded test (96.1%); 2 gap(s), 0 safety-critical gap(s).
[coverage] wrote /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md and /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.json
[run.sh] Running 1 offline L0/L1 check(s) (no cluster; docs/14).
[offline] L1 PASS egress-class-render (R-NET-05,R-NET-06,R-NET-07)
[offline] wrote JUnit (1 case(s), 0 failure(s)) -> /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml
[run.sh] Done (offline l1). JUnit: /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml | Coverage: /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md
@@ -0,0 +1,4 @@
<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch" tests="1" failures="0" errors="0" id="0" time="">
<testcase name="vpn-killswitch" classname="" time="174.632779" />
</testsuite>
</testsuites>
@@ -0,0 +1,237 @@
[run.sh] Coverage report first (always)
[coverage] 49/51 requirements have a scaffolded test (96.1%); 2 gap(s), 0 safety-critical gap(s).
[coverage] wrote /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md and /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.json
[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-NET-06').
[run.sh] LIGHT pass: 1 test(s), parallel=2 (pinned v0.2.12)
Version: 0.2.12
Loading config (/work/.claude/conf-clone/conformance/config/chainsaw-config.yaml)...
- Using test file: chainsaw-test
- TestDirs [/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch]
- SkipDelete false
- FailFast false
- ReportFormat 'JUNIT-TEST'
- ReportName 'chainsaw-report-light'
- ReportPath '/work/.claude/conf-clone/conformance/_artifacts'
- Namespace ''
- FullName false
- IncludeTestRegex ''
- ExcludeTestRegex ''
- ApplyTimeout 1m30s
- AssertTimeout 3m0s
- CleanupTimeout 2m0s
- DeleteTimeout 2m0s
- ErrorTimeout 1m0s
- ExecTimeout 3m0s
- DeletionPropagationPolicy Background
- Parallel 2
- Template true
- NoCluster false
- PauseOnFailure false
Loading tests...
- vpn-killswitch (/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch)
Loading values...
Running tests...
=== RUN chainsaw
=== PAUSE chainsaw
=== CONT chainsaw
=== RUN chainsaw/vpn-killswitch
=== PAUSE chainsaw/vpn-killswitch
=== CONT chainsaw/vpn-killswitch
l.go:52: | 01:44:38 | vpn-killswitch | @chainsaw  | CREATE | OK | v1/Namespace @ chainsaw-credible-bird
l.go:52: | 01:44:38 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | TRY | BEGIN |
l.go:52: | 01:44:38 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -eu
kubectl label namespace "$NAMESPACE" \
pod-security.kubernetes.io/enforce=privileged \
pod-security.kubernetes.io/warn=privileged \
pod-security.kubernetes.io/audit=privileged \
--overwrite
echo "NS_PRIVILEGED_OK"
l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | SCRIPT | LOG |
=== STDOUT
namespace/chainsaw-credible-bird labeled
NS_PRIVILEGED_OK
l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | SCRIPT | DONE |
l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | APPLY | RUN | v1/Pod @ chainsaw-credible-bird/egress-internet
l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | CREATE | OK | v1/Pod @ chainsaw-credible-bird/egress-internet
l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | APPLY | DONE | v1/Pod @ chainsaw-credible-bird/egress-internet
l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | APPLY | RUN | v1/Pod @ chainsaw-credible-bird/egress-vpn
l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | CREATE | OK | v1/Pod @ chainsaw-credible-bird/egress-vpn
l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | APPLY | DONE | v1/Pod @ chainsaw-credible-bird/egress-vpn
l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | ASSERT | RUN | v1/Pod @ chainsaw-credible-bird/egress-internet
l.go:52: | 01:44:41 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | ASSERT | DONE | v1/Pod @ chainsaw-credible-bird/egress-internet
l.go:52: | 01:44:41 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | ASSERT | RUN | v1/Pod @ chainsaw-credible-bird/egress-vpn
l.go:52: | 01:44:41 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | ASSERT | DONE | v1/Pod @ chainsaw-credible-bird/egress-vpn
l.go:52: | 01:44:41 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -eu
ns="$NAMESPACE"
get_ip() {
kubectl -n "$ns" exec "$1" -c curl -- \
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true
}
wan="" vpn=""
for i in $(seq 1 12); do
[ -n "$wan" ] || wan="$(get_ip egress-internet)"
[ -n "$vpn" ] || vpn="$(get_ip egress-vpn)"
if [ -n "$wan" ] && [ -n "$vpn" ] && [ "$wan" != "$vpn" ]; then break; fi
[ "$wan" = "$vpn" ] && vpn="" # not steered yet; re-poll
sleep 5
done
echo "wan=$wan vpn=$vpn"
[ -n "$wan" ] || { echo "FAIL: no WAN IP from internet-class pod" >&2; exit 1; }
[ -n "$vpn" ] || { echo "FAIL: vpn-class egress not working at baseline" >&2; exit 1; }
[ "$wan" != "$vpn" ] || { echo "FAIL: vpn-class == WAN at baseline (not tunneled)" >&2; exit 1; }
echo "TUNNEL_HEALTHY_OK"
l.go:52: | 01:44:43 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | SCRIPT | LOG |
=== STDOUT
wan=136.51.115.252 vpn=158.173.241.11
TUNNEL_HEALTHY_OK
l.go:52: | 01:44:43 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | SCRIPT | DONE |
l.go:52: | 01:44:43 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | TRY | END |
l.go:52: | 01:44:43 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | TRY | BEGIN |
l.go:52: | 01:44:43 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | APPLY | RUN | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
l.go:52: | 01:44:43 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | CREATE | OK | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
l.go:52: | 01:44:43 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | APPLY | DONE | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
l.go:52: | 01:44:43 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | ASSERT | RUN | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
l.go:52: | 01:44:48 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | ASSERT | DONE | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
l.go:52: | 01:44:48 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -u
ns="$NAMESPACE"
wan="$(kubectl -n "$ns" exec egress-internet -c curl -- \
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
for i in $(seq 1 8); do
out="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
curl -fsS --max-time 8 https://api.ipify.org 2>/dev/null || true)"
if [ -n "$out" ]; then
if [ -n "$wan" ] && [ "$out" = "$wan" ]; then
echo "LEAK: vpn-class reached the WAN IP ($out) with the tunnel DOWN" >&2; exit 1
fi
echo "LEAK: vpn-class reached the internet ($out) with the tunnel DOWN" >&2; exit 1
fi
echo "attempt $i: vpn-class blackholed (no response), good"
sleep 3
done
echo "NO_LEAK_OK"
l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | SCRIPT | LOG |
=== STDOUT
attempt 1: vpn-class blackholed (no response), good
attempt 2: vpn-class blackholed (no response), good
attempt 3: vpn-class blackholed (no response), good
attempt 4: vpn-class blackholed (no response), good
attempt 5: vpn-class blackholed (no response), good
attempt 6: vpn-class blackholed (no response), good
attempt 7: vpn-class blackholed (no response), good
attempt 8: vpn-class blackholed (no response), good
NO_LEAK_OK
l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | SCRIPT | DONE |
l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -eu
ns="$NAMESPACE"
out="$(kubectl -n "$ns" exec egress-internet -c curl -- \
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
echo "internet_ip=$out"
[ -n "$out" ] || { echo "FAIL: internet-class egress broke while tunnel down (must be unaffected)" >&2; exit 1; }
echo "INTERNET_UNAFFECTED_OK"
l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | SCRIPT | LOG |
=== STDOUT
internet_ip=136.51.115.252
INTERNET_UNAFFECTED_OK
l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | SCRIPT | DONE |
l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | TRY | END |
l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | FINALLY | BEGIN |
l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -eu
ns="$NAMESPACE"
# Distinct Job name (tunnel-restart) so it never collides with the
# tunnel-stop Job. Left-aligned manifest so the block scalar yields
# valid YAML at column 0 at runtime.
cat <<'JOB' | kubectl -n "$ns" apply -f -
apiVersion: batch/v1
kind: Job
metadata:
name: tunnel-restart
labels: { app: vpn-killswitch-fault }
spec:
backoffLimit: 3
activeDeadlineSeconds: 90
ttlSecondsAfterFinished: 600
template:
metadata:
labels: { app: vpn-killswitch-fault }
spec:
restartPolicy: Never
hostPID: true
nodeSelector:
homelab.keithjackson.dev/egress-gateway: "true"
tolerations:
- operator: Exists
containers:
- name: nsenter
image: busybox:1.37.0
securityContext:
privileged: true
command: ["nsenter","-t","1","-m","-u","-i","-n","-p","--","systemctl","start","vpn-egress-tunnel"]
JOB
# Block teardown until the restart actually completed on the host — a
# leaked-down tunnel is unacceptable. `wait` exits non-zero on failure
# or timeout, which (set -e) fails the step and surfaces the problem.
kubectl -n "$ns" wait --for=condition=complete --timeout=110s job/tunnel-restart
echo "TUNNEL_SELF_HEAL_OK"
l.go:52: | 01:46:21 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | SCRIPT | LOG |
=== STDOUT
job.batch/tunnel-restart created
job.batch/tunnel-restart condition met
TUNNEL_SELF_HEAL_OK
l.go:52: | 01:46:21 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | SCRIPT | DONE |
l.go:52: | 01:46:21 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | FINALLY | END |
l.go:52: | 01:46:21 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | TRY | BEGIN |
l.go:52: | 01:46:21 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -eu
ns="$NAMESPACE"
wan="$(kubectl -n "$ns" exec egress-internet -c curl -- \
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
vpn=""
for i in $(seq 1 24); do
vpn="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
curl -fsS --max-time 8 https://api.ipify.org 2>/dev/null || true)"
if [ -n "$vpn" ] && [ "$vpn" != "$wan" ]; then break; fi
vpn=""
sleep 5
done
echo "recovered_vpn=$vpn wan=$wan"
[ -n "$vpn" ] || { echo "FAIL: vpn-class egress did not recover after tunnel restart" >&2; exit 1; }
[ "$vpn" != "$wan" ] || { echo "FAIL: vpn-class egress recovered onto the WAN (not tunneled)" >&2; exit 1; }
echo "TUNNEL_RECOVERED_OK"
l.go:52: | 01:46:25 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | SCRIPT | LOG |
=== STDOUT
recovered_vpn=158.173.241.11 wan=136.51.115.252
TUNNEL_RECOVERED_OK
l.go:52: | 01:46:25 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | SCRIPT | DONE |
l.go:52: | 01:46:25 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | TRY | END |
l.go:52: | 01:46:25 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | CLEANUP | BEGIN |
l.go:52: | 01:46:25 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | DELETE | OK | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
l.go:52: | 01:46:25 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | CLEANUP | END |
l.go:52: | 01:46:25 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | CLEANUP | BEGIN |
l.go:52: | 01:46:25 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | DELETE | OK | v1/Pod @ chainsaw-credible-bird/egress-vpn
l.go:52: | 01:46:56 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | DELETE | OK | v1/Pod @ chainsaw-credible-bird/egress-internet
l.go:52: | 01:47:26 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | CLEANUP | END |
l.go:52: | 01:47:26 | vpn-killswitch | @chainsaw  | CLEANUP | BEGIN |
l.go:52: | 01:47:26 | vpn-killswitch | @chainsaw  | DELETE | OK | v1/Namespace @ chainsaw-credible-bird
l.go:52: | 01:47:33 | vpn-killswitch | @chainsaw  | CLEANUP | END |
--- PASS: chainsaw (0.00s)
--- PASS: chainsaw/vpn-killswitch (174.63s)
PASS
Tests Summary...
- Passed tests 1
- Failed tests 0
- Skipped tests 0
Done.
[run.sh] HEAVY pass: no heavy tests selected.
merge_junit: merged 1 report(s) -> /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
[run.sh] Done. JUnit: /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml | Coverage: /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md
@@ -0,0 +1,4 @@
<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch" tests="1" failures="0" errors="0" id="0" time="">
<testcase name="vpn-killswitch" classname="" time="173.473422" />
</testsuite>
</testsuites>
@@ -0,0 +1,237 @@
[run.sh] Coverage report first (always)
[coverage] 49/51 requirements have a scaffolded test (96.1%); 2 gap(s), 0 safety-critical gap(s).
[coverage] wrote /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md and /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.json
[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-NET-06').
[run.sh] LIGHT pass: 1 test(s), parallel=2 (pinned v0.2.12)
Version: 0.2.12
Loading config (/work/.claude/conf-clone/conformance/config/chainsaw-config.yaml)...
- Using test file: chainsaw-test
- TestDirs [/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch]
- SkipDelete false
- FailFast false
- ReportFormat 'JUNIT-TEST'
- ReportName 'chainsaw-report-light'
- ReportPath '/work/.claude/conf-clone/conformance/_artifacts'
- Namespace ''
- FullName false
- IncludeTestRegex ''
- ExcludeTestRegex ''
- ApplyTimeout 1m30s
- AssertTimeout 3m0s
- CleanupTimeout 2m0s
- DeleteTimeout 2m0s
- ErrorTimeout 1m0s
- ExecTimeout 3m0s
- DeletionPropagationPolicy Background
- Parallel 2
- Template true
- NoCluster false
- PauseOnFailure false
Loading tests...
- vpn-killswitch (/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch)
Loading values...
Running tests...
=== RUN chainsaw
=== PAUSE chainsaw
=== CONT chainsaw
=== RUN chainsaw/vpn-killswitch
=== PAUSE chainsaw/vpn-killswitch
=== CONT chainsaw/vpn-killswitch
l.go:52: | 01:49:24 | vpn-killswitch | @chainsaw  | CREATE | OK | v1/Namespace @ chainsaw-beloved-toad
l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | TRY | BEGIN |
l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -eu
kubectl label namespace "$NAMESPACE" \
pod-security.kubernetes.io/enforce=privileged \
pod-security.kubernetes.io/warn=privileged \
pod-security.kubernetes.io/audit=privileged \
--overwrite
echo "NS_PRIVILEGED_OK"
l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | SCRIPT | LOG |
=== STDOUT
namespace/chainsaw-beloved-toad labeled
NS_PRIVILEGED_OK
l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | SCRIPT | DONE |
l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | APPLY | RUN | v1/Pod @ chainsaw-beloved-toad/egress-internet
l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | CREATE | OK | v1/Pod @ chainsaw-beloved-toad/egress-internet
l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | APPLY | DONE | v1/Pod @ chainsaw-beloved-toad/egress-internet
l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | APPLY | RUN | v1/Pod @ chainsaw-beloved-toad/egress-vpn
l.go:52: | 01:49:25 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | CREATE | OK | v1/Pod @ chainsaw-beloved-toad/egress-vpn
l.go:52: | 01:49:25 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | APPLY | DONE | v1/Pod @ chainsaw-beloved-toad/egress-vpn
l.go:52: | 01:49:25 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | ASSERT | RUN | v1/Pod @ chainsaw-beloved-toad/egress-internet
l.go:52: | 01:49:26 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | ASSERT | DONE | v1/Pod @ chainsaw-beloved-toad/egress-internet
l.go:52: | 01:49:26 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | ASSERT | RUN | v1/Pod @ chainsaw-beloved-toad/egress-vpn
l.go:52: | 01:49:26 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | ASSERT | DONE | v1/Pod @ chainsaw-beloved-toad/egress-vpn
l.go:52: | 01:49:26 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -eu
ns="$NAMESPACE"
get_ip() {
kubectl -n "$ns" exec "$1" -c curl -- \
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true
}
wan="" vpn=""
for i in $(seq 1 12); do
[ -n "$wan" ] || wan="$(get_ip egress-internet)"
[ -n "$vpn" ] || vpn="$(get_ip egress-vpn)"
if [ -n "$wan" ] && [ -n "$vpn" ] && [ "$wan" != "$vpn" ]; then break; fi
[ "$wan" = "$vpn" ] && vpn="" # not steered yet; re-poll
sleep 5
done
echo "wan=$wan vpn=$vpn"
[ -n "$wan" ] || { echo "FAIL: no WAN IP from internet-class pod" >&2; exit 1; }
[ -n "$vpn" ] || { echo "FAIL: vpn-class egress not working at baseline" >&2; exit 1; }
[ "$wan" != "$vpn" ] || { echo "FAIL: vpn-class == WAN at baseline (not tunneled)" >&2; exit 1; }
echo "TUNNEL_HEALTHY_OK"
l.go:52: | 01:49:27 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | SCRIPT | LOG |
=== STDOUT
wan=136.51.115.252 vpn=158.173.241.11
TUNNEL_HEALTHY_OK
l.go:52: | 01:49:27 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | SCRIPT | DONE |
l.go:52: | 01:49:27 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | TRY | END |
l.go:52: | 01:49:27 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | TRY | BEGIN |
l.go:52: | 01:49:27 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | APPLY | RUN | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
l.go:52: | 01:49:27 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | CREATE | OK | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
l.go:52: | 01:49:27 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | APPLY | DONE | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
l.go:52: | 01:49:27 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | ASSERT | RUN | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
l.go:52: | 01:49:31 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | ASSERT | DONE | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
l.go:52: | 01:49:31 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -u
ns="$NAMESPACE"
wan="$(kubectl -n "$ns" exec egress-internet -c curl -- \
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
for i in $(seq 1 8); do
out="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
curl -fsS --max-time 8 https://api.ipify.org 2>/dev/null || true)"
if [ -n "$out" ]; then
if [ -n "$wan" ] && [ "$out" = "$wan" ]; then
echo "LEAK: vpn-class reached the WAN IP ($out) with the tunnel DOWN" >&2; exit 1
fi
echo "LEAK: vpn-class reached the internet ($out) with the tunnel DOWN" >&2; exit 1
fi
echo "attempt $i: vpn-class blackholed (no response), good"
sleep 3
done
echo "NO_LEAK_OK"
l.go:52: | 01:51:01 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | SCRIPT | LOG |
=== STDOUT
attempt 1: vpn-class blackholed (no response), good
attempt 2: vpn-class blackholed (no response), good
attempt 3: vpn-class blackholed (no response), good
attempt 4: vpn-class blackholed (no response), good
attempt 5: vpn-class blackholed (no response), good
attempt 6: vpn-class blackholed (no response), good
attempt 7: vpn-class blackholed (no response), good
attempt 8: vpn-class blackholed (no response), good
NO_LEAK_OK
l.go:52: | 01:51:01 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | SCRIPT | DONE |
l.go:52: | 01:51:01 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -eu
ns="$NAMESPACE"
out="$(kubectl -n "$ns" exec egress-internet -c curl -- \
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
echo "internet_ip=$out"
[ -n "$out" ] || { echo "FAIL: internet-class egress broke while tunnel down (must be unaffected)" >&2; exit 1; }
echo "INTERNET_UNAFFECTED_OK"
l.go:52: | 01:51:02 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | SCRIPT | LOG |
=== STDOUT
internet_ip=136.51.115.252
INTERNET_UNAFFECTED_OK
l.go:52: | 01:51:02 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | SCRIPT | DONE |
l.go:52: | 01:51:02 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | TRY | END |
l.go:52: | 01:51:02 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | FINALLY | BEGIN |
l.go:52: | 01:51:02 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -eu
ns="$NAMESPACE"
# Distinct Job name (tunnel-restart) so it never collides with the
# tunnel-stop Job. Left-aligned manifest so the block scalar yields
# valid YAML at column 0 at runtime.
cat <<'JOB' | kubectl -n "$ns" apply -f -
apiVersion: batch/v1
kind: Job
metadata:
name: tunnel-restart
labels: { app: vpn-killswitch-fault }
spec:
backoffLimit: 3
activeDeadlineSeconds: 90
ttlSecondsAfterFinished: 600
template:
metadata:
labels: { app: vpn-killswitch-fault }
spec:
restartPolicy: Never
hostPID: true
nodeSelector:
homelab.keithjackson.dev/egress-gateway: "true"
tolerations:
- operator: Exists
containers:
- name: nsenter
image: busybox:1.37.0
securityContext:
privileged: true
command: ["nsenter","-t","1","-m","-u","-i","-n","-p","--","systemctl","start","vpn-egress-tunnel"]
JOB
# Block teardown until the restart actually completed on the host — a
# leaked-down tunnel is unacceptable. `wait` exits non-zero on failure
# or timeout, which (set -e) fails the step and surfaces the problem.
kubectl -n "$ns" wait --for=condition=complete --timeout=110s job/tunnel-restart
echo "TUNNEL_SELF_HEAL_OK"
l.go:52: | 01:51:05 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | SCRIPT | LOG |
=== STDOUT
job.batch/tunnel-restart created
job.batch/tunnel-restart condition met
TUNNEL_SELF_HEAL_OK
l.go:52: | 01:51:05 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | SCRIPT | DONE |
l.go:52: | 01:51:05 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | FINALLY | END |
l.go:52: | 01:51:05 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | TRY | BEGIN |
l.go:52: | 01:51:05 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -eu
ns="$NAMESPACE"
wan="$(kubectl -n "$ns" exec egress-internet -c curl -- \
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
vpn=""
for i in $(seq 1 24); do
vpn="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
curl -fsS --max-time 8 https://api.ipify.org 2>/dev/null || true)"
if [ -n "$vpn" ] && [ "$vpn" != "$wan" ]; then break; fi
vpn=""
sleep 5
done
echo "recovered_vpn=$vpn wan=$wan"
[ -n "$vpn" ] || { echo "FAIL: vpn-class egress did not recover after tunnel restart" >&2; exit 1; }
[ "$vpn" != "$wan" ] || { echo "FAIL: vpn-class egress recovered onto the WAN (not tunneled)" >&2; exit 1; }
echo "TUNNEL_RECOVERED_OK"
l.go:52: | 01:51:08 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | SCRIPT | LOG |
=== STDOUT
recovered_vpn=158.173.241.11 wan=136.51.115.252
TUNNEL_RECOVERED_OK
l.go:52: | 01:51:08 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | SCRIPT | DONE |
l.go:52: | 01:51:08 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | TRY | END |
l.go:52: | 01:51:08 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | CLEANUP | BEGIN |
l.go:52: | 01:51:08 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | DELETE | OK | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
l.go:52: | 01:51:08 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | CLEANUP | END |
l.go:52: | 01:51:08 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | CLEANUP | BEGIN |
l.go:52: | 01:51:08 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | DELETE | OK | v1/Pod @ chainsaw-beloved-toad/egress-vpn
l.go:52: | 01:51:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | DELETE | OK | v1/Pod @ chainsaw-beloved-toad/egress-internet
l.go:52: | 01:52:10 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | CLEANUP | END |
l.go:52: | 01:52:10 | vpn-killswitch | @chainsaw  | CLEANUP | BEGIN |
l.go:52: | 01:52:10 | vpn-killswitch | @chainsaw  | DELETE | OK | v1/Namespace @ chainsaw-beloved-toad
l.go:52: | 01:52:18 | vpn-killswitch | @chainsaw  | CLEANUP | END |
--- PASS: chainsaw (0.00s)
--- PASS: chainsaw/vpn-killswitch (173.47s)
PASS
Tests Summary...
- Passed tests 1
- Failed tests 0
- Skipped tests 0
Done.
[run.sh] HEAVY pass: no heavy tests selected.
merge_junit: merged 1 report(s) -> /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
[run.sh] Done. JUnit: /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml | Coverage: /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md
+5
View File
@@ -0,0 +1,5 @@
{"chainsaw_version": "v0.2.12", "env": "dev", "git_sha": "57d4df8c4f992732a49f3e2daddcace866778ab0", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NODE_LOSS_VOLUME_AVAILABLE_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-02"], "run_id": "20260901T235117Z-e9a4b66", "sentinel": "NODE_LOSS_VOLUME_AVAILABLE_OK", "test": "node-loss", "ts_utc": "2026-09-01T23:51:17Z"}
{"chainsaw_version": "v0.2.12", "env": "dev", "git_sha": "fdc3265585f4b775c0a6ae3fb33c837b0c643655", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK, DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-03"], "run_id": "20260902T001907Z-fdc3265", "sentinel": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK", "test": "dr-restore", "ts_utc": "2026-09-02T00:19:07Z"}
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "19fc8bcb27eab42a0007b28a831519d3e325c7b6", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L1", "note": "L1 offline check exited 0; run_l1_offline.py renders its skips as JUnit <skipped/>, so a green JUnit is truthful without a sentinel", "proof": "proven", "requirements": ["R-NET-05", "R-NET-06", "R-NET-07"], "run_id": "20260913T014321Z-19fc8bc", "sentinel": null, "test": "egress-class-render", "ts_utc": "2026-09-13T01:43:21Z"}
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "19fc8bcb27eab42a0007b28a831519d3e325c7b6-dirty", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NS_PRIVILEGED_OK, TUNNEL_HEALTHY_OK, NO_LEAK_OK, INTERNET_UNAFFECTED_OK, TUNNEL_SELF_HEAL_OK, TUNNEL_RECOVERED_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-NET-06"], "run_id": "20260913T014437Z-19fc8bc", "sentinel": "TUNNEL_RECOVERED_OK", "test": "vpn-killswitch", "ts_utc": "2026-09-13T01:44:37Z"}
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "9d0814ae61e6a0ba64d3789a71f1cd9c985efbf0", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NS_PRIVILEGED_OK, TUNNEL_HEALTHY_OK, NO_LEAK_OK, INTERNET_UNAFFECTED_OK, TUNNEL_SELF_HEAL_OK, TUNNEL_RECOVERED_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-NET-06"], "run_id": "20260913T014923Z-9d0814a", "sentinel": "TUNNEL_RECOVERED_OK", "test": "vpn-killswitch", "ts_utc": "2026-09-13T01:49:23Z"}
+47 -4
View File
@@ -93,9 +93,52 @@ and coverage extraction exact and robust.
break the thing and assert fail-closed / no-data-loss, with `finally` restoring state.
- **Never weaken a test to go green (R-TEST-06).** If the spec can't be met, escalate to
the PM — do not relax the assertion or the requirement.
- **Never write a per-environment value into a test (R-ENV-02).** No hostname suffix, no
Gateway VIP, no LB-pool range. One authored suite must run green against dev *and* prod,
so a test **asks the cluster it is pointed at** what it is. The mechanism is a chainsaw
binding over the same ConfigMap Flux substitutes from:
```yaml
spec:
bindings:
- name: baseDomain
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
```
Declare it at **test level** (step `outputs`/`bindings` are step-scoped), then consume it:
`($baseDomain)` for a pod `env:` value, `(join('.', ['myhost', $baseDomain]))` where a
full hostname is needed — including inside a `fixtures/*.yaml` applied with `apply: file:`,
which chainsaw templates with the same bindings. Prefer reading the *actual* observed value
over restating a range (e.g. assert the DNS answer equals `GATEWAY_VIP`, or equals the LB IP
the Service was assigned) — it is both stricter and environment-independent. Never take the
environment from a CLI flag or an exported variable: a flag someone forgets is how a suite
ends up silently asserting the other cluster's expectations. L1 goldens are the one
exception: they are rendering fixtures and carry the literal `${BASE_DOMAIN}` token.
### How a scaffold declares itself (and how `status.py` tells one from a verifier)
Scaffolds in this tree carry `TODO(T-x.y)` markers where a fixture/assertion lands as a
later roadmap phase delivers the capability under test. `lib/status.py` treats such a
marker — in the `chainsaw-test.yaml`, in `render-check.sh`, or in `check.sh` — as
outranking any `conformance.home.k3s/executable: "true"` annotation, so a placeholder body
can never be reported as an executable test.
later roadmap phase delivers the capability under test. `lib/status.py` classifies each
test EXECUTABLE or SCAFFOLD from the test's own files — `chainsaw-test.yaml`,
`render-check.sh`, `check.sh` — and either signal below **outranks**
`conformance.home.k3s/executable: "true"`, so a placeholder body can never be reported as
an executable test. Declare a scaffold in **both** positions:
1. **`# STATUS: scaffold — <what is missing>`** in the file's header comment block. The
declared state must be the first token after `STATUS:` (executable tests say
`# STATUS: EXECUTABLE` / `IMPLEMENTED` / `RUNNABLE`).
2. **A `TODO`/`FIXME` marker carrying a message, in the body** — the file from its first
non-comment line onward. `echo "TODO(T-4.2): inject a fault to fire a rule"`,
or an inline comment on a missing fixture:
`file: fixtures/smtp-sink.yaml # TODO(T-4.1): hermetic SMTP capture sink`.
What is **not** a scaffold marker, and why the distinction is load-bearing: a marker
mentioned in the **header prose** (the Given/When/Then and the mandatory "what this does
NOT prove" list), and a marker that is the **entire content of a quoted literal** — a
placeholder *detector*'s own vocabulary, e.g. `SEED = ["TBD", "TODO", "FIXME", ...]` in
`l0-static/placeholder-guard-coupling/check.sh`. A whole-file `"TODO" in text` test used to
classify that 32-assertion check as a scaffold and, through it, report the safety-critical
R-STATE-03 as `exec=1 scaffold=1`. If your check must hold a marker as data, keep each
idiom a **standalone quoted literal in a list**; a marker with text attached (a regex
alternation, several idioms in one string) still classifies as a scaffold, because
ambiguity here only ever downgrades.
@@ -0,0 +1,119 @@
# =============================================================================
# L0 STATIC — every metric an alert rule queries is CLAIMED by a scrape target this repo configures
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# THE DEFECT THIS CLOSES. An alert rule whose metric is never scraped is not a broken alert — it is
# an ABSENT alert that reads as a present one. `longhorn_volume_robustness == 2` over a TSDB with no
# longhorn_* series returns no vector, so the rule is permanently quiet, permanently "healthy", and
# indistinguishable from a correct rule on a healthy cluster. It survives review (the PromQL is
# right), it survives `promtool test rules` (the unit test supplies the series itself), and it
# survives every dashboard (nobody looks for a panel that was never built).
#
# MEASURED ON PROD, 2026-09-13, before this change — 13 rules across three families were inert:
# count(longhorn_volume_robustness) => (empty) 4 rules
# count({__name__=~"certmanager.*"}) => (empty) 3 rules
# count(hubble_drop_total) => (empty) 1 rule
# count by (job) (up) => 20 jobs, none Longhorn/cert-manager/Hubble
# So storage health was unmonitored on a cluster whose prod-1 had just had a failing SSD
# (DECISIONS-LOG), TLS expiry was unmonitored with the live `wildcard-home` wildcard 89 days from
# expiry, and the platform had no policy-drop visibility at all. The Flux family (gotk_*, 5 rules)
# was the same defect and was fixed separately in a79693d.
#
# GIVEN every alerting rule in every PrometheusRule in the repo (platform/**, apps/wave-*),
# WHEN each rule's PromQL is parsed for the metric names it selects,
# THEN (1) every metric name is CLAIMED by exactly one declared scrape owner, and each owner's
# claim is PROVEN AGAINST THE REPO — the file that enables that scrape must still contain
# the enabling declaration, so deleting `metrics.serviceMonitor.enabled` from
# platform/longhorn/helmrelease.yaml fails this check instead of silently re-inerting four
# storage alerts;
# (2) an owner's NEGATIVE conditions hold too, where the enablement can be defeated without
# being deleted — cert-manager's chart calls `fail` if podmonitor and servicemonitor are
# both set, Longhorn's own NetworkPolicies drop the scrape unless
# networkpolicy-metrics.yaml is wired, and kube_node_labels does not exist at all unless
# kube-state-metrics' metricLabelsAllowlist names the label;
# (3) a metric with NO owner FAILS, unless it is declared in KNOWN_UNWIRED_METRICS — a
# shrink-only register (hubble_drop_total today, with the CNI-rollout reason recorded);
# (4) `up{job="X"}` is audited on its own axis, because `up` always exists and so the metric
# check above can never see this failure: where the repo authors a Service and a
# ServiceMonitor in the SAME file, the monitor's endpoint port must be a port that Service
# actually defines. All 12 apps/* `<app>-down` SLI rules fail this today (the monitors name
# port `metrics`; no app Service defines one), so they are registered in
# KNOWN_UNWIRED_JOBS with the live proof that their `up{job=...}` series do not exist;
# AND (5) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope: whole-repo`,
# so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into EVERY gate
# invocation, and ci/gate.sh blocks the merge when the l0 stage fails.
#
# WHY BOTH REGISTERS ARE RATCHETS AND NOT BLANKET FAILURES: the same reasoning as
# alert-rules-unit-tested. Failing outright on the 12 app SLI rules would make the gate red until
# exporters exist for Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr and friends —
# none of which speak Prometheus natively — so the gate would be disabled, and a disabled gate
# protects nothing. Merely printing the gap is how these 13 rules stayed inert for three phases.
# So each gap is DECLARED, counted on every run, and CANNOT GROW: a new alert on an unclaimed
# metric FAILS, a new Service+ServiceMonitor pair with a mismatched port FAILS, and a STALE
# register entry (the gap was closed, or the alert deleted) FAILS too, so neither register can rot
# into fiction.
#
# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (PyYAML, the same dependency as
# catalog/render.py and dev-nodes-provisioned-as-code); conformance/run.sh --layer l0 and ci/l0.sh
# run it directly, and this Chainsaw Test wraps the SAME script.
#
# NOT proven (all named in check.sh's header, which carries the full list). The one worth stating
# here because it bit this very repo: an owner claims metrics by PREFIX, so this proves the SCRAPE
# TARGET exists, never that the exporter emits that exact SERIES NAME.
# `kubevirt_vmi_phase_count{phase="Failed"}` sat inside the scraped kubevirt_ family on an UP target
# and did not exist at all on the deployed KubeVirt (corrected here to
# `kubevirt_vmi_info{phase="failed"}`, lowercase — a capital-F matcher is equally inert). Catching
# that needs the exporter's own metric list, i.e. a live read; it belongs to a future L1/L3 check
# that diffs alert metrics against /api/v1/label/__name__/values.
#
# Also not proven: that a target is UP. This is static — it asserts the
# enabling declaration is in git, never that the endpoint answers, that the chart still honours the
# value key, or that a NetworkPolicy is sufficient. Those were established by live prod reads
# recorded in the commit message (longhorn_volume_robustness 44 series, certmanager_* 104 series,
# all six new targets up==1). It also cannot check a monitor whose Service comes from a Helm chart
# rather than from this repo — the port name lives outside the tree — which is why (4) is scoped to
# same-file Service+ServiceMonitor pairs. And it does not judge thresholds or reachability-to-human
# (R-OBS-02 / the L3 alerting test).
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: alert-metrics-have-scrape-targets
labels:
conformance.home.k3s/R-OBS-04: "true"
annotations:
conformance.home.k3s/requirements: "R-OBS-04"
conformance.home.k3s/layer: "L0"
# WHOLE-REPO SCOPE — this invariant quantifies over every PrometheusRule in the tree AND over
# every scrape-enabling declaration that feeds them, which live in eight different components
# (platform/monitoring, platform/longhorn, platform/cert-manager, platform/velero,
# platform/kubevirt, platform/keycloak, platform/cnpg, apps/wave-*). No single requirement
# path-maps to that set.
#
# WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh maps apps/* to NOTHING and
# platform/monitoring/* to R-OBS-01/R-OBS-02 — not R-OBS-04. A change that adds an alert under
# apps/ or platform/kubevirt/, or one that flips a chart's `serviceMonitor.enabled` back to
# false under platform/longhorn/, therefore attributes a requirement set that would FILTER THIS
# CHECK OUT of exactly the change that re-inerts the alerts. Declaring the scope makes
# ci/l0.sh Stage 3b union it into every run instead, and makes an honest SKIP loud (rc 3 =
# UNWIRED, never green) rather than silently green.
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Every metric name queried by every alert rule is claimed by a scrape target whose enabling
declaration is still present in this repo, every `up{job=...}` selector names a job a repo
monitor can actually produce, and the two gaps that remain are enumerated in registers that
can only shrink — so an alert can never again be authored, reviewed and merged against a
metric nothing scrapes.
spec:
steps:
- name: audit-every-alert-metric-against-the-configured-scrape-targets
description: >-
Parse every metric name out of every alerting rule, resolve each against the declared
scrape-owner table, prove each owner's enabling declaration (and its negative conditions)
still hold in git, audit `up{job=...}` against same-file Service/ServiceMonitor port
agreement, and ratchet both registers of declared gaps.
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,702 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — alert-metrics-have-scrape-targets
# Serves: R-OBS-04 ("The platform alerts on its own health (nodes, storage, GitOps, certs,
# VPN)." — spec docs/10-observability-and-alerting.md)
#
# WHY THIS EXISTS — AN ALERT ON AN UNSCRAPED METRIC IS AN ABSENT ALERT THAT READS AS A PRESENT ONE
# -----------------------------------------------------------------------------------------------
# `longhorn_volume_robustness == 2` evaluated against a TSDB that holds no longhorn_* series
# returns no vector. The rule is therefore permanently quiet, permanently "healthy", and
# INDISTINGUISHABLE from a correct rule on a healthy cluster. It passes review (the PromQL is
# right). It passes `promtool test rules` (the unit test supplies the series itself, so a unit
# test proves nothing about whether anything scrapes them). It shows up on no dashboard, because
# nobody looks for a panel that was never built.
#
# Measured on prod 2026-09-13, BEFORE the change that added this check — 13 rules were inert:
# count(longhorn_volume_robustness) => (empty) LonghornVolumeDegraded/Faulted/
# CapacityHigh, LonghornNodeStorageHigh
# count({__name__=~"certmanager.*"}) => (empty) CertificateExpiringSoon/
# ExpiryImminent/NotReady
# count(hubble_drop_total) => (empty) HubbleDeniedFlowSpike
# count by (job) (up) => 20 jobs, none Longhorn, cert-manager or Hubble
# Storage health was unmonitored on the cluster whose prod-1 had just had a failing SSD; TLS
# expiry was unmonitored with the live `wildcard-home` wildcard 89 days out; there was no
# policy-drop visibility at all. The gotk_* family (5 more rules) was the same defect, fixed
# separately in a79693d.
#
# WHAT MAKES THIS CLASS DIFFERENT from the two failure modes alert-rules-unit-tested catches: a
# permanently-firing rule is loud, and a rule that behaves wrongly against synthetic series is
# caught by promtool. This one is caught by NEITHER, because the missing thing is not in the rule
# file at all — it is a `serviceMonitor.enabled` that was never set in a completely different
# component's HelmRelease. So the audit has to cross from the alert to the scrape config, which is
# what the owner table below does.
#
# GIVEN every alerting rule in every PrometheusRule in the repo,
# WHEN each rule's PromQL is parsed for the metric names it selects,
# THEN (1) every metric name is CLAIMED by exactly one declared owner in SCRAPE_OWNERS, and the
# owner's `require` regexes still match its file — so deleting the enablement from
# platform/longhorn/helmrelease.yaml FAILS here instead of silently re-inerting four
# storage alerts;
# (2) each owner's `forbid` regexes do NOT match, for the cases where the enablement can be
# defeated without being deleted (cert-manager's chart `fail`s if podmonitor and
# servicemonitor are both enabled; an `enabled: false` inside the exporter's own block);
# (3) a metric with no owner FAILS unless declared in KNOWN_UNWIRED_METRICS (shrink-only);
# (4) `up{job="X"}` is audited separately — `up` itself always exists, so (1) is blind to
# this — by requiring that where this repo authors a Service and a ServiceMonitor in the
# SAME file, the monitor's endpoint port is a port that Service actually defines;
# violations must be declared in KNOWN_UNWIRED_JOBS (shrink-only);
# AND (5) the check is LOAD-BEARING: this test declares `conformance.home.k3s/scope: whole-repo`
# so ci/l0.sh unions it into every gate run, and ci/gate.sh blocks on an l0 failure.
#
# WHY THE REGISTERS ARE RATCHETS (the decision and its reasons) — the same three options as
# alert-rules-unit-tested faced:
# * FAIL outright on all 13: correct in principle, but 12 of them are app SLI rules that need
# Prometheus exporters for Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr and Seerr —
# none of which speak Prometheus natively. The gate would be red for every change until that
# work lands, so it would get disabled, and a disabled gate protects nothing.
# * merely REPORT: that is precisely how these rules stayed inert across phases 4-6.
# * RATCHET (chosen): each gap is declared here, printed with counts on every run, and CANNOT
# GROW. A new alert on an unclaimed metric FAILS. A new same-file Service+ServiceMonitor pair
# with a mismatched port FAILS. A STALE entry — the gap was closed, or the alert deleted —
# also FAILS, so neither register can rot into fiction.
#
# WHAT THIS DOES *NOT* PROVE (honest gaps — this is a STATIC check):
# - It does not prove a target is UP, that an endpoint answers, that the chart still honours the
# values key, or that a NetworkPolicy is sufficient. Those were established by live prod reads
# recorded in the commit message: with the enablements applied, longhorn_volume_robustness
# returned 44 series, {__name__=~"certmanager.*"} returned 104, and all six new targets read
# up==1. A live probe needs a cluster, which L0 by definition does not have.
# - It cannot audit a ServiceMonitor whose Service is created by a Helm chart rather than by this
# repo: the port name lives outside the tree. That is why (4) is scoped to same-file
# Service+ServiceMonitor pairs — the apps/* ServiceDefinition pattern — and why the
# platform/* monitors (oauth2-proxy, kubevirt, keycloak, flux) are out of its reach. Their
# liveness is covered by the fact that their metrics have owners under (1).
# - IT PROVES THE SCRAPE TARGET, NOT THE SERIES NAME. An owner claims a metric by PREFIX, so a
# metric in a scraped family whose exact name the exporter does not emit still passes. This is
# a real residual hole and it was live in this repo: `kubevirt_vmi_phase_count{phase="Failed"}`
# (KubeVirtVMLaunchFailing) sat inside the scraped `kubevirt_` family on an UP target, and the
# series simply does not exist on the deployed KubeVirt — verified on prod 2026-09-13, where the
# only kubevirt_vmi_* names are `info` and `number_of_outdated`, and the only phase-bearing
# names are the phase_transition_time histograms. Corrected in the same change to
# `kubevirt_vmi_info{phase="failed"}` (note the LOWERCASE value KubeVirt writes: a capital-F
# matcher selects nothing and is just as inert). Catching that class needs the exporter's own
# metric list, which is a live read, not a static one — it belongs to a future L1/L3 check that
# diffs every alert metric against `/api/v1/label/__name__/values`. The same limit applies to
# every LABEL VALUE an alert matches on.
# - The metric-name parser is a PromQL approximation, not a parser: it strips string literals,
# label-matcher bodies and aggregation label lists, then treats remaining bare identifiers as
# metric selectors. A metric referenced ONLY through a `{__name__=~"..."}` regex selector would
# be missed. No rule in this repo does that, and assertion (0) fails if the parser stops
# finding metrics at all.
# - It does not judge thresholds, and does not prove an alert reaches a human (R-OBS-02 and the
# L3 alerting test own that).
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
#
# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, and because this test
# declares the whole-repo scope the runner turns it into rc 3 = UNWIRED, never green) only if the
# repo contains no PrometheusRule at all — i.e. there is genuinely nothing to audit.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
# Honest SKIP (never green for a scope-declaring test) if there is nothing to audit at all.
if ! grep -rl 'kind: PrometheusRule' "${ROOT}/platform" "${ROOT}/apps" >/dev/null 2>&1; then
echo "check SKIPPED: no PrometheusRule found under platform/ or apps/ — no alert rules to audit" >&2
exit 3
fi
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
"""Static audit: every alert rule's metrics are claimed by a scrape target this repo configures.
PyYAML is used deliberately rather than the line-based parsing the neighbouring checks favour:
alert `expr:` values are multi-line block scalars, and a line-based reader of PromQL would be the
kind of approximate parser that silently stops matching — the exact failure class this check
exists to prevent. Same dependency as catalog/render.py and dev-nodes-provisioned-as-code.
"""
from __future__ import annotations
import re
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree
FAILURES: list[str] = []
CHECKS = 0
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
def read(p: Path) -> str:
try:
return p.read_text()
except (OSError, UnicodeDecodeError):
return ""
# =============================================================================================
# THE DECLARED SCRAPE-OWNER TABLE (assertions 1 and 2).
#
# Each entry claims a set of metric names (exact names and/or prefixes) and names the file whose
# declaration is what makes those metrics exist in the TSDB, plus the regexes that prove it. This
# is the crossing from "an alert queries X" to "something in this repo scrapes X", and it is the
# part that makes the check a REGRESSION guard rather than a one-time audit: every `require`
# regex is re-evaluated against the working tree on every gate run.
#
# `require` — ALL must match the file. The enablement itself.
# `forbid` — NONE may match. For enablements that can be defeated without being deleted.
#
# Longest prefix wins, so kube_velero_schedule_ is resolved before kube_.
# =============================================================================================
KPS = "platform/monitoring/kube-prometheus-stack.yaml"
SCRAPE_OWNERS: list[dict] = [
{
"owner": "kube-prometheus-stack: bundled node-exporter DaemonSet + its ServiceMonitor",
"prefixes": ("node_",),
"file": KPS,
# The subchart block must be present and configured here (it is where the exporter's
# resources live). The chart default is enabled:true, so the regression to guard is a
# DISABLE, not a missing enable.
"require": [r"^\s*prometheus-node-exporter:\s*$"],
"forbid": [r"prometheus-node-exporter:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
"why": "node_* powers all seven platform-health.nodes rules including the two IO-pressure "
"rules that exist because of prod-1's etcd WAL starvation",
},
{
"owner": "kube-prometheus-stack: the kubelet ServiceMonitor (cAdvisor + volume stats)",
"prefixes": ("kubelet_",),
"file": KPS,
# Chart default kubelet.enabled:true. There is no explicit enable to require, so the only
# honest assertion is that it has not been turned off the way kubeProxy/kubeEtcd were.
"require": [r"kubeProxy:"], # sentinel: the exporter-toggle block is still authored here
"forbid": [r"\n\s*kubelet:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
"why": "kubelet_volume_stats_* powers MonitoringVolumeFillingUp and all three "
"pvc-capacity-alerts rules",
},
{
"owner": "kube-prometheus-stack: kube-state-metrics custom-resource-state for Velero Schedules",
"prefixes": ("kube_velero_schedule_",),
"file": KPS,
"require": [
r"customResourceState:\s*\n\s*enabled:\s*true",
r"metricNamePrefix:\s*kube_velero_schedule",
# The RBAC is MANDATORY, not cosmetic: without velero.io read access KSM emits the
# custom-resource-state metrics as an EMPTY SET and logs a forbidden error — silent,
# and the inert-alert failure mode all over again (measured; see the block's header).
r'apiGroups:\s*\["velero\.io"\]',
r'resources:\s*\["schedules"\]',
],
"forbid": [r"customResourceState:\s*\n\s*enabled:\s*false"],
"why": "kube_velero_schedule_* is the only thing that can see a Schedule that has NEVER "
"backed up (VeleroScheduleNeverBackedUp / SchedulePaused / NoVolumeDataBackup); "
"Velero itself publishes nothing until a backup is attempted",
},
{
"owner": "kube-prometheus-stack: bundled kube-state-metrics",
"prefixes": ("kube_",),
"file": KPS,
"require": [r"^\s*kube-state-metrics:\s*$"],
"forbid": [r"kube-state-metrics:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
# kube_node_labels is a SPECIAL CASE and is checked on its own below: KSM exposes no
# custom node labels by default, so the metric does not exist unless metricLabelsAllowlist
# names the label. The prefix claim alone would pass while VPNTunnelDown's join matched
# nothing.
"why": "kube_* powers the Keycloak, KubeVirt-crashloop, CNPG-colocation and Velero-schedule "
"rules",
},
{
"owner": "kube-state-metrics metricLabelsAllowlist (the egress-gateway node label)",
"exact": ("kube_node_labels",),
"file": KPS,
"require": [
r"metricLabelsAllowlist:",
r"nodes=\[homelab\.keithjackson\.dev/egress-gateway\]",
],
"forbid": [],
"why": "KSM exposes NO custom labels by default. Without the allowlist entry "
"kube_node_labels carries no egress-gateway label and VPNTunnelDown's join silently "
"matches nothing — a rule that is present, evaluating, and structurally inert",
},
{
"owner": "CNPG: enablePodMonitor on the Cluster CRs (the operator renders one PodMonitor each)",
"prefixes": ("cnpg_",),
"file": "platform/keycloak/postgres-cluster.yaml",
"require": [r"^\s*enablePodMonitor:\s*true\s*$"],
"forbid": [],
"why": "cnpg_* powers all six platform-health.cnpg rules, including the WAL-archive backlog "
"and no-recoverability-point rules that guard R-STATE-03",
},
{
"owner": "Flux controllers: platform/monitoring/flux-podmonitor.yaml (merged a79693d)",
"prefixes": ("gotk_",),
"file": "platform/monitoring/flux-podmonitor.yaml",
"require": [r"kind:\s*PodMonitor", r"port:\s*http-prom"],
"forbid": [],
"why": "gotk_* powers FluxReconciliationFailure, FluxResourceSuspended and the three "
"automated-update rules — on a platform whose entire delivery model is Flux",
},
{
"owner": "cert-manager: prometheus.servicemonitor.enabled in the HelmRelease",
"prefixes": ("certmanager_",),
"file": "platform/cert-manager/helmrelease.yaml",
"require": [r"prometheus:\s*\n\s*servicemonitor:\s*\n\s*enabled:\s*true"],
# The chart calls `fail` when both monitor kinds are enabled, which would break the whole
# HelmRelease rather than just the scrape. Measured against v1.21.2.
"forbid": [r"podmonitor:\s*\n\s*enabled:\s*true"],
"why": "certmanager_* powers CertificateExpiringSoon / ExpiryImminent / NotReady — the only "
"warning that an ACME DNS-01 renewal of the live wildcard has quietly failed",
},
{
"owner": "Longhorn: metrics.serviceMonitor.enabled in the HelmRelease",
"prefixes": ("longhorn_",),
"file": "platform/longhorn/helmrelease.yaml",
"require": [r"metrics:\s*\n\s*serviceMonitor:\s*\n\s*enabled:\s*true"],
"forbid": [],
"why": "longhorn_* powers the four storage rules on a cluster whose recurring failure mode "
"is a disk",
# The ServiceMonitor ALONE is not sufficient here — see the companion assertion below.
"companion": {
"file": "platform/longhorn/kustomization.yaml",
"require": [r"^\s*-\s*networkpolicy-metrics\.yaml\s*$"],
"why": "Longhorn 1.12.1 ships its own effectively-unconditional NetworkPolicies "
"(guarded on networkPolicies.restrictInternalTraffic, default true — NOT on "
"networkPolicies.enabled) whose longhorn-manager policy admits only other "
"Longhorn components in the same namespace. MEASURED on prod: with the "
"ServiceMonitor applied and no allowance, all three targets were DISCOVERED and "
"every scrape failed with 'context deadline exceeded' after 10.0s — a drop, not "
"a refusal — and the TSDB stayed empty. So without networkpolicy-metrics.yaml "
"the four storage alerts are inert even though the monitor exists",
},
},
{
"owner": "KubeVirt/CDI: platform/kubevirt/servicemonitor.yaml",
"prefixes": ("kubevirt_",),
"file": "platform/kubevirt/servicemonitor.yaml",
"require": [r"kind:\s*ServiceMonitor", r"port:\s*metrics"],
"forbid": [],
"why": "kubevirt_* powers the four platform-health.kubevirt rules (note the CDI rules use "
"the kubevirt_cdi_* names, not bare cdi_*)",
},
{
"owner": "Velero: metrics.serviceMonitor.enabled in the HelmRelease",
"prefixes": ("velero_",),
"file": "platform/velero/helmrelease.yaml",
"require": [r"metrics:\s*\n\s*serviceMonitor:\s*\n\s*enabled:\s*true"],
"forbid": [],
"why": "velero_* powers VeleroBackupFailure / PartialFailure / NoRecentBackup — the rules "
"that failed to notice prod running with zero volume backups",
},
]
# `up` is produced by EVERY scrape target, so no owner can meaningfully claim it and assertion (1)
# is structurally blind to `up{job="X"}` naming a job nothing produces. That axis is assertion (4).
UNIVERSAL = {"up"}
# =============================================================================================
# REGISTER 1 — KNOWN_UNWIRED_METRICS. Shrink-only. A metric here is queried by an alert and
# claimed by NOTHING, so that alert is inert and the platform is blind to what it describes.
# Removing an entry means wiring the scrape. An entry that stops being referenced by any alert,
# or that gains an owner, FAILS as stale.
# =============================================================================================
KNOWN_UNWIRED_METRICS: dict[str, str] = {
"hubble_drop_total":
"HubbleDeniedFlowSpike — the platform has NO policy-drop / NetworkPolicy-regression "
"visibility. Unlike Longhorn and cert-manager (whose charts already serve /metrics and "
"only lacked a ServiceMonitor) Hubble publishes NOTHING to scrape until it is switched on "
"in Cilium's own config: platform/cilium/helmrelease.yaml sets hubble.{enabled,relay,ui} "
"but not hubble.metrics.enabled, and not prometheus.enabled for the agent. Turning them on "
"OPENS PORT 9965 ON EVERY cilium-agent AND ROLLS THE CNI DaemonSet across all three nodes "
"— the single most dangerous rollout on this cluster (the 2026-09-01 k8sServiceHost outage; "
"cilium was moved to its own Flux Kustomization precisely to isolate that blast radius) — "
"and requires a matching edit to roles/cilium/templates/cilium-values.yaml.j2 in the same "
"commit or a rebuilt node's bootstrap CNI config silently disagrees with the reconciled "
"one. Deliberately DEFERRED to its own change with a one-node canary; see the rule's "
"header in platform/monitoring/alerts.yaml.",
}
# =============================================================================================
# REGISTER 2 — KNOWN_UNWIRED_JOBS. Shrink-only. Each is a `up{job="X"}` selector whose monitor
# names an endpoint port that the Service it selects does not define, so the monitor discovers
# ZERO targets, no `up{job="X"}` series exists, and `up{job="X"} == 0` can never fire. This is the
# nastiest variant of the class: `up` exists in abundance, so nothing about the metric name looks
# wrong, and `PrometheusTargetsMissing` cannot see it either — a target that was never discovered
# has no up series to be 0.
#
# All twelve are the apps/* ServiceDefinition SLI pattern: the rendered ServiceMonitor names
# `port: metrics`, and no app Service defines a port of that name (verified: zero occurrences of
# `name: metrics` across all twelve manifests). VERIFIED on prod 2026-09-13: `count by (job) (up)`
# listed 20 jobs and none of these twelve appeared; all twelve `<app>-down` rules returned no
# series. Closing this needs a Prometheus exporter per app — Plex, Pi-hole, Jackett, qBittorrent,
# Sonarr, Radarr and Seerr do not speak Prometheus natively — so it is its own piece of work, and
# for keycloak specifically the job labels that DO exist are `keycloak-service`/`keycloak-discovery`
# (platform/keycloak/servicemonitor.yaml), so that one is a rename away rather than an exporter.
# =============================================================================================
KNOWN_UNWIRED_JOBS: dict[str, str] = {
"gitea": "apps/wave-0/gitea.yaml — Service defines only port-3000",
"jackett": "apps/wave-0/jackett.yaml — no exporter",
"pihole": "apps/wave-0/pihole.yaml — needs pihole-exporter",
"plex": "apps/wave-0/plex.yaml — no native Prometheus endpoint",
"qbittorrent": "apps/wave-0/qbittorrent.yaml — needs qbittorrent-exporter",
"vm-demo": "apps/wave-0/vm-demo.yaml — a KubeVirt demo VM, no exporter",
"keycloak": "apps/wave-1/keycloak.yaml — the live jobs are keycloak-service / "
"keycloak-discovery (platform/keycloak/servicemonitor.yaml); this SLI rule names "
"neither, so it is a rename away from being live rather than an exporter away",
"nextcloud": "apps/wave-1/nextcloud.yaml — needs nextcloud-exporter (serverinfo token)",
"radarr": "apps/wave-1/radarr.yaml — needs exportarr",
"semaphore": "apps/wave-1/semaphore.yaml — no exporter",
"sonarr": "apps/wave-1/sonarr.yaml — needs exportarr",
"seerr": "apps/wave-2/seerr.yaml — no exporter",
}
# =============================================================================================
# (0) PARSE EVERY ALERT RULE
# =============================================================================================
# PromQL metric-name approximation. Documented as a gap in the header: string literals,
# label-matcher bodies and aggregation/vector-matching label lists are stripped first, then every
# remaining bare identifier that is not a PromQL keyword or function is treated as a metric
# selector. Assertion (0b) fails loudly if this stops finding metrics at all, which is the way an
# approximate parser silently stops guarding.
PROMQL_KEYWORDS = set("""
abs absent absent_over_time ceil changes clamp clamp_max clamp_min day_of_month day_of_week
day_of_year days_in_month delta deriv exp floor histogram_quantile histogram_count histogram_sum
histogram_fraction histogram_stddev histogram_stdvar holt_winters hour idelta increase irate
label_join label_replace ln log2 log10 minute month predict_linear rate resets round scalar sgn
sort sort_desc sort_by_label sort_by_label_desc sqrt time timestamp vector year avg count
count_values group max min quantile stddev stdvar sum topk bottomk limitk limit_ratio
last_over_time avg_over_time min_over_time max_over_time sum_over_time count_over_time
quantile_over_time stddev_over_time stdvar_over_time present_over_time mad_over_time
double_exponential_smoothing info ts_of_max_over_time ts_of_min_over_time ts_of_last_over_time
by without on ignoring group_left group_right offset bool and or unless if atan2 start end
inf nan NaN Inf
""".split())
IDENT_RE = re.compile(r"(?<![\w:.])([a-zA-Z_:][a-zA-Z0-9_:]*)")
UP_JOB_RE = re.compile(r"""\bup\s*\{[^}]*\bjob\s*=\s*["']([^"']+)["']""")
def metric_names(expr: str) -> list[str]:
e = re.sub(r'"(?:[^"\\]|\\.)*"', '""', expr)
e = re.sub(r"'(?:[^'\\]|\\.)*'", "''", e)
e = re.sub(r"`(?:[^`\\]|\\.)*`", "``", e)
e = re.sub(r"\{[^{}]*\}", " ", e) # label matchers
for _ in range(6): # nested aggregations
e = re.sub(r"\b(?:by|without|on|ignoring|group_left|group_right)\s*\([^()]*\)", " ", e)
out, seen = [], set()
for m in IDENT_RE.finditer(e):
n = m.group(1)
if n in PROMQL_KEYWORDS or re.fullmatch(r"\d+[smhdwy]?", n) or n in seen:
continue
seen.add(n)
out.append(n)
return out
def is_golden_fixture(p: Path) -> bool:
"""conformance/tests/**/golden/** holds EXPECTED-OUTPUT fixtures, not deployed manifests.
conformance/tests/l1-rendering/egress-class-render/golden/*.expected.yaml each contain a
rendered `<app>-down` PrometheusRule. They are the assertion data of another test, never
reconciled into a cluster, so auditing their scrape targets would be auditing a copy — the
same "reference, not declaration" distinction ci/touched-requirements.sh draws for ci/tests/
fixture paths. The templates they are rendered FROM are audited through apps/ and catalog/.
"""
parts = p.parts
return "conformance" in parts and "golden" in parts
rule_files = sorted(
p for p in ROOT.rglob("*.yaml")
if ".git" not in p.parts and not is_golden_fixture(p) and "kind: PrometheusRule" in read(p)
)
check(len(rule_files) > 0,
"no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "
"this check would silently audit nothing")
# metric -> [ "<relpath>:<alert>", ... ] ; and the same shape for up{job="X"} selectors
refs: dict[str, list[str]] = {}
job_refs: dict[str, list[str]] = {}
n_rules = 0
for f in rule_files:
rel = str(f.relative_to(ROOT))
try:
docs = list(yaml.safe_load_all(read(f)))
except yaml.YAMLError as exc:
check(False, f"{rel} is not parseable YAML ({exc}); every alert in it is unaudited")
continue
for d in docs:
if not isinstance(d, dict) or d.get("kind") != "PrometheusRule":
continue
for grp in (d.get("spec") or {}).get("groups") or []:
for r in grp.get("rules") or []:
if "alert" not in r:
continue # recording rules: not this check's subject
n_rules += 1
site = f"{rel}:{r['alert']}"
expr = str(r.get("expr", ""))
for n in metric_names(expr):
refs.setdefault(n, []).append(site)
for j in UP_JOB_RE.findall(expr):
job_refs.setdefault(j, []).append(site)
check(n_rules > 0,
"no alerting rule parsed out of any PrometheusRule — the parse contract broke and this "
"check would pass vacuously while every alert's metric went unaudited")
check(len(refs) >= 20,
f"only {len(refs)} distinct metric name(s) parsed out of {n_rules} alerting rule(s). The "
f"PromQL approximation has stopped matching — which is exactly how an approximate parser "
f"turns into a guard that passes while checking nothing. Fix metric_names() or replace it "
f"with a real parser")
# =============================================================================================
# (1)(2) RESOLVE EVERY METRIC TO AN OWNER, AND PROVE THE OWNER'S CLAIM AGAINST THE REPO
# =============================================================================================
def resolve(metric: str) -> dict | None:
"""Longest-prefix / exact-name match against SCRAPE_OWNERS."""
best, best_len = None, -1
for o in SCRAPE_OWNERS:
if metric in o.get("exact", ()): # exact names win outright
return o
for pfx in o.get("prefixes", ()):
if metric.startswith(pfx) and len(pfx) > best_len:
best, best_len = o, len(pfx)
return best
# Every owner's FILE must exist, whether or not a metric currently lands on it: an owner pointing
# at a deleted file is a claim nothing backs.
for o in SCRAPE_OWNERS:
check((ROOT / o["file"]).is_file(),
f"SCRAPE_OWNERS names {o['file']} as the scrape owner for "
f"{o.get('prefixes') or o.get('exact')}, and that file does not exist in this repo — the "
f"claim is unbacked and the alerts that depend on it are inert")
claimed: dict[str, dict] = {}
unowned: list[str] = []
for metric in sorted(refs):
if metric in UNIVERSAL:
continue
o = resolve(metric)
if o is None:
if metric in KNOWN_UNWIRED_METRICS:
continue # declared gap; ratcheted below
unowned.append(metric)
check(False,
f"alert metric `{metric}` (queried by {', '.join(sorted(set(refs[metric])))}) is "
f"claimed by NO entry in SCRAPE_OWNERS. Nothing in this repo configures a scrape "
f"that produces it, so every rule above is STRUCTURALLY INERT: it evaluates to no "
f"vector, stays silent forever, and is indistinguishable from a healthy result. "
f"Either wire the scrape and add the owner entry, or — if it cannot be wired yet — "
f"declare it in KNOWN_UNWIRED_METRICS with the reason, so the gap is reported "
f"instead of silent")
continue
check(metric not in KNOWN_UNWIRED_METRICS,
f"`{metric}` is listed in KNOWN_UNWIRED_METRICS but is now claimed by "
f"{o['owner']} — the register is shrink-only: delete the entry so the next real gap is "
f"the one that shows up")
claimed[metric] = o
# The owners that are actually load-bearing right now get their declarations verified.
for o in SCRAPE_OWNERS:
if o not in claimed.values():
continue # nothing queries it today; nothing to guard
body = read(ROOT / o["file"])
for rx in o.get("require", []):
check(re.search(rx, body, re.M) is not None,
f"{o['file']} no longer matches /{rx}/, which is what makes {o['owner']} scrape. "
f"Removing it re-inerts every alert on "
f"{o.get('prefixes') or o.get('exact')} — {o['why']}")
for rx in o.get("forbid", []):
check(re.search(rx, body, re.M) is None,
f"{o['file']} matches the FORBIDDEN pattern /{rx}/, which defeats {o['owner']} "
f"even though the enablement is still authored. {o['why']}")
comp = o.get("companion")
if comp:
cbody = read(ROOT / comp["file"])
check((ROOT / comp["file"]).is_file(),
f"{comp['file']} is missing — {comp['why']}")
for rx in comp.get("require", []):
check(re.search(rx, cbody, re.M) is not None,
f"{comp['file']} no longer matches /{rx}/. The monitor alone is NOT enough here: "
f"{comp['why']}")
# The Longhorn NetworkPolicy must actually grant the path it exists to grant, and no more: the
# whole point of splitting it out instead of flipping networkPolicies.restrictInternalTraffic off.
NP = ROOT / "platform/longhorn/networkpolicy-metrics.yaml"
if any(m.startswith("longhorn_") for m in refs):
np = read(NP)
check(np != "", f"{NP.relative_to(ROOT)} is missing; the Longhorn scrape is dropped and its "
f"four alerts are inert")
check(re.search(r"kind:\s*NetworkPolicy", np) is not None,
"platform/longhorn/networkpolicy-metrics.yaml is not a NetworkPolicy")
check(re.search(r"app:\s*longhorn-manager", np) is not None,
"the Longhorn metrics NetworkPolicy does not select app=longhorn-manager, so it does not "
"open the pods that serve /metrics")
check(re.search(r"kubernetes\.io/metadata\.name:\s*monitoring", np) is not None,
"the Longhorn metrics NetworkPolicy does not admit namespace monitoring, so Prometheus "
"still cannot reach longhorn-manager and the storage alerts stay inert")
check(re.search(r"port:\s*9500", np) is not None,
"the Longhorn metrics NetworkPolicy does not name port 9500 (longhorn-manager's `manager` "
"port, which serves /metrics)")
# LEAST PRIVILEGE (R-SEC-02): ingress only, one port, and NOT the webhook/recovery ports.
check(re.search(r"policyTypes:\s*\n\s*-\s*Ingress\s*$", np, re.M) is not None
and "Egress" not in np,
"the Longhorn metrics NetworkPolicy is not Ingress-only — it must grant exactly the "
"scrape path and nothing else (R-SEC-02)")
# Comments are stripped first: the file's header deliberately NAMES 9502/9503 to record that
# they stay closed, and a substring search over prose would fail on the documentation of the
# very property it is asserting.
np_spec = "\n".join(re.sub(r"#.*$", "", ln) for ln in np.splitlines())
for closed in ("9502", "9503"):
check(closed not in np_spec,
f"the Longhorn metrics NetworkPolicy names port {closed} (admission webhook / "
f"recovery backend). Only 9500 serves metrics; opening the others to ns monitoring "
f"is reach this observability path does not need (R-SEC-02)")
# stale KNOWN_UNWIRED_METRICS entries
for m in sorted(KNOWN_UNWIRED_METRICS):
check(m in refs,
f"KNOWN_UNWIRED_METRICS declares `{m}`, which NO alert rule in this repo queries any "
f"more — a stale exemption hides the next real gap; delete the entry")
# =============================================================================================
# (4) up{job="X"} — the axis assertion (1) is structurally blind to.
#
# Scoped to files where THIS REPO authors both the Service and the ServiceMonitor (the apps/*
# ServiceDefinition pattern), because only then is the port name inside the tree. A monitor whose
# Service comes from a Helm chart cannot be audited statically and is out of scope (header gap).
# =============================================================================================
def yaml_docs(path: Path) -> list[dict]:
try:
return [d for d in yaml.safe_load_all(read(path)) if isinstance(d, dict)]
except yaml.YAMLError:
return []
# job name -> (relpath, monitor endpoint ports, ports the co-authored Services define)
same_file_monitors: dict[str, tuple[str, set[str], set[str]]] = {}
for p in sorted(ROOT.rglob("*.yaml")):
if ".git" in p.parts:
continue
txt = read(p)
if "kind: ServiceMonitor" not in txt or "kind: Service" not in txt:
continue
docs = yaml_docs(p)
monitors = [d for d in docs if d.get("kind") == "ServiceMonitor"]
services = [d for d in docs if d.get("kind") == "Service"]
if not monitors or not services:
continue
svc_ports: set[str] = set()
for s in services:
for port in (s.get("spec") or {}).get("ports") or []:
if isinstance(port, dict) and port.get("name"):
svc_ports.add(str(port["name"]))
for m in monitors:
ep_ports = {str(e["port"]) for e in (m.get("spec") or {}).get("endpoints") or []
if isinstance(e, dict) and e.get("port")}
name = (m.get("metadata") or {}).get("name")
if name:
same_file_monitors[str(name)] = (str(p.relative_to(ROOT)), ep_ports, svc_ports)
for job in sorted(job_refs):
sites = ", ".join(sorted(set(job_refs[job])))
mon = same_file_monitors.get(job)
if mon is None:
# No same-file Service+ServiceMonitor pair is named for this job. It may well be produced
# by a platform monitor against a chart Service (oauth2-proxy, kubevirt, velero, flux all
# are), which this check cannot see — so this is NOT a failure, only unauditable.
continue
rel, ep_ports, svc_ports = mon
missing = sorted(ep_ports - svc_ports)
if missing:
check(job in KNOWN_UNWIRED_JOBS,
f"`up{{job=\"{job}\"}}` ({sites}) can never produce a series: {rel}'s ServiceMonitor "
f"scrapes port(s) {missing}, and the Service(s) in that same file define only "
f"{sorted(svc_ports) or ['(no named ports)']}. The monitor therefore discovers ZERO "
f"targets, there is no `up{{job=\"{job}\"}}` to compare against 0, and the rule is "
f"structurally inert — note PrometheusTargetsMissing cannot see this either, because "
f"a target that was never discovered has no up series to be 0. Give the Service a "
f"named port the workload actually serves metrics on (and an exporter if it has "
f"none), or declare it in KNOWN_UNWIRED_JOBS")
else:
check(job not in KNOWN_UNWIRED_JOBS,
f"`{job}` is listed in KNOWN_UNWIRED_JOBS but {rel}'s ServiceMonitor port(s) "
f"{sorted(ep_ports)} now all exist on its Service — the register is shrink-only: "
f"delete the entry")
# stale KNOWN_UNWIRED_JOBS entries
for job in sorted(KNOWN_UNWIRED_JOBS):
check(job in job_refs,
f"KNOWN_UNWIRED_JOBS declares `{job}`, which no alert rule selects via up{{job=...}} any "
f"more — a stale exemption hides the next real gap; delete the entry")
# =============================================================================================
# (5) THE CHECK IS LOAD-BEARING. Asserted against the DECLARATIVE mechanism, in both halves —
# neither alone is sufficient. The per-test hand-written ci/l0.sh stage is deliberately NOT used
# (whole-repo-invariants-wired FAILS the gate if one reappears); the property is DECLARED here and
# HONOURED by ci/l0.sh's --include-scope union.
# =============================================================================================
SCOPE_DECL_RE = r"^\s*conformance\.home\.k3s/scope:\s*[\"']?whole-repo"
SELF = read(ROOT / "conformance" / "tests" / "l0-static" / "alert-metrics-have-scrape-targets"
/ "chainsaw-test.yaml")
check(re.search(SCOPE_DECL_RE, SELF, re.M) is not None,
"alert-metrics-have-scrape-targets/chainsaw-test.yaml does not declare "
"conformance.home.k3s/scope: whole-repo, so the merge gate selects this check only when a "
"change happens to touch R-OBS-04. ci/touched-requirements.sh maps apps/* to NOTHING and "
"platform/monitoring/* to R-OBS-01/R-OBS-02 — so adding an alert under apps/ or flipping a "
"chart's serviceMonitor.enabled back to false would be filtered out of exactly the change "
"that re-inerts the alerts")
L0 = read(ROOT / "ci" / "l0.sh")
GATE = read(ROOT / "ci" / "gate.sh")
check("--include-scope" in L0,
"ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is a "
"purely requirement-scoped run again and the scope annotation above is inert decoration")
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) is not None
and re.search(r"fail\).*BLOCK=1", GATE) is not None,
"ci/gate.sh does not set the merge block when the l0 stage fails")
# ======================================================================== report
owners_used = {id(o): o for o in claimed.values()}
print(f"Alert metric inventory: {n_rules} alerting rule(s) across {len(rule_files)} "
f"PrometheusRule file(s) -> {len(refs)} distinct metric name(s)")
print(f" claimed by a declared scrape owner: {len(claimed)}/{len(refs) - len(UNIVERSAL & set(refs))}"
f" owners exercised: {len(owners_used)}/{len(SCRAPE_OWNERS)}")
for o in SCRAPE_OWNERS:
ms = sorted(m for m, oo in claimed.items() if oo is o)
mark = " " if ms else "· "
print(f" {mark}{o['file']:<46} {len(ms):>2} metric(s) {o['owner'][:64]}")
print(f"\n DECLARED GAPS (shrink-only registers):")
print(f" KNOWN_UNWIRED_METRICS: {len(KNOWN_UNWIRED_METRICS)} — "
f"{', '.join(sorted(KNOWN_UNWIRED_METRICS))}")
print(f" KNOWN_UNWIRED_JOBS: {len(KNOWN_UNWIRED_JOBS)} inert up{{job=...}} SLI rule(s) — "
f"{', '.join(sorted(KNOWN_UNWIRED_JOBS))}")
if unowned:
print(f"\n UNCLAIMED (inert, undeclared): {', '.join(unowned)}", file=sys.stderr)
print()
if not FAILURES:
print(f"check OK: alert-metrics-have-scrape-targets — {CHECKS} assertions (R-OBS-04)")
raise SystemExit(0)
print(f"CHECK FAIL (alert-metrics-have-scrape-targets): {len(FAILURES)} failed, "
f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
for f in FAILURES:
print(f" FAIL {f}", file=sys.stderr)
raise SystemExit(1)
PY
@@ -0,0 +1,95 @@
# =============================================================================
# L0 STATIC — the alert rules are UNIT-TESTED, by a pinned engine, and the gaps are declared
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# R-OBS-04 ("the platform alerts on its own health") is implemented by PrometheusRule CRs, and
# this repo has shipped both ways they fail SILENTLY:
# * PERMANENTLY FIRING — the CNPG primaries alert (fixed 2026-09-12), VPNTunnelDown before it;
# * STRUCTURALLY INERT — VeleroNoRecentBackup, whose `time() - <absent series>` yields no
# series, so a schedule that had NEVER backed up could not raise it. Prod ran with zero
# volume backups and nothing alerted, masked by a resources-only schedule completing
# cleanly every night (2026-09-12).
# An inert rule and a correct rule are indistinguishable while the platform is healthy. Only
# `promtool test rules` separates them, by constructing the bad state on purpose, offline.
#
# The unit tests were then written — and RUN BY NOTHING. promtool was not in the pinned
# toolchain, no gate stage evaluated the files, and the two harnesses that grew up independently
# each fetched an engine off the network mid-run if promtool was missing (a GitHub release
# download in one, a prometheus container image in the other). Coverage that looks present and
# is enforced nowhere is the failure mode this test ends.
#
# GIVEN the repository as authored,
# WHEN the ONE rule-unit harness (platform/monitoring/tests/run.sh, which flattens the
# COMMITTED PrometheusRule CRs into the rules file promtool wants — never a copy) is run
# over every test file in the repo with the PINNED promtool,
# THEN promtool is pinned in BOTH versions.env files in lockstep at the Prometheus version prod
# actually runs, baked into the image and installed on the CI runner; the promtool in use
# reports exactly that version; no harness acquires an engine at run time; there is
# exactly ONE harness; every rule-unit test file PASSES; the harness is proven able to
# FAIL by a mutated synthetic test on every run; and every alert rule with NO unit test is
# ENUMERATED in a shrink-only register, so the gap is reported rather than hidden.
#
# MISSING TESTS: RATCHET, NOT SKIP, NOT BLANKET-FAIL. 48 of 55 alert rules have no unit test
# today. Failing on all of them would make the gate red until ~48 tests exist (so it would be
# disabled, and a disabled gate protects nothing); merely reporting is precisely how the CNPG
# tests came to be enforced by nothing. So the gap is DECLARED in check.sh, printed with counts
# every run, and CANNOT GROW: a new PrometheusRule, or a new alert in an already-tested file,
# FAILS without a test, and a stale register entry FAILS too. A missing test is never an exit-3
# skip — "SKIPPED is not green" is preserved for the one thing that is genuinely an environment
# gap: promtool being absent.
#
# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (standard library + the harness
# it delegates to); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
# Chainsaw Test wraps the SAME script.
#
# NOT proven (named in check.sh's header): that an alert reaches a human (Alertmanager routing
# and SMTP are L3 / R-OBS-02), that a threshold is the right one, that a rule file is actually
# reconciled into a cluster, or that an author's assertions are meaningful — coverage of one
# alert is detected by its name appearing in a covering test file, and only the mutation
# self-test proves the harness CAN fail. R-OBS-04 is tagged for the static slice only: the rules
# the repo ships behave, under the engine prod runs, as their own unit tests say.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: alert-rules-unit-tested
labels:
conformance.home.k3s/R-OBS-04: "true"
conformance.home.k3s/R-TEST-02: "true"
annotations:
conformance.home.k3s/requirements: "R-OBS-04, R-TEST-02"
conformance.home.k3s/layer: "L0"
# WHOLE-REPO SCOPE — this invariant quantifies over every PrometheusRule in the tree
# (apps/wave-0..2, platform/cnpg, platform/kubevirt, platform/monitoring), and the register
# it maintains is a property of the WHOLE repo, not of any one path.
#
# WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh's coarse path map keys
# apps/* to NOTHING and platform/monitoring/* to R-OBS-01/02 — not R-OBS-04. So adding a new
# untested PrometheusRule under apps/, or editing one under platform/kubevirt/, attributes
# no requirement this test carries, and a requirement-scoped selection would filter the
# check out of exactly the change that adds the untested alert. The same blind spot that let
# `prod-baremetal-not-dev-surface` be filtered out of the run that gated its own breakage
# (DECISIONS-LOG 2026-09-12).
#
# Declaring the scope also makes an honest SKIP loud: lib/run_l1_offline.py turns a rc-3
# skip of a scope-declaring check into the runner's rc 3, and ci/l0.sh into the gate's rc 3.
# So a gate run on a toolchain WITHOUT promtool reports UNWIRED instead of green — which is
# the whole reason this check exists rather than another hand-written ci/l0.sh stage.
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Every committed alert rule that has unit tests passes them under the pinned promtool that
matches prod's Prometheus, exactly one harness runs them, nothing fetches a rule engine at
run time, the harness is proven able to fail, and every alert rule without a unit test is
enumerated in a register that can only shrink.
spec:
steps:
- name: run-every-alert-rule-unit-test-and-report-the-gaps
description: >-
Assert the promtool pin is in lockstep and baked, run the single rule-unit harness over
every test file in the repo, prove it can fail via a mutated synthetic test, and
enumerate every PrometheusRule with no unit tests against the shrink-only register.
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,550 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — alert-rules-unit-tested
# Serves: R-OBS-04 ("Platform alerts on its own health (nodes, storage, GitOps, certs,
# VPN)." — spec docs/10-observability-and-alerting.md)
# R-TEST-02 ("Tests are reproducible: deterministic, hermetic, single-command."
# — spec docs/14-testing-and-conformance.md)
#
# WHY THIS EXISTS — AN UNTESTED ALERT IS NOT A MONITORING CONTROL
# --------------------------------------------------------------
# This repo has shipped BOTH halves of the alerting failure space to prod:
# * a PERMANENTLY-FIRING `critical` (the CNPG alert fixed 2026-09-12; VPNTunnelDown before
# it), and
# * a STRUCTURALLY INERT rule — VeleroNoRecentBackup, whose `time() - <absent series>`
# yields no series, so a schedule that had NEVER backed up could not raise it. Prod ran
# with no volume backups at all and nothing alerted (2026-09-12).
# Neither is visible from the expression, from `kubectl apply`, or from a healthy cluster: an
# inert rule and a correct rule are indistinguishable while everything is fine. `promtool test
# rules` is the only tool that separates them, because the BAD state can be constructed on
# purpose. Unit tests for those rules were then WRITTEN — and evaluated by nothing: promtool
# was not in the pinned toolchain, no gate stage ran the files, and each of the two harnesses
# that existed acquired an engine off the network mid-run if one was missing. Coverage that
# looks present and is never enforced is the failure mode this check ends.
#
# GIVEN the repository as authored,
# WHEN the ONE rule-unit harness (platform/monitoring/tests/run.sh, which flattens the
# COMMITTED PrometheusRule CRs — never a copy) is asked which manifests are under test
# and then run over every test file in the repo,
# THEN (1) promtool is a PINNED member of the toolchain: PROMTOOL_VERSION is declared in BOTH
# infra/devcontainer/versions.env and ci/versions.env and the two AGREE (the repo's
# lockstep rule), the Containerfile takes it as a build ARG, installs it from the
# pinned prometheus release and records it for provenance, dev.sh passes it, and the
# CI runner installs it for the l0 stage;
# (2) the promtool ACTUALLY IN USE reports exactly that pinned version — a unit test is
# evidence only if the engine matches the one prod evaluates rules with;
# (3) NO harness acquires a rule engine at RUN TIME — neither `curl | tar -xz` of a
# GitHub release nor a `podman/docker run` of a prometheus image. Both existed on
# 2026-09-12 and both are removed; this is the regression guard;
# (4) there is exactly ONE harness. Two were written independently within an hour of
# each other; the one nothing runs is the one that rots;
# (5) every rule-unit test file in the repo PASSES, run through that harness (which
# `promtool check rules` first, so a non-parsing rules file cannot make unit tests
# pass vacuously by finding no rules at all);
# (6) the harness is NOT VACUOUS: a synthetic PrometheusRule + companion test is run
# through the SAME harness twice — intact (must PASS) and with one expectation
# mutated (must FAIL) — so a harness that has silently stopped evaluating anything
# is caught on every run, not only when someone thinks to try it;
# AND (7) the alert rules with NO unit tests are ENUMERATED and RATCHETED: each is declared
# below, printed with counts on every run, and the register can only SHRINK.
#
# WHY A MISSING TEST IS A RATCHET AND NOT AN OUTRIGHT FAIL (the decision, and its reasons)
# ---------------------------------------------------------------------------------------
# 48 of the repo's 55 alert rules have no unit test today. Three options were on the table:
# * FAIL on any missing test — correct in principle, but it makes the merge gate red for
# every change until ~48 unit tests exist, so in practice the gate gets disabled, and a
# disabled gate protects nothing.
# * merely REPORT — the gap prints and nothing holds it. That is exactly how the CNPG unit
# tests came to exist and be enforced by nothing; "it is in the output somewhere" is not
# enforcement.
# * RATCHET (chosen) — the gap is DECLARED IN THE REPO as a reviewable register, printed with
# counts on every gate run, and it CANNOT GROW: a new PrometheusRule, or a new alert in a
# file that already has tests, FAILS unless it ships a unit test. Removing an entry requires
# writing the test, and a STALE entry (file gone, or now covered) also FAILS, so the
# register cannot rot into fiction.
# "SKIPPED is not green" is honoured: a missing test is NEVER an exit-3 skip here. This check
# exits 0 only with the gap fully enumerated, and exits 1 the moment the debt grows, the
# register drifts, or a test that exists stops passing.
# The one exit-3 path is promtool itself being ABSENT — an environment gap, not a code property
# (the same convention as ci/l0.sh's kyverno handling), and this test DECLARES
# `conformance.home.k3s/scope: whole-repo`, which makes lib/run_l1_offline.py turn that skip
# into the runner's rc 3 and ci/l0.sh into the gate's rc 3: UNWIRED, never green. promtool
# PRESENT AT THE WRONG VERSION is a FAILURE, not a skip: that case still prints SUCCESS while
# evaluating the rules with an engine prod does not run.
#
# WHAT THIS DOES *NOT* PROVE (honest gaps):
# - It does not prove an alert reaches a human. promtool evaluates rules against synthetic
# series in-process; Alertmanager routing, SMTP delivery and inhibition are L3
# (tests/l3-fault-injection/alerting) and R-OBS-02, and are not claimed here.
# - It does not prove the THRESHOLDS are right, only that a rule behaves as its own unit
# test says.
# - It does not prove a rule file is RECONCILED into a cluster. An alert file no
# kustomization includes is a separate "looks applied and isn't" failure, not audited here.
# - Coverage of an individual alert is detected by its NAME appearing in a test file that
# covers its manifest. A test file that merely mentions a name without asserting anything
# counts as covered; only the mutation self-test (6) proves the harness CAN fail, not that
# every author wrote a meaningful assertion. Review still owns test quality.
# - It does not read the live cluster, so "the pin matches prod's Prometheus" is asserted
# against the repo's pin, which was established by a live read recorded in
# infra/devcontainer/versions.env (2026-09-12: v3.14.0-distroless).
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
HARNESS="${ALERT_RULE_HARNESS:-${ROOT}/platform/monitoring/tests/run.sh}"
# Honest SKIP (never green) if the thing under test is not present at all. Because this test
# declares scope: whole-repo, the runner turns this into rc 3 for the whole L0 stage — a
# PrometheusRule-less repo is a vanished tree, not a pass.
if ! grep -rqls --include='*.yaml' -e '^[[:space:]]*kind:[[:space:]]*PrometheusRule[[:space:]]*$' \
"${ROOT}/platform" "${ROOT}/apps" 2>/dev/null; then
echo "check SKIPPED: no PrometheusRule under platform/ or apps/ — nothing to test" >&2
exit 3
fi
# promtool resolution. PROMTOOL= overrides (the same override the harness honours); otherwise
# PATH. Absent -> every OFFLINE assertion still runs and the check exits 3, so a toolchain gap
# is reported as a gap and never as a pass.
PROMTOOL_BIN="${PROMTOOL:-$(command -v promtool || true)}"
if [ -n "${PROMTOOL_BIN}" ] && [ -x "${PROMTOOL_BIN}" ]; then
PROMTOOL_BIN="$(cd "$(dirname "${PROMTOOL_BIN}")" && pwd)/$(basename "${PROMTOOL_BIN}")"
else
PROMTOOL_BIN=""
fi
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" "${HARNESS}" "${PROMTOOL_BIN}" <<'PY'
"""Assert the alert-rule unit tests are pinned, enforced, non-vacuous, and that the alert
rules with no tests are enumerated and ratcheted. Standard library + the harness itself."""
from __future__ import annotations
import os
import re
import shutil
import subprocess
import sys
import tempfile
from collections import defaultdict
from pathlib import Path
ROOT = Path(sys.argv[1])
HARNESS = Path(sys.argv[2])
PROMTOOL = sys.argv[3]
sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree
FAILURES: list[str] = []
CHECKS = 0
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
# =============================================================================
# THE DECLARED REGISTER (assertion 7). Shrink-only; see the header for why this is a ratchet.
#
# UNTESTED_FILES — PrometheusRule files no test file covers AT ALL. The twelve apps/* entries
# are single `<app>-down` liveness rules generated from one ServiceDefinition template;
# platform/kubevirt/alerts.yaml (4 rules) is the substantial one and is where the next tests
# belong. A file listed here that later gains coverage, or stops carrying a PrometheusRule,
# FAILS as stale.
# =============================================================================
UNTESTED_FILES = {
"apps/wave-0/gitea.yaml",
"apps/wave-0/jackett.yaml",
"apps/wave-0/pihole.yaml",
"apps/wave-0/plex.yaml",
"apps/wave-0/qbittorrent.yaml",
"apps/wave-0/vm-demo.yaml",
"apps/wave-1/keycloak.yaml",
"apps/wave-1/nextcloud.yaml",
"apps/wave-1/radarr.yaml",
"apps/wave-1/semaphore.yaml",
"apps/wave-1/sonarr.yaml",
"apps/wave-2/seerr.yaml",
"platform/kubevirt/alerts.yaml",
}
# UNTESTED_ALERTS — individual alerts inside a file that IS covered by a test file, but whose
# own rule nothing asserts on. This is the half that stops a new alert being slipped into an
# already-"tested" file untested: anything not listed here must appear in a covering test.
# The monitoring block is large because backups.test.yaml (the repo's first suite) deliberately
# covers the Velero/backup rules only; pvc-capacity-alerts.yaml is flattened into the same
# rules file by the suite convention but has no assertions of its own yet.
UNTESTED_ALERTS = {
("platform/cnpg/alerts.yaml", "CNPGWALArchiveBacklog"),
("platform/cnpg/alerts.yaml", "CNPGNoRecoverabilityPoint"),
("platform/cnpg/alerts.yaml", "CNPGNoRecentBaseBackup"),
("platform/cnpg/alerts.yaml", "CNPGWALVolumeNotRecycling"),
("platform/cnpg/alerts.yaml", "CNPGMetricsCollectorErroring"),
("platform/monitoring/alerts.yaml", "NodeExporterDown"),
("platform/monitoring/alerts.yaml", "NodeMemoryPressureHigh"),
("platform/monitoring/alerts.yaml", "NodeDiskFillingUp"),
("platform/monitoring/alerts.yaml", "NodeCPUPressureHigh"),
("platform/monitoring/alerts.yaml", "NodeCPULoadHigh"),
("platform/monitoring/alerts.yaml", "NodeIOPressureHigh"),
("platform/monitoring/alerts.yaml", "NodeIOPressureCritical"),
("platform/monitoring/alerts.yaml", "LonghornVolumeDegraded"),
("platform/monitoring/alerts.yaml", "LonghornVolumeFaulted"),
("platform/monitoring/alerts.yaml", "LonghornVolumeCapacityHigh"),
("platform/monitoring/alerts.yaml", "LonghornNodeStorageHigh"),
("platform/monitoring/alerts.yaml", "FluxReconciliationFailure"),
("platform/monitoring/alerts.yaml", "FluxResourceSuspended"),
("platform/monitoring/alerts.yaml", "CertificateExpiringSoon"),
("platform/monitoring/alerts.yaml", "CertificateExpiryImminent"),
("platform/monitoring/alerts.yaml", "CertificateNotReady"),
("platform/monitoring/alerts.yaml", "VPNTunnelDown"),
("platform/monitoring/alerts.yaml", "HubbleDeniedFlowSpike"),
("platform/monitoring/alerts.yaml", "KeycloakDown"),
("platform/monitoring/alerts.yaml", "KeycloakDatabaseDown"),
("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerFailing"),
("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerStale"),
("platform/monitoring/alerts.yaml", "MonitoringVolumeFillingUp"),
("platform/monitoring/alerts.yaml", "PrometheusTargetsMissing"),
("platform/monitoring/alerts.yaml", "ImageUpdateAutomationFailing"),
("platform/monitoring/alerts.yaml", "ImagePolicyStalled"),
("platform/monitoring/alerts.yaml", "UpdateHealthGateFailed"),
("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeFillingUp"),
("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeAlmostFull"),
("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeFillingUpPredicted"),
}
# --------------------------------------------------------------- (1) the pins, in lockstep
def env_pin(rel: str, key: str) -> str | None:
f = ROOT / rel
if not f.is_file():
return None
m = re.search(rf'^{re.escape(key)}=["\']?([^"\'\s#]+)', f.read_text(), re.M)
return m.group(1) if m else None
DEV_PIN = env_pin("infra/devcontainer/versions.env", "PROMTOOL_VERSION")
CI_PIN = env_pin("ci/versions.env", "PROMTOOL_VERSION")
check(DEV_PIN is not None,
"infra/devcontainer/versions.env declares no PROMTOOL_VERSION — promtool is not part of the "
"baked toolchain, so the alert unit tests are evaluated by whatever the host happens to "
"have, or by nothing at all (which is where this repo was on 2026-09-12)")
check(CI_PIN is not None,
"ci/versions.env declares no PROMTOOL_VERSION — the gate's own version file does not pin the "
"rule engine, so CI and the devcontainer can silently diverge")
check(DEV_PIN is None or DEV_PIN == CI_PIN,
f"PROMTOOL_VERSION is NOT in lockstep: infra/devcontainer/versions.env={DEV_PIN!r} vs "
f"ci/versions.env={CI_PIN!r}. The repo's rule is that the conformance-tool versions in the "
f"two files stay identical, so a laptop and CI run the same engine")
CF = ROOT / "infra/devcontainer/Containerfile"
cf_text = CF.read_text() if CF.is_file() else ""
check(re.search(r"^ARG PROMTOOL_VERSION\s*$", cf_text, re.M),
"infra/devcontainer/Containerfile has no `ARG PROMTOOL_VERSION` — the pin is declared but "
"the image build does not consume it")
check("prometheus/prometheus/releases/download" in cf_text and "promtool" in cf_text,
"infra/devcontainer/Containerfile does not install promtool from the pinned prometheus "
"release tarball (promtool ships inside it)")
check('echo "promtool=${PROMTOOL_VERSION}"' in cf_text,
"infra/devcontainer/Containerfile does not record promtool in /etc/dev-toolchain-versions — "
"that provenance line is how an operator tells a stale image from a current one")
DEVSH = ROOT / "infra/devcontainer/dev.sh"
check(DEVSH.is_file() and "--build-arg PROMTOOL_VERSION=" in DEVSH.read_text(),
"infra/devcontainer/dev.sh does not pass --build-arg PROMTOOL_VERSION — the ARG would be "
"empty and the build would fetch a bogus URL")
INSTALL = ROOT / ".gitea/workflows/scripts/install-tools.sh"
if INSTALL.is_file():
itext = INSTALL.read_text()
check("PROMTOOL_VERSION" in itext
and re.search(r"^\s*l0\)[^\n]*install_promtool", itext, re.M),
".gitea/workflows/scripts/install-tools.sh does not install promtool for the l0 stage — "
"on the CI runner this check would exit 3 (UNWIRED) forever and the alert unit tests "
"would be enforced nowhere, which is the exact gap it exists to close")
# --------------------------------------------- (3)(4) one harness, and it fetches no engine
check(HARNESS.is_file() and os.access(HARNESS, os.X_OK),
f"the rule-unit harness is missing or not executable at {HARNESS.relative_to(ROOT)} — "
f"nothing would run the alert unit tests")
# Any script that invokes `promtool test rules` is a rule-unit harness. There must be exactly
# one: two harnesses were written independently on 2026-09-12 (a cnpg-only script and this one),
# and the one nothing runs is the one that rots out of sync with the rules.
harnesses: list[str] = []
fetchers: list[str] = []
FETCH_RE = re.compile(
r"(curl|wget)[^\n]*prometheus/prometheus/releases" # GitHub release download
r"|(podman|docker)\s+run[^\n]*prometheus", # container-image pull
re.I)
# Two exclusions, both the repo's existing "reference, not declaration" rule (see
# ci/touched-requirements.sh changed_diff() rules 2-3):
# * THIS CHECK'S OWN DIRECTORY. It names every forbidden pattern in order to forbid it, so
# scanning itself would make both guards permanently red — the self-referential trap.
# * THE PINNED TOOLCHAIN INSTALLERS. Downloading the pinned promtool is their JOB, at a
# reviewed seam (an image build, or the runner's tool-install step), from a version in
# versions.env. What is forbidden is a RULE-UNIT RUN acquiring its own engine: then the
# version, the network and the machine are decided silently at test time, and a missing
# toolchain becomes a green run instead of an exit 3.
SELF_DIR = "conformance/tests/l0-static/alert-rules-unit-tested/"
INSTALLERS = (".gitea/workflows/scripts/install-tools.sh", "infra/devcontainer/")
for p in sorted(ROOT.rglob("*.sh")):
rel = str(p.relative_to(ROOT))
if rel.startswith((".git/", "conformance/runs/", SELF_DIR)):
continue
try:
text = p.read_text()
except (OSError, UnicodeDecodeError):
continue
body = "\n".join(l for l in text.splitlines() if not l.lstrip().startswith("#"))
if "promtool" in body and re.search(r"test\s+rules", body):
harnesses.append(rel)
if FETCH_RE.search(body) and not rel.startswith(INSTALLERS):
fetchers.append(rel)
check(harnesses == [str(HARNESS.relative_to(ROOT))],
f"expected exactly ONE rule-unit harness ({HARNESS.relative_to(ROOT)}) but found "
f"{harnesses}. A second harness means two behaviours and two things to keep in step; the "
f"one the gate does not run is the one that rots. Fold it in instead")
check(not fetchers,
f"{fetchers} acquire a rule ENGINE at run time (a prometheus release download or a "
f"prometheus container image). promtool must come from the pinned toolchain "
f"(PROMTOOL_VERSION): a run-time fetch decides the version, the network and the machine "
f"silently at test time, makes the tests non-hermetic (R-TEST-02), and quietly turns 'the "
f"toolchain has no promtool' into a green run — which is how it stayed missing")
# ------------------------------------------------------------------ discover the alert files
# Alert files: tracked YAML carrying a PrometheusRule. conformance/ is excluded deliberately —
# the PrometheusRules under tests/l1-rendering/*/golden/ are EXPECTED-RENDER fixtures for the
# render checks, not rules any cluster evaluates. Everything a cluster actually gets lives under
# apps/, platform/, platform-config/ or clusters/.
def tracked_yaml() -> list[Path]:
out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z", "*.yaml", "*.yml"],
capture_output=True, text=True)
if out.returncode == 0 and out.stdout:
return [ROOT / p for p in out.stdout.split("\0") if p]
# No usable index (the gate container mounts a worktree whose .git points outside the
# mount). Fall back to a filesystem walk rather than auditing nothing.
return sorted(p for p in ROOT.rglob("*.y*ml") if ".git/" not in str(p))
KIND_RE = re.compile(r"^\s*kind:\s*PrometheusRule\s*$", re.M)
ALERT_RE = re.compile(r"^\s*-\s*alert:\s*(\S+)", re.M)
alert_files: dict[str, list[str]] = {}
for p in tracked_yaml():
rel = str(p.relative_to(ROOT))
if rel.startswith("conformance/") or not p.is_file():
continue
try:
text = p.read_text()
except (OSError, UnicodeDecodeError):
continue
if KIND_RE.search(text):
alert_files[rel] = ALERT_RE.findall(text)
check(len(alert_files) > 0,
"no PrometheusRule found outside conformance/ — discovery or the repo layout changed, and "
"this check would silently audit nothing")
# ------------------- ask the HARNESS which manifests are under test (never re-derive it here)
# Two implementations of "which rules does this test cover" would eventually disagree, and the
# disagreement shows up as a file the gate believes is tested and the harness never touches.
listing = subprocess.run(["bash", str(HARNESS), "--list"], capture_output=True, text=True,
cwd=str(ROOT))
check(listing.returncode == 0,
f"`{HARNESS.relative_to(ROOT)} --list` failed (rc={listing.returncode}): "
f"{(listing.stderr or listing.stdout).strip()[-600:]}")
covering: dict[str, list[str]] = defaultdict(list) # manifest -> [test files]
for line in listing.stdout.splitlines():
if "\t" in line:
tfile, manifest = line.split("\t", 1)
covering[manifest.strip()].append(tfile.strip())
for manifest in sorted(covering):
check(manifest in alert_files,
f"the harness reports {manifest} as under test, but it is not one of the "
f"PrometheusRule files found in the repo — the mapping and the tree disagree")
# --------------------------------------------------------- (2) the engine is the pinned one
version_line = ""
if PROMTOOL:
vp = subprocess.run([PROMTOOL, "--version"], capture_output=True, text=True)
out = (vp.stdout + vp.stderr).strip()
version_line = out.splitlines()[0] if out else ""
m = re.search(r"version\s+(\d+\.\d+\.\d+)", version_line)
found = m.group(1) if m else None
check(found is not None,
f"could not parse a version from `promtool --version`: {version_line!r}")
check(found is None or DEV_PIN is None or found == DEV_PIN,
f"promtool reports {found}, but the toolchain pins PROMTOOL_VERSION={DEV_PIN} (the "
f"Prometheus version prod runs). promtool IS the rule engine: a unit test green under "
f"one version and red under another is worse than no test. Run the gate in the pinned "
f"devcontainer (rebuild it: ./infra/devcontainer/dev.sh build), or set PROMTOOL= to a "
f"v{DEV_PIN} binary. A FAILURE, not a skip, because the wrong engine still prints "
f"SUCCESS")
# ------------------------------------------------- (5) every rule-unit test file must pass
if PROMTOOL:
env = dict(os.environ, PROMTOOL=PROMTOOL)
run = subprocess.run(["bash", str(HARNESS)], capture_output=True, text=True,
cwd=str(ROOT), env=env)
tailed = "\n ".join((run.stdout + run.stderr).strip().splitlines()[-40:])
check(run.returncode == 0,
f"`{HARNESS.relative_to(ROOT)}` FAILED (rc={run.returncode}) — the committed alert "
f"rules do not behave as their own unit tests say:\n {tailed}")
# ------------------------------------------- (6) the harness is proven non-vacuous
# A synthetic PrometheusRule + companion test through the SAME harness, outside the repo so
# discovery never picks it up. Intact must PASS; a mutated expectation must FAIL. Without
# this, a harness that stopped evaluating anything (an empty extraction, a rule file written
# under the wrong name) would report green forever — which is the disease being treated.
work = tempfile.mkdtemp(prefix="alert-rules-selftest.")
try:
(Path(work) / "synth.yaml").write_text("""apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: selftest
spec:
groups:
- name: selftest
rules:
- alert: SelfTestCanary
expr: selftest_metric > 0
for: 5m
annotations:
runbook_url: https://${BASE_DOMAIN}/runbooks/selftest
""")
test_yaml = """rule_files:
- generated.rules.yaml
evaluation_interval: 1m
tests:
- interval: 1m
name: canary fires only after `for` elapses
input_series:
- series: 'selftest_metric'
values: '1x10'
promql_expr_test:
- expr: 'ALERTS{alertname="SelfTestCanary",alertstate="firing"}'
eval_time: EVAL_TIME
exp_samples:
- labels: 'ALERTS{alertname="SelfTestCanary",alertstate="firing"}'
value: 1
"""
synth_test = Path(work) / "synth.promtool-test.yaml"
synth_test.write_text(test_yaml.replace("EVAL_TIME", "6m"))
good = subprocess.run(["bash", str(HARNESS), str(synth_test)],
capture_output=True, text=True, cwd=str(ROOT), env=env)
check(good.returncode == 0,
"self-test: the INTACT synthetic unit test did not pass, so the harness itself is "
"broken and every green result above is meaningless:\n "
+ "\n ".join((good.stdout + good.stderr).strip().splitlines()[-20:]))
# Mutation: assert firing at 2m, which `for: 5m` forbids.
synth_test.write_text(test_yaml.replace("EVAL_TIME", "2m"))
bad = subprocess.run(["bash", str(HARNESS), str(synth_test)],
capture_output=True, text=True, cwd=str(ROOT), env=env)
check(bad.returncode != 0,
"self-test: a MUTATED expectation (firing before `for:` elapsed) still PASSED — "
"the harness is not actually evaluating anything, so every green result above is "
"vacuous")
finally:
shutil.rmtree(work, ignore_errors=True)
# --------------------------------------------------------- (7) the gap inventory + ratchet
INVENTORY: list[str] = []
GAP_FILES: list[str] = []
alerts_total = covered_total = 0
for rel in sorted(alert_files):
names = alert_files[rel]
alerts_total += len(names)
check(len(names) > 0,
f"{rel} declares a PrometheusRule but no `alert:` rule was parsed out of it — the "
f"parser or the file shape changed, and any unit test over it would pass vacuously")
tests = covering.get(rel, [])
if not tests:
GAP_FILES.append(rel)
INVENTORY.append(f" {rel:<46} alerts={len(names):<3} tested=0/{len(names):<3} NO TEST FILE")
check(rel in UNTESTED_FILES,
f"{rel} carries {len(names)} alert rule(s) and NO test file covers it. A new "
f"PrometheusRule must ship promtool unit tests: this repo has shipped both a "
f"permanently-firing critical and a structurally inert alert, and neither was "
f"visible without them. Add `{rel[:-5]}.promtool-test.yaml` (companion convention) "
f"or a <component>/tests/*.test.yaml suite. If this is deliberate debt it must be "
f"DECLARED in UNTESTED_FILES in this check — the register is what keeps the gap "
f"visible instead of silent")
continue
check(rel not in UNTESTED_FILES,
f"{rel} is listed in UNTESTED_FILES but is now covered by {tests} — the register is "
f"shrink-only: delete the line so the next missing test is caught")
text = "".join((ROOT / t).read_text() for t in tests if (ROOT / t).is_file())
covered = [n for n in names if re.search(rf"\b{re.escape(n)}\b", text)]
covered_total += len(covered)
for n in names:
if n in covered:
check((rel, n) not in UNTESTED_ALERTS,
f"{rel}: alert {n} is listed in UNTESTED_ALERTS but IS asserted on now — "
f"remove the stale entry (the register is shrink-only)")
else:
check((rel, n) in UNTESTED_ALERTS,
f"{rel}: alert {n} has NO unit test in {tests}. This file is already covered, "
f"so a new alert in it must be tested too — otherwise a 'tested' alert file "
f"quietly grows untested rules")
INVENTORY.append(f" {rel:<46} alerts={len(names):<3} tested={len(covered)}/{len(names):<3} "
f"{','.join(tests)}")
# stale register entries
for rel in sorted(UNTESTED_FILES - set(alert_files)):
check(False, f"UNTESTED_FILES lists {rel}, which carries no PrometheusRule in this repo any "
f"more — a stale exemption hides the next real gap; delete the line")
for rel in sorted({f for f, _ in UNTESTED_ALERTS} - set(alert_files)):
check(False, f"UNTESTED_ALERTS lists alerts in {rel}, which carries no PrometheusRule in this "
f"repo any more — delete those lines")
# --------------------------------------------------------------------------------- report
print("Alert-rule unit-test inventory (PrometheusRule -> covering test file(s)):")
for line in INVENTORY:
print(line)
print(f"\nharness: {HARNESS.relative_to(ROOT)} promtool: {version_line or 'NOT FOUND'}")
print(f"pinned: PROMTOOL_VERSION={DEV_PIN} (devcontainer) / {CI_PIN} (ci) — prod runs Prometheus "
f"v{DEV_PIN}")
print(f"alert files: {len(alert_files)} covered by a test file: "
f"{len(alert_files) - len(GAP_FILES)} with NO test file: {len(GAP_FILES)}")
print(f"alert rules: {alerts_total} unit-tested: {covered_total} "
f"NOT unit-tested: {alerts_total - covered_total}")
if GAP_FILES:
print("\nDECLARED GAPS — alert files nothing tests at all (shrink-only register; each is a")
print("rule nothing has ever evaluated, NOT a rule known to work):")
for g in GAP_FILES:
print(f" - {g} ({len(alert_files[g])} alert(s))")
if UNTESTED_ALERTS:
print("\nDECLARED GAPS — individual alerts inside a covered file that nothing asserts on:")
for f, n in sorted(UNTESTED_ALERTS):
print(f" - {f} {n}")
print()
if not PROMTOOL and not FAILURES:
print("check SKIPPED (exit 3, NOT a pass): promtool is not on PATH, so NO alert rule was "
"parsed and NO unit test was evaluated.", file=sys.stderr)
print(f" promtool is pinned at v{DEV_PIN} in infra/devcontainer/versions.env + "
f"ci/versions.env.", file=sys.stderr)
print(" Run the gate in the pinned toolbox (./infra/devcontainer/dev.sh build, then "
"make -C ci gate), or set PROMTOOL=/path/to/promtool.", file=sys.stderr)
print(f" The {CHECKS} offline assertions above (pins, lockstep, one harness, no run-time "
f"engine fetch, the gap register) DID run and passed.", file=sys.stderr)
print(" This test declares scope: whole-repo, so the runner reports rc 3 — UNWIRED, never "
"green.", file=sys.stderr)
raise SystemExit(3)
if not FAILURES:
print(f"check OK: alert-rules-unit-tested — {CHECKS} assertions "
f"({covered_total}/{alerts_total} alert rules unit-tested, "
f"{len(GAP_FILES)} files with no test file, all declared) (R-OBS-04, R-TEST-02)")
raise SystemExit(0)
print(f"CHECK FAIL (alert-rules-unit-tested): {len(FAILURES)} failed, "
f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
for f in FAILURES:
print(f" FAIL {f}", file=sys.stderr)
raise SystemExit(1)
PY
@@ -0,0 +1,89 @@
# =============================================================================
# L0 STATIC — every alert's runbook_url resolves to a runbook that is actually served
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# Every platform-health alert rule in this repo carries
# `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations across four files, over 34
# distinct names. For the whole of phases 4-6 there was NO runbooks/ directory and NO HTTPRoute serving
# that path: an in-cluster probe of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404.
# So an alert reaching a human handed them a dead link at the one moment documentation is read, while the
# annotation made the problem look solved — the same green-surface-over-absent-mechanism class as the
# import Job that reported success while storing placeholder secrets, and the guard whose glob silently
# stopped matching.
#
# The link is served now (runbooks/, clusters/base/runbooks.yaml). This test is the guard that keeps it
# true, and it asserts BOTH halves of the claim because the regression is silent in both directions: an
# alert with a new runbook_url and no runbook links to a 404, AND a runbook committed without its
# configMapGenerator entry is in git, reviewed, merged — and still 404s in the cluster. "The file exists"
# and "the URL resolves" are different claims; only the second one helps an operator.
#
# GIVEN every `runbook_url` annotation on every PrometheusRule in the repo,
# WHEN each is parsed out of git and resolved against runbooks/,
# THEN (1) each URL has EXACTLY the shape the server resolves — https://${BASE_DOMAIN}/runbooks/<name>,
# the ${BASE_DOMAIN} token (never a literal hostname), no `.md`, no trailing slash, no query;
# (2) every referenced <name> has a runbooks/<name>.md in git;
# (3) each runbook clears a QUALITY FLOOR — the four operator sections, a real body, and the
# dev-container kubectl invocation with --context k3s-prod — so a stub cannot satisfy (2) and
# make the link look fixed while still helping nobody;
# (4) every runbook is PROJECTED INTO THE SERVER by runbooks/kustomization.yaml's
# configMapGenerator under the extensionless key the URL asks for;
# (5) route, listener and server agree with the annotations (PathPrefix == the annotations' prefix,
# a sectionName that platform/gateway/gateway.yaml actually defines, the namespace opted in to
# the Gateway, and an nginx location for that same prefix with a text/plain default_type);
# (6) the Flux tier is wired into clusters/base AND declares NO postBuild.substituteFrom —
# substitution would run envsubst over the runbook prose and silently empty every `${...}`
# shell example in it (it fails open: DECISIONS-LOG 2026-09-01, the emptied Cilium
# k8sServiceHost), and the flux-postbuild-substitution check cannot see it because that check
# scans files reachable through `resources:`, not configMapGenerator inputs;
# AND (7) the check is LOAD-BEARING: this Test DECLARES
# `conformance.home.k3s/scope: whole-repo`, so ci/l0.sh Stage 3b's
# `--include-scope whole-repo` unions it into EVERY gate invocation rather than only the runs
# whose touched requirements happen to include R-OBS-04; and ci/gate.sh sets the merge block
# when the l0 stage fails. A SKIP of a scope-declaring check is UNWIRED, never green.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library only);
# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test wraps the SAME
# script.
#
# NOT proven (named in check.sh's header): this does not FETCH the URL. That it returns HTTP 200 with
# real content was proven by an in-cluster probe recorded in the commit message; a live probe needs a
# cluster, which L0 by definition does not have. What is asserted here is every static precondition for
# that 200. It also does not judge whether a runbook is correct or current — (3) is a floor, not a
# review — and it does not prove Alertmanager delivers the annotation to a human (that is the L3
# `alerting` test). R-OBS-04 is tagged for the narrow slice only: the platform's own health alerts carry
# runbook links that resolve to real, served procedure rather than to a 404.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: alert-runbooks-resolvable
labels:
conformance.home.k3s/R-OBS-04: "true"
annotations:
conformance.home.k3s/requirements: "R-OBS-04"
conformance.home.k3s/layer: "L0"
# WHOLE-REPO SCOPE — this is what makes the check unconditional. Stage 3b of ci/l0.sh runs
# `conformance/run.sh --layer l0 --include-scope whole-repo`, which UNIONS every test carrying this
# annotation into the selection regardless of which requirements the change touched. That union is
# load-bearing here: ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02" —
# NOT R-OBS-04 — and maps platform/kubevirt/* and runbooks/* to nothing at all, so the single most
# likely breaking change (adding an alert with a runbook_url, or adding a runbook without its
# configMapGenerator entry) attributes a requirement set that would otherwise FILTER THIS CHECK OUT
# of exactly the change that introduces the bug. Declaring the scope replaces the hand-written
# per-test ci/l0.sh stage that this repo used to need (and that nobody remembered to add).
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Every runbook_url on every alert rule resolves to a runbook that exists in git, clears a quality
floor, and is actually projected into the server that answers that path — so an alert can never
hand an operator a dead link.
spec:
steps:
- name: audit-every-alert-runbook-link
description: >-
Parse every runbook_url out of every PrometheusRule and assert the runbook exists, is real, is
served at that exact path, and that the serving tier is wired and unsubstituted.
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,410 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — alert-runbooks-resolvable
# Serves: R-OBS-04 ("The platform alerts on its own health (nodes, storage, GitOps, certs,
# VPN)." — spec docs/10-observability-and-alerting.md)
#
# WHY THIS EXISTS — A DEAD LINK IN AN ALERT IS WORSE THAN NO LINK
# --------------------------------------------------------------
# Every platform-health alert rule in this repo carries
# runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"
# 42 such annotations across four files, over 34 distinct names. For the whole of phases 4-6
# there was NO runbooks/ directory and NO HTTPRoute serving that path: a live in-cluster probe
# of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404. So an alert reaching a
# human at 3am handed them a link that fails, at the one moment the documentation is read. The
# annotation LOOKED like the problem was solved, which is the specific failure this repo keeps
# finding (an import Job that reports success while storing placeholder secrets; a guard whose
# glob silently stopped matching) — a green-looking surface over an absent mechanism.
#
# The link is now served (runbooks/, clusters/base/runbooks.yaml). This check is the guard that
# keeps it true, because the regression is SILENT IN BOTH DIRECTIONS and nothing else catches it:
# * add an alert with a new runbook_url and forget the runbook -> that alert links to a 404,
# and no test, lint, render or policy notices;
# * write runbooks/<name>.md but forget to add it to the ConfigMap generator -> the file is in
# git, reviewable, apparently done, and the URL still 404s in the cluster. "The file exists"
# and "the URL resolves" are different claims and only the second one helps an operator.
# Both halves are asserted below, which is why this check reaches past the markdown into the
# kustomization, the HTTPRoute, the nginx config and the Flux tier.
#
# GIVEN the repository as authored,
# WHEN every `runbook_url` annotation on every PrometheusRule is parsed out of git,
# THEN (1) each URL has EXACTLY the shape the server resolves —
# https://${BASE_DOMAIN}/runbooks/<name>, `${BASE_DOMAIN}` as the token (never a literal
# hostname: a literal is how prod was authored to serve dev's hostnames, see the
# ServiceDefinition schema's note), no `.md` suffix, no trailing slash, no query;
# (2) every referenced <name> has a runbooks/<name>.md in git;
# (3) each runbook clears a QUALITY FLOOR — the four operator sections, a real body, and the
# dev-container kubectl invocation — so a one-line stub cannot satisfy (2) and make the
# link look fixed while still helping nobody (the failure mode this check is most likely
# to be defeated by is a placeholder file, not a missing one);
# (4) every runbook markdown file is actually PROJECTED INTO THE SERVER by
# runbooks/kustomization.yaml's configMapGenerator, under the EXTENSIONLESS key the URL
# asks for — the "exists in git but 404s in the cluster" half;
# (5) the route and the server agree with the annotations: the HTTPRoute's PathPrefix is the
# prefix the URLs use, it attaches to a Gateway listener that actually exists, its
# namespace is opted in to that Gateway, and the nginx config answers that same prefix;
# (6) the Flux tier that reconciles all of it is wired into clusters/base AND declares NO
# postBuild.substituteFrom — substitution here would run envsubst over the runbook prose
# and silently empty every `${...}` shell example in it (it fails open: DECISIONS-LOG
# 2026-09-01, the emptied Cilium k8sServiceHost);
# AND (7) this check is LOAD-BEARING: ci/l0.sh runs it on every gate run and ci/gate.sh sets the
# merge block when the l0 stage fails.
#
# WHY (7) IS AN UNCONDITIONAL ci/l0.sh STAGE and not left to the requirement-scoped run: Stage 3b
# scopes `conformance/run.sh --layer l0` to the TOUCHED requirements (R-TEST-03), and
# ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02" — NOT R-OBS-04 — and
# maps platform/kubevirt/* and runbooks/* to nothing at all. So the single most likely breaking
# change (adding an alert with a runbook_url to platform/monitoring/alerts.yaml, or adding a
# runbook and forgetting the generator entry) attributes a requirement set that would FILTER THIS
# CHECK OUT. Declaring `conformance.home.k3s/scope: whole-repo` makes ci/l0.sh Stage 3b union it into
# EVERY gate invocation instead; a SKIP of a scope-declaring check is UNWIRED, never green.
#
# WHAT THIS DOES *NOT* PROVE (honest gaps — this is a STATIC check):
# - It does not fetch the URL. That it returns HTTP 200 with real content was proven by an
# in-cluster probe (recorded in the commit message and DECISIONS-LOG); a live probe needs a
# cluster, which L0 by definition does not have. What is asserted here is every static
# precondition for that 200: the file, its projection, the key name, the route, the prefix.
# - It does not judge whether a runbook is CORRECT or current. (3) is a floor, not a review: it
# can tell a stub from a runbook, not a good runbook from a stale one. Nothing automatable can.
# - It does not prove Alertmanager actually delivers the annotation to a human. That is the L3
# `alerting` test.
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06) and is not consulted.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
# Honest SKIP (never green) if there are no alert rules at all — nothing to audit yet.
if ! grep -rl 'kind: PrometheusRule' "${ROOT}/platform" >/dev/null 2>&1; then
echo "check SKIPPED: no PrometheusRule found under platform/ — no alert rules to audit" >&2
exit 3
fi
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
"""Static audit: every alert runbook_url resolves to a runbook that is actually served.
Pure standard library, no PyYAML — the same constraint and the same style as the neighbouring
L0 checks (lib/coverage.py, lib/status.py). Every parse contract is documented at its parser;
all of them are line-constrained shapes in hand-authored manifests.
"""
from __future__ import annotations
import re
import sys
from pathlib import Path
ROOT = Path(sys.argv[1])
sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree
FAILURES: list[str] = []
ADVISORIES: list[str] = []
CHECKS = 0
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
def read(p: Path) -> str:
try:
return p.read_text()
except (OSError, UnicodeDecodeError):
return ""
# =============================================================== (1) parse the annotations
# Parse contract: `runbook_url:` is an annotation on a Prometheus alerting rule, authored one per
# line as `runbook_url: "<url>"` (single or double quoted, or bare). Only files that declare
# `kind: PrometheusRule` are scanned, so a runbook_url quoted inside this check or inside prose
# elsewhere in the repo is not mistaken for an alert annotation. Commented-out lines are skipped:
# platform/monitoring/alerts.yaml carries a `# runbook_url points at a docs base` header comment.
SCOPE_DECL_RE = r"^\s*conformance\.home\.k3s/scope:\s*[\"\']?whole-repo"
RUNBOOK_URL_RE = re.compile(r"""^\s*runbook_url:\s*["']?([^"'\s#]+)["']?\s*$""")
rule_files = sorted(
p for p in ROOT.rglob("*.yaml")
if ".git" not in p.parts and "kind: PrometheusRule" in read(p)
)
check(len(rule_files) > 0,
"no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "
"this check would silently audit nothing")
# url -> [ "<relpath>:<lineno>", ... ]
refs: dict[str, list[str]] = {}
for f in rule_files:
rel = f.relative_to(ROOT)
for lineno, line in enumerate(read(f).splitlines(), 1):
if line.lstrip().startswith("#"):
continue
m = RUNBOOK_URL_RE.match(line)
if m:
refs.setdefault(m.group(1), []).append(f"{rel}:{lineno}")
n_annotations = sum(len(v) for v in refs.values())
check(n_annotations > 0,
"no runbook_url annotation parsed out of any PrometheusRule — the parser contract broke; "
"this check would pass vacuously while every alert link rots")
# ---- URL SHAPE. The server resolves exactly one shape. Anything else is a link that 404s even
# though a file with a plausible name exists, so shape is a failure, not a style note.
URL_RE = re.compile(r"^https://\$\{BASE_DOMAIN\}/(runbooks)/([a-z0-9][a-z0-9-]*)$")
names: dict[str, list[str]] = {} # runbook name -> annotation sites
prefixes: set[str] = set()
for url, sites in sorted(refs.items()):
m = URL_RE.match(url)
check(m is not None,
f"runbook_url '{url}' ({', '.join(sites)}) is not the shape this platform serves. "
f"Required: https://${{BASE_DOMAIN}}/runbooks/<name> — the ${{BASE_DOMAIN}} TOKEN (never a "
f"literal hostname: a literal is how prod got authored to serve dev's public names), a "
f"lowercase-hyphen <name>, and NO '.md' suffix, trailing slash or query. The runbook "
f"server maps the URL path straight onto an extensionless ConfigMap key, so any other "
f"shape is a 404 no matter which files exist")
if m:
prefixes.add(m.group(1))
names.setdefault(m.group(2), []).extend(sites)
check(len(prefixes) <= 1,
f"alert annotations use more than one runbook path prefix ({sorted(prefixes)}); the single "
f"HTTPRoute + nginx location can only serve one, so the others are dead links")
PREFIX = (sorted(prefixes) or ["runbooks"])[0]
# =================================================== (2)(3) the runbooks exist and are real
RUNBOOK_DIR = ROOT / PREFIX
check(RUNBOOK_DIR.is_dir(),
f"{PREFIX}/ is not a directory in this repo, so every one of the {n_annotations} "
f"runbook_url annotations is a dead link")
# The quality floor. (2) alone is satisfiable by `touch runbooks/node-down.md`, which makes the
# link resolve and still abandons the operator — a worse outcome than the 404, because the 404 at
# least told the truth. These four sections are the operator's path through an incident: what the
# alert means HERE, what to type first, what is probably wrong, and when to stop. `dev.sh` +
# `--context k3s-prod` are required because kubectl does not work on the workstation at all
# (the toolchain lives only in the pinned dev container) and because dev.sh defaults its
# current-context to k3s-DEV — a runbook that omits the context teaches the reader to answer
# questions about the wrong cluster during an incident on this one.
REQUIRED_SECTIONS = (
"## What this means here",
"## First commands",
"## Likely causes, most likely first",
"## When to escalate or stop",
)
MIN_BYTES = 1500
for name, sites in sorted(names.items()):
md = RUNBOOK_DIR / f"{name}.md"
if not md.is_file():
check(False,
f"runbook '{name}' does not exist: {md.relative_to(ROOT)} is missing, but "
f"{len(sites)} alert annotation(s) link to it ({', '.join(sites[:3])}"
f"{', ...' if len(sites) > 3 else ''}). An alert firing now hands the operator a 404. "
f"Write {PREFIX}/{name}.md (see {PREFIX}/README.md for the required sections)")
continue
body = read(md)
check(len(body) >= MIN_BYTES,
f"{md.relative_to(ROOT)} is {len(body)} bytes, under the {MIN_BYTES}-byte floor — a stub "
f"that makes the link resolve while still helping nobody is worse than the 404 it "
f"replaced, because it looks fixed")
for sec in REQUIRED_SECTIONS:
check(sec in body,
f"{md.relative_to(ROOT)} has no '{sec}' section — the four operator sections are what "
f"make this a runbook rather than a restatement of the alert name")
check("dev.sh" in body,
f"{md.relative_to(ROOT)} never mentions dev.sh — kubectl does not exist on the "
f"workstation; every command in a runbook here must be runnable as "
f"`infra/devcontainer/dev.sh run bash -c '...'` and the runbook must say so")
check("--context k3s-prod" in body,
f"{md.relative_to(ROOT)} never passes --context k3s-prod — dev.sh merges both clusters "
f"and picks a current-context PER INVOCATION — a default that has already changed once "
f"(k3s-dev until dev was decommissioned on 2026-09-13, k3s-prod now). A context-less command "
f"silently answers about whichever cluster the default happens to be, in the middle of an "
f"incident")
# ADVISORY, not a failure: a runbook nothing links to yet. Writing the runbook BEFORE the alert is
# the right order, and the repo's precedent for "authored but unreferenced" is advisory (see the
# dead-knob advisory in flux-postbuild-substitution/check.sh).
served_extra = {"README"}
if RUNBOOK_DIR.is_dir():
for md in sorted(RUNBOOK_DIR.glob("*.md")):
stem = md.stem
if stem in served_extra or stem in names:
continue
ADVISORIES.append(f"{md.relative_to(ROOT)} is a runbook no alert links to (written ahead of "
f"its alert, or the alert was removed — advisory, not a failure)")
# ================================= (4) every runbook is PROJECTED INTO THE SERVER, not just in git
# THIS IS THE HALF A FILE-EXISTENCE CHECK WOULD MISS. The pod serves a ConfigMap, not the git tree.
# Parse contract: runbooks/kustomization.yaml declares one configMapGenerator whose `files:` entries
# are `- <key>=<file>.md`, one per line. The key is EXTENSIONLESS on purpose (the URL carries no
# `.md`, and the ConfigMap volume mount turns each key into a file the nginx root resolves directly).
KUST = RUNBOOK_DIR / "kustomization.yaml"
check(KUST.is_file(), f"{PREFIX}/kustomization.yaml is missing — the Flux tier renders nothing and "
f"every runbook 404s no matter what is in git")
GEN_FILE_RE = re.compile(r"^\s*-\s*([A-Za-z0-9._-]+)=([A-Za-z0-9._/-]+\.md)\s*(?:#.*)?$")
generated: dict[str, str] = {}
kust_text = read(KUST)
if kust_text:
check("configMapGenerator:" in kust_text,
f"{PREFIX}/kustomization.yaml has no configMapGenerator — the runbook markdown is not "
f"rendered into the ConfigMap the server mounts")
for line in kust_text.splitlines():
if line.lstrip().startswith("#"):
continue
m = GEN_FILE_RE.match(line)
if m:
generated[m.group(1)] = m.group(2)
check(len(generated) > 0,
f"{PREFIX}/kustomization.yaml declares no `<key>=<file>.md` generator entries — the "
f"parser contract broke, or the markdown is being added with implicit keys (which would "
f"carry the '.md' suffix the alert URLs do NOT have, so every link would 404)")
for name in sorted(names):
check(generated.get(name) == f"{name}.md",
f"runbook '{name}' is not projected into the server: {PREFIX}/kustomization.yaml has no "
f"configMapGenerator entry `- {name}={name}.md`. The file can be in git, reviewed and "
f"merged and the URL still returns 404 — 'the file exists' and 'the URL resolves' are "
f"different claims")
# And the reverse, so a generator entry cannot point at a file that is not there.
for key, fname in sorted(generated.items()):
check((RUNBOOK_DIR / fname).is_file(),
f"{PREFIX}/kustomization.yaml generates key '{key}' from {PREFIX}/{fname}, which does not "
f"exist — `kustomize build` fails, so the WHOLE tier stops reconciling and every runbook "
f"goes stale at once")
# ============================== (5) the route, the listener and the server agree with the URLs
ROUTE = RUNBOOK_DIR / "serving" / "httproute.yaml"
NGINX = RUNBOOK_DIR / "serving" / "nginx-config.yaml"
NS = RUNBOOK_DIR / "serving" / "namespace.yaml"
GATEWAY = ROOT / "platform" / "gateway" / "gateway.yaml"
route_text = read(ROUTE)
check(bool(route_text), f"{PREFIX}/serving/httproute.yaml is missing — nothing routes the "
f"/{PREFIX} prefix to the server")
# Parse contract: a single HTTPRoute with one PathPrefix match and one parentRefs entry naming a
# listener via sectionName; hand-authored, one key per line.
m_path = re.search(r"^\s*type:\s*PathPrefix\s*$\n\s*value:\s*(\S+)\s*$", route_text, re.M)
check(m_path is not None and m_path.group(1) == f"/{PREFIX}",
f"{PREFIX}/serving/httproute.yaml does not match PathPrefix /{PREFIX} "
f"(found {m_path.group(1) if m_path else 'no PathPrefix match'}). The alert annotations point "
f"at /{PREFIX}/<name>; if the route's prefix drifts from the annotations the links 404 while "
f"both files look individually correct")
m_sec = re.search(r"^\s*sectionName:\s*(\S+)", route_text, re.M)
check(m_sec is not None,
f"{PREFIX}/serving/httproute.yaml declares no parentRefs sectionName. The apex host needs the "
f"dedicated apex listener: the wildcard `*.${{BASE_DOMAIN}}` listener does NOT match the bare "
f"apex, so a route without sectionName binds nothing and serves nothing")
if m_sec:
listener = m_sec.group(1)
gw = read(GATEWAY)
check(re.search(rf"^\s*-\s*name:\s*{re.escape(listener)}\s*$", gw, re.M) is not None,
f"{PREFIX}/serving/httproute.yaml attaches to Gateway listener '{listener}', which "
f"platform/gateway/gateway.yaml does not define — the route would report NotAllowed and "
f"never serve")
# An HTTPRoute with no `hostnames:` inherits the listener's hostname, which is how this tier avoids
# needing Flux substitution (see (6)). Either form is routable; a LITERAL hostname is not acceptable,
# for the same reason the schema forbids it in a ServiceDefinition.
m_host = re.search(r"^\s*hostnames:\s*$\n\s*-\s*(\S+)", route_text, re.M)
check(m_host is None or "${BASE_DOMAIN}" in m_host.group(1),
f"{PREFIX}/serving/httproute.yaml hardcodes hostname '{m_host.group(1) if m_host else ''}' — "
f"use the ${{BASE_DOMAIN}} token, or omit `hostnames:` entirely and inherit the listener's")
ns_text = read(NS)
check('gateway-access: "shared"' in ns_text or "gateway-access: shared" in ns_text,
f"{PREFIX}/serving/namespace.yaml does not carry gateway-access=shared — the shared Gateway "
f"selects namespaces by that label, so without it the HTTPRoute is refused attachment")
nginx_text = read(NGINX)
check(f"location /{PREFIX}/" in nginx_text,
f"{PREFIX}/serving/nginx-config.yaml has no `location /{PREFIX}/` block — the route would "
f"deliver the request and the server would answer 404 for every runbook")
check("default_type" in nginx_text and "text/plain" in nginx_text,
f"{PREFIX}/serving/nginx-config.yaml does not force a text/plain default_type. The ConfigMap "
f"keys are EXTENSIONLESS, so with the stock MIME table nginx serves them as "
f"application/octet-stream and a browser DOWNLOADS the runbook instead of showing it")
# ====================================== (6) the Flux tier is wired, and is NOT substituted
FLUX = ROOT / "clusters" / "base" / f"{PREFIX}.yaml"
BASE_KUST = ROOT / "clusters" / "base" / "kustomization.yaml"
flux_text = read(FLUX)
check(bool(flux_text),
f"clusters/base/{PREFIX}.yaml is missing — nothing reconciles {PREFIX}/ into the cluster, so "
f"the runbooks exist in git and the URL still 404s")
check(re.search(rf"^\s*path:\s*\./{re.escape(PREFIX)}\s*$", flux_text, re.M) is not None,
f"clusters/base/{PREFIX}.yaml does not set `path: ./{PREFIX}`")
check(re.search(r"^\s*wait:\s*false\s*$", flux_text, re.M) is not None,
f"clusters/base/{PREFIX}.yaml is not wait:false — a leaf web server that can be legitimately "
f"unhealthy must never gate the reconcile chain (DECISIONS-LOG: 'Headlamp gating stall')")
# Strip comments before testing: this file's own comment block EXPLAINS at length why
# substituteFrom must never be added here, and a naive substring test would fire on the warning.
flux_code = "\n".join(l for l in flux_text.splitlines() if not l.lstrip().startswith("#"))
check("substituteFrom" not in flux_code,
f"clusters/base/{PREFIX}.yaml declares postBuild.substituteFrom. It MUST NOT: this tier "
f"renders documentation into a ConfigMap, and Flux's envsubst would run over the prose and "
f"silently EMPTY every `${{...}}` shell example in every runbook — it fails open, so the "
f"Kustomization would still report Ready (DECISIONS-LOG 2026-09-01, the emptied Cilium "
f"k8sServiceHost). The flux-postbuild-substitution check cannot see this: it scans files "
f"reachable through `resources:`, not configMapGenerator inputs")
check(re.search(rf"^\s*-\s*{re.escape(PREFIX)}\.yaml\b", read(BASE_KUST), re.M) is not None,
f"clusters/base/kustomization.yaml does not list {PREFIX}.yaml — the Flux Kustomization is "
f"authored and never applied, which is the 'looks done, is absent' class this repo has been "
f"bitten by repeatedly")
# ============================================ (7) this check is load-bearing in the merge gate
L0 = read(ROOT / "ci" / "l0.sh")
GATE = read(ROOT / "ci" / "gate.sh")
# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism, in two halves.
# It is not enough that check.sh exists: the merge gate must SELECT it on every run. The repo used to
# get that with a hand-written "UNCONDITIONAL" stage per whole-repo check in ci/l0.sh, hardcoding the
# path; those stages are gone, because the next whole-repo invariant only got one if somebody
# REMEMBERED to write it. The property is now DECLARED by the test and HONOURED by the gate, so both
# halves are asserted — neither alone is sufficient. The mechanism itself (that the union really
# selects, and that a scope-declared SKIP exits 3 rather than passing green) is proven end-to-end and
# falsifiably by conformance/tests/l0-static/whole-repo-invariants-wired.
SELF_YAML = (ROOT / "conformance" / "tests" / "l0-static" / "alert-runbooks-resolvable"
/ "chainsaw-test.yaml")
SELF = read(SELF_YAML)
check(re.search(SCOPE_DECL_RE, SELF, re.M) is not None,
"alert-runbooks-resolvable/chainsaw-test.yaml does not declare "
"conformance.home.k3s/scope: whole-repo, so the merge gate selects this check only when a "
"change happens to touch a requirement it is labelled with. ci/touched-requirements.sh maps "
"platform/monitoring/* to 'R-OBS-01 R-OBS-02' (NOT R-OBS-04) and maps platform/kubevirt/* and "
"runbooks/* to nothing at all — so adding an alert with a new runbook_url, or adding a runbook "
"without its configMapGenerator entry, would be filtered out of exactly the change that "
"introduces the dead link")
check("--include-scope" in L0,
"ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is a "
"purely requirement-scoped run again and the scope annotation above is inert decoration — this "
"check would then run only by coincidence")
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
"ci/gate.sh does not set the merge block when the l0 stage fails")
# ======================================================================== report
print(f"Alert runbook inventory: {n_annotations} runbook_url annotation(s) across "
f"{len(rule_files)} PrometheusRule file(s) -> {len(names)} distinct runbook(s), "
f"prefix /{PREFIX}")
missing = sorted(n for n in names if not (RUNBOOK_DIR / f"{n}.md").is_file())
print(f" runbooks present: {len(names) - len(missing)}/{len(names)} "
f"projected into the ConfigMap: {sum(1 for n in names if generated.get(n) == f'{n}.md')}/{len(names)}")
if missing:
print(f" MISSING: {', '.join(missing)}", file=sys.stderr)
if ADVISORIES:
print("\nADVISORIES (not failures):")
for a in ADVISORIES:
print(f" - {a}")
print()
if not FAILURES:
print(f"check OK: alert-runbooks-resolvable — {CHECKS} assertions (R-OBS-04)")
raise SystemExit(0)
print(f"CHECK FAIL (alert-runbooks-resolvable): {len(FAILURES)} failed, "
f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
for f in FAILURES:
print(f" FAIL {f}", file=sys.stderr)
raise SystemExit(1)
PY
@@ -0,0 +1,114 @@
# =============================================================================
# L0 STATIC — the five auditability proofs can be produced on demand (R-SEC-03) [EXECUTABLE]
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
# Spec: docs/00 R-SEC-03 ("All the above properties MUST be demonstrable/auditable (an
# operator can prove isolation, egress path, cert validity, and backup integrity)"),
# made concrete by docs/07 §"Auditability requirements (R-SEC-03) — what an operator
# must be able to prove": Isolation (denied-probe test), Egress path (egress IP
# check), Kill-switch (fault-injection test), TLS (cert status + external check),
# Backups (DR drill) — "These proofs are part of the platform acceptance suite, not
# one-time checks", with the acceptance gate "The five auditability proofs above can
# be produced on demand."
#
# HOW THIS AVOIDS BEING A TAUTOLOGY (the whole design question for a meta-requirement)
# -----------------------------------------------------------------------------------
# "A test exists for R-NET-01, therefore isolation is auditable" would pass in any suite that
# compiles and would keep passing while every proof rotted into a scaffold. Four distinct
# failure modes are asserted instead, each by DRIVING the real machinery:
# (a) each of the five proofs has an EXECUTABLE test — not a placeholder scaffold — at the layer
# the spec's own words require (fault injection / a DR drill => L3), taken from
# lib/status.py's own verdict so this check cannot disagree with the status report;
# (b) the proofs are PRODUCIBLE by the documented command: `run.sh --requirements <the five>`
# must resolve to real test dirs AND pass the merge gate's coverage assertion;
# (c) each Chainsaw-layer proof declares a PASS SENTINEL that lib/record_run.py can
# attribute — a proof that passes but records nothing leaves the operator with a green
# terminal and no evidence;
# (d) the evidence chain REFUSES FORGERIES: this check mounts two of its own against the
# real tools — a green JUnit for a real proof test whose log holds only that test's SKIP
# token, and a ledger row claiming `proven` at the repository's ROOT commit — and
# requires record_run.py to report not-proven and status.py to report `stale-proof`. It
# also drives `record_run.py --selftest` and `status.py --selftest`, which NOTHING in
# ci/ or run.sh ran before this test existed.
# The five proofs are not hand-listed where the spec can be read: when the read-only spec repo
# is mounted, docs/07's bullet labels are parsed and must match this check's mapping exactly,
# so a sixth proof added to the spec fails here instead of being silently unverified.
#
# WHY L0: every clause is a property of the repository and its own tooling, needs no cluster,
# and belongs in the merge gate — an audit chain is worth having only if it is intact BEFORE
# anyone needs it.
#
# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted),
# WHEN lib/status.py, lib/common.sh's requirement resolver, run.sh's coverage gate,
# lib/record_run.py's sentinel parser and both evidence tools' self-tests are DRIVEN,
# and two synthetic forgeries are presented to them,
# THEN each of docs/07's five proofs maps to an executable test at the required layer, the
# documented one-command invocation selects and asserts them, each is capturable as a
# ledger row, and neither a self-skip nor a stale row can be recorded as proof.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + the harness's
# own stdlib tooling; no cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run
# it directly, and this Chainsaw Test wraps the SAME script.
#
# WHAT THIS TEST FIXED WHILE BEING WRITTEN (recorded, not hidden): three of the tests that ARE
# the automated half of docs/07's isolation and TLS proofs — l2-behavioral/default-deny,
# l2-behavioral/trusted-tls and l2-behavioral/gateway-ingress-reachable — declared NO pass
# sentinel, so lib/record_run.py could never record them as proven and those two audit proofs
# were structurally unrecordable. Each now ends with a proof-of-pass sentinel step. The
# assertion here was NOT relaxed to accommodate them (R-TEST-06).
#
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
# * That the five proofs PASS. This asserts they are present, strong, runnable by one
# command, capturable, and that the chain refuses forgeries — not that isolation holds
# today (R-NET-01's test) or that the restore works (R-STATE-03's).
# * HUBBLE. docs/07 names Hubble flow logs as part of the isolation proof; no test in this
# suite queries Hubble. Only the denied-probe half is automated — a real, named gap.
# * LEDGER COMPLETENESS OVER TIME. Nothing offline can prove a row was never removed; that
# would need signed history.
# * R-SEC-03's wording is broader than docs/07's five bullets. Suite-wide coverage of every
# requirement is R-TEST-01/04's claim (tests/meta/harness-selfcheck) and is deliberately
# not re-asserted here.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: auditability-proofs-producible
labels:
conformance.home.k3s/R-SEC-03: "true"
annotations:
conformance.home.k3s/requirements: "R-SEC-03"
conformance.home.k3s/layer: "L0"
# WHOLE-REPO SCOPE. This is a META-invariant over the ENTIRE conformance suite: the five
# spec-named proofs must each still have an EXECUTABLE test at the right layer, with a
# pass sentinel, resolvable by the documented command. Every one of its inputs lives under
# conformance/, which ci/touched-requirements.sh keys to R-TEST-01..04 — never to R-SEC-03.
# So downgrading conformance/tests/l3-fault-injection/dr-restore to a scaffold attributes
# R-STATE-03/R-TEST-*, and requirement-scoped selection drops the one check that notices
# the audit evidence just stopped being producible.
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Given the tracked tree and the harness's own tooling, when status.py, the requirement
resolver, run.sh's coverage gate, record_run.py's sentinel parser and both evidence
tools' self-tests are driven and two synthetic forgeries are presented to them, then each
of docs/07's five auditability proofs has an executable test at the layer the spec
requires, is selectable and asserted by the documented one-command invocation, is
capturable as a ledger row, and neither a self-skip nor a stale proof can be recorded as
evidence.
spec:
steps:
- name: then-each-of-the-five-proofs-is-producible-and-the-evidence-chain-refuses-forgeries
description: >-
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
conformance/run.sh --layer l0 execute).
try:
- script:
content: bash ./check.sh
# ---------------------------------------------------------------------------------------------
# NOTE ON WORDING (deliberate, do not "improve"): this file and check.sh say "placeholder
# scaffold" rather than the four-letter marker lib/status.py looks for, because that detector is
# a bare substring test over the whole file. Writing the marker in PROSE — even to say a proof
# must NOT be one — makes status.py classify THIS test as a scaffold, so the R-SEC-03 test would
# report itself as unexecutable. Observed and worked around here; the detector's crudeness is
# reported to the PM rather than loosened, since loosening a fake-green guard to suit one file is
# the wrong trade (R-TEST-06).
@@ -0,0 +1,454 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — auditability-proofs-producible
# Serves: R-SEC-03 ("All the above properties MUST be demonstrable/auditable (an operator can
# prove isolation, egress path, cert validity, and backup integrity)."
# — spec docs/00-vision-and-scope.md).
#
# The spec makes this meta-requirement concrete. docs/07-security-and-secrets.md
# §"Auditability requirements (R-SEC-03) — what an operator must be able to prove":
# - **Isolation**: "service X cannot reach service Y" — via Hubble flow logs + a
# denied-probe test.
# - **Egress path**: "service X's public IP is the VPN's" — via an egress IP check.
# - **Kill-switch**: "with VPN down, X leaks nothing" — via a fault-injection test.
# - **TLS**: "every endpoint serves a valid, non-self-signed, unexpired cert" — via cert
# status + external check.
# - **Backups**: "irreplaceable data is restorable" — via a DR drill.
# "These proofs are part of the platform acceptance suite, not one-time checks."
# and its acceptance gate: "The five auditability proofs above can be produced on demand."
#
# HOW A META-REQUIREMENT IS TESTED WITHOUT BECOMING A TAUTOLOGY
# ------------------------------------------------------------
# The trap is obvious and this check is built to avoid it: "a test exists for R-NET-01,
# therefore isolation is auditable" is true of any suite that compiles, proves nothing, and
# would keep passing while every proof rotted into a scaffold. Four separate failure modes
# have to be caught for "an operator can PROVE it" to mean anything, and each one is asserted
# by DRIVING the real machinery rather than by reading files:
#
# (a) THE PROOF EXISTS, AT THE RIGHT STRENGTH. Each of the five named proofs maps to a
# requirement that must have an EXECUTABLE verifying test — not a placeholder scaffold — and,
# where the spec's own words name fault injection ("with VPN down"; "a DR drill"), an
# executable L3 test specifically. Executable-vs-scaffold is not re-derived here: this
# check RUNS lib/status.py and reads its verdict, so it cannot disagree with the
# repository's own status report.
# (b) THE PROOF IS PRODUCIBLE ON DEMAND, BY THE DOCUMENTED COMMAND. `run.sh
# --requirements <the five IDs>` must resolve to a non-empty test set and must ASSERT
# their coverage (that flag is the merge gate's own path: it passes --require to
# coverage.py). This check drives run.sh itself, so if the resolver, the labels or the
# gate wiring break, R-SEC-03 fails.
# (c) THE PROOF IS CAPTURABLE. A test that passes but records nothing leaves an operator
# with a green terminal and no evidence. For each Chainsaw-layer proof, the test must
# declare a PASS SENTINEL that lib/record_run.py can actually attribute — computed with
# record_run.py's OWN parser, so deleting a test's sentinel makes this check fail
# instead of silently making the requirement unprovable.
# (d) THE EVIDENCE CANNOT BE FORGED, AND IS NOT HONOURED WHEN STALE. This is the clause
# that makes the check bite. Two INDEPENDENT forgery attempts are mounted against the
# real tools, in a throwaway directory:
# * a Chainsaw run in which the JUnit says a real proof test PASSED while the run log
# carries only that test's SKIP token — the exact shape of a self-skipping L2/L3
# test. record_run.py must refuse to call it `proven`.
# * a ledger row claiming `proven` at a commit that PREDATES the requirement's
# implementing code — status.py must degrade it to `stale-proof`, never green.
# Both tools' own built-in self-tests (`--selftest`) are also driven, because nothing
# in ci/ or run.sh runs them today: the guards that keep a skip from becoming a proof
# were themselves unexercised in the merge gate until this check.
#
# The FIVE PROOFS ARE NOT HAND-LISTED where the spec can be read: when the read-only spec
# repo is mounted, the bullet labels are parsed out of docs/07 and must match this check's
# mapping EXACTLY. Add a sixth proof to the spec and this check fails as unmapped rather
# than quietly ignoring it.
#
# WHY L0: every clause above is a property of the repository and its own tooling, provable
# offline, and it belongs in the merge gate — the point of an audit chain is that it is intact
# BEFORE anyone needs it. Nothing here needs a cluster, and nothing here would be more true
# with one.
#
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them):
# - THAT THE FIVE PROOFS PASS. This asserts the proofs are present, strong, runnable by one
# documented command, capturable as evidence, and that the evidence chain refuses
# forgeries. Whether isolation actually holds today is R-NET-01's test; whether the DR
# restore works is R-STATE-03's. R-SEC-03 is about the operator's ABILITY to produce the
# proof, which is what is asserted.
# - HUBBLE. docs/07 names "Hubble flow logs" as part of the isolation proof. No test in this
# suite queries Hubble, and this check does not pretend otherwise: it asserts the
# denied-probe half, which is the half that is automated. The Hubble half remains an
# operator-interactive capability — a real, named gap in R-SEC-03's coverage.
# - THE OPERATOR-FACING NARRATIVE. That a human can find and read the evidence (the status
# report's prose, the ledger's provenance fields) is a usability property, not a testable
# one.
# - LEDGER COMPLETENESS OVER TIME. The ledger is append-only by convention and by review;
# nothing here can prove a row was never removed from it (that would need signed history).
# - THAT THE PROOFS COVER "ALL THE ABOVE PROPERTIES". R-SEC-03's own wording is broader than
# the five bullets docs/07 enumerates. The five are what the spec made concrete and are
# what is asserted; suite-wide coverage of every requirement is R-TEST-01/04's claim,
# tested by tests/meta/harness-selfcheck, and is deliberately NOT re-asserted here.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
: "${SPEC_REPO:=${ROOT}/../home-network-k3s}"
export SPEC_REPO
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
import json
import os
import re
import shutil
import subprocess
import sys
import tempfile
from pathlib import Path
ROOT = Path(sys.argv[1])
CONF = ROOT / "conformance"
LIB = CONF / "lib"
FAILURES: list[str] = []
NOTES: list[str] = []
CHECKS = 0
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
def note(msg):
NOTES.append(msg)
def die():
print(f"CHECK FAIL (auditability-proofs-producible): {len(FAILURES)} of {CHECKS} "
"assertions failed", file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
def run(cmd, **kw) -> subprocess.CompletedProcess:
return subprocess.run(cmd, capture_output=True, text=True, cwd=kw.pop("cwd", CONF), **kw)
# ================================================= THE FIVE PROOFS docs/07 ENUMERATES
# key = the bullet label docs/07 uses (verified against the spec below when it is mounted)
# reqs = the requirement(s) whose verifying test IS that proof
# layer= the WEAKEST layer the spec's own words allow. Where docs/07 says "a fault-injection
# test" / "a DR drill", an L2 happy-path test does not discharge the proof.
PROOFS = {
"Isolation": {"reqs": ["R-NET-01"], "layers": {"L2", "L3"},
"why": '"service X cannot reach service Y" — via Hubble flow logs + a '
"denied-probe test"},
"Egress path": {"reqs": ["R-NET-05"], "layers": {"L2", "L3"},
"why": '"service X\'s public IP is the VPN\'s" — via an egress IP check'},
"Kill-switch": {"reqs": ["R-NET-06"], "layers": {"L3"},
"why": '"with VPN down, X leaks nothing" — via a fault-injection test'},
"TLS": {"reqs": ["R-TLS-01"], "layers": {"L2", "L3"},
"why": '"every endpoint serves a valid, non-self-signed, unexpired cert" '
"— via cert status + external check"},
"Backups": {"reqs": ["R-STATE-03"], "layers": {"L3"},
"why": '"irreplaceable data is restorable" — via a DR drill'},
}
PROOF_REQS = sorted({r for p in PROOFS.values() for r in p["reqs"]})
# ---- SELF-POLICING: the proof list must still be the SPEC's proof list ---------------
spec = Path(os.environ.get("SPEC_REPO", ""))
sec_doc = spec / "docs" / "07-security-and-secrets.md"
if sec_doc.is_file():
text = sec_doc.read_text()
m = re.search(r"^##\s+Auditability requirements \(R-SEC-03\).*?$(.*?)^##\s", text,
re.M | re.S)
check(m is not None,
"docs/07 no longer carries an '## Auditability requirements (R-SEC-03)' section — "
"the proof list this check is coupled to has moved or been removed, so the mapping "
"below can no longer be trusted (R-TEST-06: follow the spec, never the other way)")
if m:
labels = re.findall(r"^\s*-\s+\*\*([^*]+)\*\*:", m.group(1), re.M)
check(set(labels) == set(PROOFS),
f"docs/07 enumerates the auditability proofs {sorted(labels)} but this check "
f"maps {sorted(PROOFS)} — every proof the spec names must map to a verifying "
"test, and a proof this check does not know about would be silently unverified")
check(len(labels) == 5,
f"docs/07 enumerates {len(labels)} auditability proof(s); its own acceptance "
"gate says 'The five auditability proofs above can be produced on demand' — the "
"prose and the list have drifted apart")
else:
note(f"the read-only spec repo is not mounted at {spec} — the proof list could not be "
"re-derived from docs/07, so the mapping below was used as written. Every other "
"assertion still ran. (ci/l0.sh mounts the spec; a bare local run may not.)")
# ============================================ (a) EACH PROOF EXISTS, AT THE RIGHT STRENGTH
# Driven through lib/status.py so this check and the repository's own status report cannot
# disagree about what is executable and what is a scaffold.
tmp = Path(tempfile.mkdtemp(prefix="r-sec-03-"))
try:
out = tmp / "status"
st = run([sys.executable, str(LIB / "status.py"), "--repo", str(ROOT),
"--registry", str(CONF / "requirements" / "registry.yaml"),
"--tests", str(CONF / "tests"), "--out", str(out)])
check(st.returncode == 0,
f"lib/status.py exited {st.returncode} — the report an operator would use to find "
f"the audit evidence does not generate: {(st.stderr or st.stdout).strip()[:400]}")
status = {}
sj = out / "status.json"
if sj.is_file():
status = json.loads(sj.read_text())
check(bool(status.get("by_requirement")),
"lib/status.py produced no per-requirement report — the audit posture of each "
"requirement is not derivable, so R-SEC-03 cannot be assessed at all")
by_req = status.get("by_requirement", {})
for label, spec_proof in sorted(PROOFS.items()):
for rid in spec_proof["reqs"]:
rec = by_req.get(rid)
check(rec is not None,
f"proof '{label}' maps to {rid}, which the status report does not know — "
"either the registry lost the requirement or this mapping is stale")
if not rec:
continue
tests = (rec.get("verification") or {}).get("tests") or []
executable = [t for t in tests if t.get("state", "").startswith("executable")]
check(bool(executable),
f"proof '{label}' ({rid}) has no EXECUTABLE verifying test — its tests are "
f"{[t.get('name') + ':' + t.get('state', '?') for t in tests]}. docs/07: "
f"{spec_proof['why']}. A placeholder scaffold cannot be produced on demand, so the "
"operator cannot prove this property")
at_layer = [t for t in executable if t.get("layer") in spec_proof["layers"]]
check(bool(at_layer),
f"proof '{label}' ({rid}) has no executable test at layer(s) "
f"{sorted(spec_proof['layers'])} — it has "
f"{[t.get('name') + '@' + str(t.get('layer')) for t in executable]}. "
f"docs/07 requires this proof to be produced {spec_proof['why']}, and a "
"weaker layer does not discharge it (a rendering check is not a drill)")
# ==================================== (b) PRODUCIBLE ON DEMAND, BY THE DOCUMENTED COMMAND
# The resolver: --requirements must translate the five R-IDs into real test dirs.
resolver = run(["bash", "-c",
f'export CONF_ROOT="{CONF}"; . "{LIB}/common.sh"; '
f'conf_dirs_for_requirements "{" ".join(PROOF_REQS)}"'])
dirs = [l for l in resolver.stdout.splitlines() if l.strip()]
check(resolver.returncode == 0 and len(dirs) >= len(PROOF_REQS),
f"`run.sh --requirements {','.join(PROOF_REQS)}` resolves to {len(dirs)} test dir(s) "
f"for {len(PROOF_REQS)} proof requirement(s) — the single command docs/07's "
"acceptance gate depends on ('produced on demand') cannot select the proofs")
# The gate: the same flag ASSERTS coverage (run.sh passes --require to coverage.py), so a
# zero exit here is the repository's own statement that all five proofs are covered.
gate = run(["./run.sh", "--coverage-only", "--requirements", ",".join(PROOF_REQS)])
check(gate.returncode == 0,
f"`./run.sh --coverage-only --requirements {','.join(PROOF_REQS)}` exited "
f"{gate.returncode} — the merge gate's own coverage assertion rejects the audit "
f"proof set: {(gate.stderr or gate.stdout).strip()[:400]}")
# ================================================ (c) THE PROOF IS CAPTURABLE AS EVIDENCE
# Use record_run.py's OWN sentinel parser: a Chainsaw-layer proof with no declared pass
# token can pass forever and never be recorded as proof of anything.
sys.path.insert(0, str(LIB))
os.environ.setdefault("PYTHONDONTWRITEBYTECODE", "1")
sys.dont_write_bytecode = True
import record_run as rr # noqa: E402
# status.py owns the latest-wins rule; import it rather than reimplementing it, so the two
# cannot disagree about which rows are live evidence.
import status as _st # noqa: E402
rr_latest_by_test = _st.latest_by_test
smap = rr.sentinel_map(CONF / "tests")
check(len(smap) >= 10,
f"record_run.py's sentinel map contains {len(smap)} test(s) — the parser lost its "
"grip on the test tree, so the assertions below would assert nothing")
for label, spec_proof in sorted(PROOFS.items()):
for rid in spec_proof["reqs"]:
rec = by_req.get(rid) or {}
tests = (rec.get("verification") or {}).get("tests") or []
names = [t.get("name") for t in tests
if t.get("state", "").startswith("executable")
and t.get("layer") in spec_proof["layers"]]
for name in names:
ent = smap.get(name)
check(ent is not None,
f"proof '{label}': record_run.py does not know the test {name!r} — its "
"result could never be attributed to a ledger row")
if not ent:
continue
if ent.get("layer") in rr.SENTINEL_EXEMPT_LAYERS:
continue # offline layers are truthful by JUnit alone
check(bool(ent.get("pass")),
f"proof '{label}': the Chainsaw test {name!r} declares NO pass sentinel, "
"so lib/record_run.py can never record it as proven. It would go green "
"in the terminal and leave the operator with no evidence — exactly the "
"gap the ledger exists to close (docs/07: the proofs are 'part of the "
"platform acceptance suite, not one-time checks')")
check(bool(ent.get("terminal_pass")),
f"proof '{label}': the test {name!r} has no TERMINAL pass token, so "
"record_run.py cannot decide its verdict")
# =============================== (d) THE EVIDENCE CANNOT BE FORGED, NOR HONOURED WHEN STALE
# Both tools' own guards, which NOTHING in ci/ or run.sh drives today.
for tool in ("record_run.py", "status.py"):
r = run([sys.executable, str(LIB / tool), "--selftest"])
check(r.returncode == 0,
f"`lib/{tool} --selftest` exited {r.returncode} — the guards that keep a "
"self-skipped or stale run from being recorded as proof are broken, so nothing "
f"in the evidence chain can be trusted: {(r.stderr or r.stdout).strip()[:500]}")
# ---- FORGERY 1: a green JUnit for a REAL proof test whose log holds only a SKIP token.
# This is the precise shape of a self-skipping L2/L3 test, and it must NOT become proof.
forge_target, forge_skip = None, None
for label, spec_proof in sorted(PROOFS.items()):
for rid in spec_proof["reqs"]:
for name, ent in smap.items():
if rid in (ent.get("requirements") or []) and ent.get("skip") \
and ent.get("layer") not in rr.SENTINEL_EXEMPT_LAYERS:
forge_target, forge_skip = name, ent["skip"][0]
break
if forge_target:
break
if forge_target:
break
check(forge_target is not None,
"no Chainsaw-layer proof test declares a skip token, so the forgery attempt below "
"could not be mounted. That is not a pass: it means this check could not verify "
"that a self-skip is refused as proof, which is the whole point of the ledger")
if forge_target:
run_dir = tmp / "20260101T000000Z-forged"
run_dir.mkdir(parents=True)
(run_dir / "report.junit.xml").write_text(
'<?xml version="1.0" encoding="UTF-8"?>\n'
'<testsuites><testsuite name="chainsaw" tests="1" failures="0" errors="0">'
f'<testcase name="{forge_target}" classname="forged"/>'
"</testsuite></testsuites>\n")
(run_dir / "run.log").write_text(
f"=== RUN chainsaw/{forge_target}\n"
f" === STDOUT\n {forge_skip}\n")
forged = run([sys.executable, str(LIB / "record_run.py"), "--run-dir", str(run_dir),
"--dry-run", "--ledger", str(tmp / "never-written.jsonl"),
"--tests", str(CONF / "tests"), "--repo", str(ROOT)])
verdicts = re.findall(rf"^\[record\]\s+\S+\s+(\S+)\s+{re.escape(forge_target)}\b",
forged.stdout, re.M)
check(verdicts and all(v != "proven" for v in verdicts),
f"lib/record_run.py recorded a FORGED run as {verdicts!r} for {forge_target!r}: "
"a JUnit saying `passed` with nothing in the log but that test's own SKIP token "
f"({forge_skip}). A self-skip must never become proof — with this hole every "
"audit proof in docs/07 could be produced by a cluster that skipped it")
check(not (tmp / "never-written.jsonl").exists(),
"record_run.py --dry-run WROTE to the ledger — a tool that records while "
"claiming not to cannot be trusted to record only real runs")
# ---- FORGERY 2: a ledger row claiming `proven` at a commit that predates the code.
# status.py must degrade it to stale-proof (not green), or an old green run would keep
# vouching for code it never executed.
root_sha = run(["git", "-C", str(ROOT), "rev-list", "--max-parents=0", "HEAD"])
old_sha = (root_sha.stdout.strip().splitlines() or [""])[0]
check(len(old_sha) == 40,
"the repository's root commit could not be resolved, so the stale-proof guard could "
"not be exercised (that is a gap in this check, not a pass)")
if len(old_sha) == 40 and forge_target:
stale_ledger = tmp / "stale.jsonl"
rid = sorted(smap[forge_target].get("requirements") or ["R-NET-06"])[0]
stale_ledger.write_text(json.dumps({
"ts_utc": "2026-01-01T00:00:00Z", "git_sha": old_sha, "env": "dev",
"chainsaw_version": "v0.0.0", "kube_version": "unknown",
"layer": smap[forge_target].get("layer", "L3"), "test": forge_target,
"requirements": [rid], "junit_result": "passed",
"sentinel": (smap[forge_target].get("terminal_pass") or "X"),
"proof": "proven", "run_id": "20260101T000000Z-forged",
"note": "synthetic row written by the R-SEC-03 conformance check",
}) + "\n")
out2 = tmp / "status-stale"
st2 = run([sys.executable, str(LIB / "status.py"), "--repo", str(ROOT),
"--registry", str(CONF / "requirements" / "registry.yaml"),
"--tests", str(CONF / "tests"), "--out", str(out2),
"--ledger", str(stale_ledger)])
check(st2.returncode == 0,
f"lib/status.py failed against a synthetic ledger: "
f"{(st2.stderr or st2.stdout).strip()[:300]}")
s2 = json.loads((out2 / "status.json").read_text()) if (out2 / "status.json").is_file() else {}
rec2 = (s2.get("by_requirement") or {}).get(rid) or {}
per = {p.get("test"): p.get("state") for p in ((rec2.get("proof") or {}).get("per_test") or [])}
check(per.get(forge_target) == "stale-proof",
f"a ledger row claiming `proven` for {forge_target!r} at the repository's ROOT "
f"commit was reported as {per.get(forge_target)!r} instead of 'stale-proof'. A "
"proof from a run that predates the code it vouches for is not evidence, and "
"honouring it would let one old green run certify every later change")
check((rec2.get("proof") or {}).get("state") != "proven",
f"{rid} is reported PROVEN on the strength of a row from the root commit — the "
"freshness guard is not holding")
# ---- the ledger itself is machine-readable, and no row proves a test that is gone ----
ledger = CONF / "runs" / "ledger.jsonl"
if ledger.is_file():
rows, bad = [], []
for i, line in enumerate(ledger.read_text().splitlines(), 1):
if not line.strip():
continue
try:
rows.append(json.loads(line))
except json.JSONDecodeError:
bad.append(i)
check(not bad,
f"conformance/runs/ledger.jsonl has unparsable line(s) {bad} — the audit record "
"is not machine-readable, so no proof can be derived from it")
# Which rows are HONOURED: status.py resolves duplicates latest-wins (latest_by_test),
# so a superseded row is not evidence of anything. The dirty-tree assertion below is
# therefore scoped to honoured rows. Without that scoping, ONE historical dirty row reds
# this gate permanently even after a clean re-run of the same test — and a dirty-then-
# clean sequence is the normal shape of "noticed the tree was dirty, fixed it, re-ran".
# The row is still kept in the ledger and its run.log stays committed: nothing is erased,
# it simply stops being treated as a live claim. Structural assertions (provenance keys,
# sentinel, test-still-exists) still apply to EVERY row.
honoured_ids = {id(v) for v in rr_latest_by_test(rows).values()}
for r in rows:
for key in ("ts_utc", "git_sha", "test", "requirements", "junit_result", "proof",
"run_id"):
check(key in r,
f"a ledger row for {r.get('test', '?')!r} lacks {key!r} — a row without "
"provenance cannot be audited")
if r.get("proof") == "proven":
# L0/L1 rows are recorded proven WITHOUT a sentinel, deliberately, by
# record_run.py's own SENTINEL_EXEMPT_LAYERS: run_l1_offline.py renders skips as
# JUnit <skipped/>, so those layers have no pass token to attribute. This
# assertion had NO layer exemption, so the two halves of the harness contradicted
# each other — ANY `--record` run touching L0 or L1 wrote a row this gate then
# rejected, which meant a requirement whose verifier set includes an L1 check
# (R-NET-06 is one) could never be both PROVEN and L0-green. Honour the same
# constant rather than restating the policy here, so the two cannot drift apart.
if r.get("layer") not in rr.SENTINEL_EXEMPT_LAYERS:
check(bool(r.get("sentinel")),
f"a ledger row claims `proven` for {r.get('test')!r} with NO sentinel — "
"the sentinel is the only thing separating a real pass from a self-skip")
check(r.get("test") in smap,
f"a ledger row claims `proven` for {r.get('test')!r}, which is not a "
"test in the suite any more — a proof of something that no longer exists")
check(id(r) not in honoured_ids or "-dirty" not in str(r.get("git_sha", "")),
f"a ledger row claims `proven` for {r.get('test')!r} from a DIRTY tree "
f"({r.get('git_sha')}) — what ran is not what is committed")
else:
note("conformance/runs/ledger.jsonl does not exist yet — the ledger-shape assertions "
"were skipped; every other assertion ran, and the forgery probes above use "
"synthetic ledgers, not this file")
finally:
shutil.rmtree(tmp, ignore_errors=True)
# --------------------------------------------------------------------------- verdict
# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
check(CHECKS >= 30,
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
"would be meaningless")
if FAILURES:
die()
for n in NOTES:
print(f" note: {n}")
print(f"check OK: auditability-proofs-producible — {CHECKS} assertions; the five docs/07 "
f"proofs ({', '.join(sorted(PROOFS))}) each map to an executable test at the layer the "
f"spec's own wording requires, `run.sh --requirements {','.join(PROOF_REQS)}` resolves "
"and asserts them, each Chainsaw-layer proof declares a sentinel record_run.py can "
"attribute, both evidence tools' self-tests pass, and two independent forgeries — a "
"self-skip presented as a pass, and a `proven` row from the root commit — were REFUSED. "
"R-SEC-03's producibility half only: whether each proof PASSES is its own "
"requirement's test, and the Hubble half of the isolation proof is automated nowhere "
"(see the header's gap list).")
PY
@@ -0,0 +1,114 @@
# =============================================================================
# L0 STATIC — the bulk tier's capacity strategy is DISTINCT from the replicated tier's
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# THE BUG CLASS THIS CLOSES. The `bulk` StorageClass existed from Phase 0 as a stub:
# `provisioner: nfs.csi.k8s.io`, `server: REPLACE-ME`, `share: REPLACE-ME`, TODO(OQ-2).
# Four ServiceDefinitions carried a comment promising to attach their media once OQ-2 was
# resolved, and the ONLY thing keeping a multi-terabyte library off the 3-replica
# `longhorn` class was those four comments. A bulk volume that lands on the replicated
# tier does not fail loudly — Longhorn accepts the PVC and starts replicating 3x of data
# three mini-PCs cannot hold. R-STATE-04 is precisely that the two tiers are distinct, so
# the distinctness is asserted rather than reviewed.
#
# It also pins the four decisions that make the STATIC form of this tier safe, each with a
# plausible "simplification" that breaks something silently: `Retain` (bulk data pre-exists
# the cluster and has consumers outside it, so anything else makes `delete pvc` a data-loss
# operation); NO dynamic provisioning (a dynamic bulk class hands out a FRESH EMPTY
# directory that mounts and Binds perfectly while the real 8 TiB is never attached — the
# most deceptive failure available here); `hard` not `soft` (a soft mount hands a server
# hiccup to the app as a short read, which for a media library is silent corruption); and
# NFSv4 (mount and lock in-protocol ⇒ ONE reachable port, tcp/2049, which is what the
# operator must open in pfSense for prod to reach this server at all).
#
# AND the EXDEV trap, which is not about storage. Sonarr/Radarr import by hardlinking or
# atomically renaming, both same-filesystem operations, and to the kernel "same filesystem"
# means SAME MOUNT. Splitting one export into two subPath mounts looks TIGHTER and breaks
# link() with EXDEV; the arrs then degrade to copy-then-delete, rewriting every import over
# NFS and destroying the seeding copy. It surfaces weeks later as "imports are slow and my
# ratio is gone". A reviewer cannot see that; this check can.
#
# GIVEN the repository as authored,
# WHEN every `class: bulk` storage entry in catalog/ is compared against the objects the
# COMMITTED apps/ goldens carry for it (what Flux reconciles — deliberately not the
# renderer, which would only prove it agrees with itself),
# THEN (1) no bulk volume is on a Longhorn-provisioned class, and its class is explicitly
# EMPTY so nothing can dynamically provision an empty stand-in;
# (2) its capacity is declared and is at least an order of magnitude above the same
# service's replicated volume — the requirement stated as arithmetic;
# (3) PV and PVC are pre-bound from BOTH ends (claimRef + volumeName) and agree;
# (4) the PV reclaim policy is Retain;
# (5) mountOptions carry `hard` and exactly one `nfsvers=4.x`, and never `soft`;
# (6) no workload mounts one export more than once (the EXDEV trap);
# (7) a readOnly entry renders nfs.readOnly AND a readOnly volumeMount AND
# ReadOnlyMany; a writable one renders neither and is ReadWriteMany; every bulk
# access mode is a `...Many`, since one export serves several namespaces and
# several nodes;
# (8) the PV's nfs.server is a BARE cluster-vars token (never a literal — dev and
# prod reach this host by different addresses), that knob has a base default, and
# EVERY overlay supplies it;
# AND (9) the check is LOAD-BEARING (ci/l0.sh runs it unconditionally, ci/gate.sh blocks
# on the l0 stage) and NON-VACUOUS (an empty bulk corpus FAILS rather than
# passing trivially, and a floor on the assertion count keeps a pass from
# degenerating into a no-op).
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library +
# PyYAML); conformance/run.sh --layer l0 runs it directly and this Chainsaw Test wraps the
# SAME script.
#
# NOT proven — the runtime half, which needs a cluster: that the server is reachable, that
# the export admits these clients, that the PVC binds, that a write succeeds, or that a
# hardlink between the library and the download path actually works. The uid analysis is
# reasoning (docs/BULK-NFS-STORAGE.md §6) and the in-cluster probes that would settle it
# are written down in §9 of that document, unrun. Nor can this prove prod's address is
# CORRECT — only that it is a supplied knob; while it is a placeholder, prod's pods will
# not mount, and catching that statically needs the placeholder guard, which is absent from
# main (§7). Read-only on the media tree is a CLIENT-side guarantee only: the export serves
# one reader and two writers from the same node addresses, so the server cannot tell them
# apart (§5).
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: bulk-tier-distinct-from-replicated
labels:
conformance.home.k3s/R-STATE-04: "true"
annotations:
conformance.home.k3s/requirements: "R-STATE-04"
conformance.home.k3s/layer: "L0"
# WHOLE-REPO SCOPE — relocated verbatim from ci/l0.sh's hand-written "Stage 3e
# (UNCONDITIONAL)", which this annotation replaces:
#
# The `bulk` StorageClass sat as a REPLACE-ME stub for six roadmap phases, and the only
# thing keeping a multi-terabyte media library off the 3-replica `longhorn` class was four
# hand-written comments in ServiceDefinitions. A bulk volume that lands on the replicated
# tier does not fail loudly — Longhorn accepts the PVC and starts replicating 3x of data
# three mini-PCs cannot hold.
#
# WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh's path map keys
# NEITHER apps/* nor platform/longhorn/* to the bulk requirement. So the change most
# likely to break this — moving a bulk volume onto the replicated tier, or narrowing one
# export into two mounts — could attribute no requirement at all and be gated by nothing.
# The invariant spans catalog/, apps/ and clusters/.
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Bulk media storage is provably NOT on the 3-replica Longhorn tier, is sized for a
different order of magnitude, cannot be silently replaced by a dynamically
provisioned empty volume, is never reclaimable by the cluster, mounts hard over
NFSv4, and is never split into two mounts of one export — so hardlinked imports keep
working. Its server is a per-environment knob, not a literal, because the two
clusters reach the same storage host by different addresses.
spec:
steps:
- name: assert-bulk-tier-is-distinct-and-statically-bound
description: >-
Read every `class: bulk` storage entry from catalog/, pair it with the
PersistentVolume, PersistentVolumeClaim and volumeMount the committed apps/ goldens
carry for it, and assert the tier separation, the static pre-binding, the reclaim
and mount-option decisions, the read-only coupling, the one-export-one-mount
invariant, and that the server address is supplied by every environment overlay.
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,496 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — bulk-tier-distinct-from-replicated
# Serves: R-STATE-04 ("Bulk media storage has a capacity strategy distinct from the
# replicated tier." — spec docs/04-storage-and-state.md)
#
# WHY THIS EXISTS — THE BULK TIER WAS A STUB FOR SIX ROADMAP PHASES, AND STUBS ROT QUIETLY
# ----------------------------------------------------------------------------------------
# `platform/longhorn/storageclasses.yaml` declared a `bulk` StorageClass from Phase 0 with
# `provisioner: nfs.csi.k8s.io`, `server: REPLACE-ME`, `share: REPLACE-ME` and a TODO(OQ-2).
# Four ServiceDefinitions carried a comment promising to attach their media "once OQ-2 is
# resolved", and each of them said, correctly, that adding `class: bulk` before then would
# leave the PVC unbound and the pod Pending. So the tier existed as a NAME with nothing
# behind it, and the only thing preventing a multi-terabyte media library from being
# authored onto the 3-replica `longhorn` class was four hand-written comments.
#
# That is the failure this check makes impossible to reintroduce. R-STATE-04 is not "bulk
# storage exists" — it is that bulk storage's capacity strategy is DISTINCT from the
# replicated tier's. Three mini-PCs cannot hold 3x of 8.33 TiB. A bulk volume that lands on
# `longhorn` does not fail loudly: Longhorn accepts the PVC and starts replicating, and the
# tier fills up. Distinctness is therefore a property worth asserting statically, on every
# gate run, rather than trusting to review.
#
# It also pins the four decisions that make the STATIC form of this tier safe, each of
# which has a plausible-looking "simplification" that silently breaks something:
#
# * RECLAIM POLICY. Bulk data pre-exists the cluster, is larger than the cluster, and has
# consumers OUTSIDE it. `Retain` is the whole reason a PV is used here instead of an
# inline volume. `Delete` on a dynamically-provisioned bulk class would make `kubectl
# delete pvc` a data-loss operation.
# * NO DYNAMIC PROVISIONING. A dynamic bulk class hands out FRESH EMPTY directories. The
# pod mounts successfully, the PVC is Bound, everything is green — and the 8 TiB of
# existing data was never attached. This is the single most deceptive failure available
# in this tier, so `storageClassName` must be explicitly empty and binding must be
# pinned from BOTH ends (claimRef + volumeName).
# * `hard`, NOT `soft`. A soft NFS mount converts a server hiccup into a short read or
# EIO handed to the application. For a media library that is silent corruption. `hard`
# hangs instead — recoverable, where truncated is not.
# * NFSv4. v4 carries mount and lock in-protocol, so the tier needs ONE reachable port
# (tcp/2049) instead of v3's rpcbind/111 + mountd + statd on ephemeral numbers. That is
# a FIREWALL fact as much as a storage one: prod reaches this server through pfSense,
# and the number of ports the operator must open is a consequence of this field.
#
# AND ONE INVARIANT THAT IS NOT ABOUT STORAGE AT ALL — the EXDEV trap (assertion 6).
# Sonarr and Radarr import by hardlinking or atomically renaming a finished download into
# the library. Both are same-filesystem operations, and to the kernel "same filesystem"
# means SAME MOUNT. Mounting one export twice (say `subPath: tv` and `subPath: downloads`)
# looks TIGHTER — less of the tree exposed per mount — and breaks `link()` with EXDEV no
# matter that the server knows they are one dataset. The arrs do not fail loudly on that:
# they degrade to copy-then-delete, rewriting every imported file over NFS and destroying
# the seeding copy. The symptom appears weeks later as "imports are slow and my ratio is
# gone". A reviewer cannot see it, so this check does: one export, at most one mount, per
# workload.
#
# GIVEN the repository as authored,
# WHEN every `class: bulk` storage entry in catalog/ is compared against the manifests
# render.py produces for it (read from the committed apps/ goldens, i.e. what Flux
# actually reconciles — not from the renderer, which would only agree with itself),
# THEN (1) DISTINCTNESS: no bulk volume is on the replicated tier. Its PV/PVC
# storageClassName is explicitly empty — never `longhorn`, never any class the
# repo declares with the Longhorn provisioner — and its PV's source is an
# EXTERNAL one (nfs), not a Longhorn volume;
# (2) CAPACITY STRATEGY: every bulk PV declares a capacity, and for each service the
# bulk volume is at least an ORDER OF MAGNITUDE larger than that service's own
# replicated volume. This is the requirement stated as arithmetic: the two tiers
# exist because the data has two magnitudes, so a "bulk" volume sized like a
# config PVC means the strategy was named and not actually applied;
# (3) NOT DYNAMICALLY PROVISIONED, AND PRE-BOUND BOTH WAYS: every bulk PVC names its
# PV by volumeName, every bulk PV names its PVC by claimRef (namespace+name), and
# the two agree — so no other claim can win a race, and nothing can provision an
# empty replacement;
# (4) RECLAIM: every bulk PV is `Retain`;
# (5) MOUNT SAFETY: every bulk PV declares mountOptions carrying `hard` and an
# `nfsvers=4.x`, and carrying NEITHER `soft` NOR an `nfsvers=3`;
# (6) ONE EXPORT, ONE MOUNT PER WORKLOAD (the EXDEV trap above);
# (7) READ-ONLY IS REAL AND CONSISTENT: a bulk entry declared readOnly renders BOTH
# `spec.nfs.readOnly: true` on the PV AND `readOnly: true` on the volumeMount,
# and its access mode is ReadOnlyMany; a writable one renders NEITHER and is
# ReadWriteMany. Every bulk access mode is a `...Many` — one export is consumed
# by several namespaces and several nodes, which ReadWriteOnce cannot express;
# (8) AUTHORED ONCE, PROMOTED (the R-ENV-02 seam this tier depends on): every bulk
# PV's `nfs.server` is the BARE cluster-vars token, never a literal address; that
# knob has a base default; and EVERY environment overlay supplies it — so the two
# clusters, which reach this storage by different addresses, cannot be served by
# one hardcoded value;
# AND (9) LOAD-BEARING + NON-VACUOUS: this check DECLARES `conformance.home.k3s/scope:
# whole-repo`, so ci/l0.sh runs it unconditionally via --include-scope, and
# ci/gate.sh blocks the merge when the l0 stage fails; and the corpus is NOT
# empty — a repo with zero bulk volumes fails here rather than passing trivially,
# because a check that is satisfiable by deleting the feature proves nothing.
#
# WHAT THIS DOES *NOT* PROVE (honest gaps)
# - Nothing about RUNTIME. Not that the server is reachable, not that the export permits
# these clients, not that the PVC binds, not that a write succeeds. The uid analysis in
# docs/BULK-NFS-STORAGE.md §6 is reasoning, and the write/hardlink probes in §9 are the
# in-cluster proof — they need a cluster and are not run here.
# - It cannot prove the prod address is CORRECT, only that it is a supplied knob. While
# prod's value is a placeholder, prod's pods will not mount (docs/BULK-NFS-STORAGE.md
# §7); catching that needs the placeholder guard, which is absent from main.
# - Read-only for the media tree is a CLIENT-side guarantee, and this check can only
# assert the client side. The export serves a reader and two writers from the same node
# addresses, so the server cannot distinguish them (docs/BULK-NFS-STORAGE.md §5).
# - Assertion (2)'s magnitude ratio is a heuristic with a deliberately generous floor. It
# catches "bulk volume sized like a config volume", not a subtly wrong capacity — and a
# static NFS PV's capacity is advisory anyway, since the ZFS pool is the real limit.
# - Assertion (6) reasons over ONE workload's mounts. It cannot see two different
# services hardlinking between each other's mounts, which is not a thing that happens.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
cd "${ROOT}"
if [ ! -d "${ROOT}/catalog" ] || [ ! -d "${ROOT}/apps" ]; then
echo "check SKIPPED (bulk-tier-distinct-from-replicated): catalog/ or apps/ absent — nothing to audit" >&2
exit 3
fi
"${PYTHON}" - "${ROOT}" <<'PY'
import re
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
FAILURES: list[str] = []
CHECKS = 0
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
def bail(msg):
"""A failure that makes the rest of the check meaningless: report and stop."""
print(f"CHECK FAIL (bulk-tier-distinct-from-replicated): {msg}", file=sys.stderr)
raise SystemExit(1)
def qty_to_bytes(q):
"""Kubernetes quantity -> bytes. Only the binary suffixes the schema permits."""
m = re.fullmatch(r"([0-9]+)(Mi|Gi|Ti)", str(q).strip())
if not m:
return None
return int(m.group(1)) * {"Mi": 2 ** 20, "Gi": 2 ** 30, "Ti": 2 ** 40}[m.group(2)]
# =====================================================================================
# The replicated tier, named by the repo rather than by this file. Any StorageClass the
# repo declares with the Longhorn provisioner is "the replicated tier" for assertion (1);
# deriving it means adding a new Longhorn class cannot create a blind spot here.
# =====================================================================================
REPLICATED_CLASSES: set[str] = set()
LONGHORN_DIR = ROOT / "platform" / "longhorn"
if LONGHORN_DIR.is_dir():
for f in sorted(LONGHORN_DIR.rglob("*.yaml")):
for doc in yaml.safe_load_all(f.read_text()):
if isinstance(doc, dict) and doc.get("kind") == "StorageClass":
if "longhorn" in str(doc.get("provisioner", "")):
REPLICATED_CLASSES.add(doc.get("metadata", {}).get("name", ""))
# `longhorn` is the name every ServiceDefinition uses for the replicated tier, so it is in
# the set even if platform/longhorn/ is not present in this checkout.
REPLICATED_CLASSES.add("longhorn")
REPLICATED_CLASSES.discard("")
# =====================================================================================
# The bulk corpus: every `class: bulk` storage entry in every ServiceDefinition, paired
# with the objects the COMMITTED goldens carry for it. Reading the goldens (not calling
# the renderer) is deliberate: the goldens are what Flux reconciles, and render==golden is
# a separate gate stage. Asking the renderer would only prove it agrees with itself.
# =====================================================================================
sds: dict[str, dict] = {}
for f in sorted((ROOT / "catalog").glob("*.yaml")):
doc = yaml.safe_load(f.read_text())
if isinstance(doc, dict) and doc.get("name"):
sds[doc["name"]] = doc
if not sds:
bail("no ServiceDefinitions under catalog/ — cannot audit the bulk tier")
# Every rendered app-tier document, indexed by (kind, name).
rendered: dict[tuple, dict] = {}
for f in sorted((ROOT / "apps").rglob("*.yaml")):
if f.name == "kustomization.yaml" or ".sops." in f.name:
continue
try:
docs = list(yaml.safe_load_all(f.read_text()))
except yaml.YAMLError:
continue
for doc in docs:
if isinstance(doc, dict) and doc.get("kind") and doc.get("metadata", {}).get("name"):
rendered[(doc["kind"], doc["metadata"]["name"])] = doc
# (svc, vol_entry) for every bulk entry, and the replicated sizes alongside for (2).
corpus: list[tuple[str, dict]] = []
replicated_bytes: dict[str, list[int]] = {}
for svc, sd in sds.items():
for vol in sd.get("storage") or []:
cls = vol.get("class", "longhorn")
if cls == "bulk":
corpus.append((svc, vol))
else:
b = qty_to_bytes(vol.get("size"))
if b:
replicated_bytes.setdefault(svc, []).append(b)
# ------------------------------------------------------------------- (9) NON-VACUOUS
# A corpus of zero would make every loop below trivially satisfied. The bulk tier is
# DELIVERED; a change that removes the last bulk volume must fail here and be argued for,
# not sail through on an empty iteration.
check(len(corpus) > 0,
"NO `class: bulk` storage entry exists in catalog/ — this check would then pass "
"vacuously, which is exactly the posture that let the bulk tier sit as a REPLACE-ME "
"stub for six roadmap phases. If the tier is genuinely being retired, retire this "
"check in the same change and say so; do not let it pass on an empty corpus")
if not corpus:
bail("empty bulk corpus — refusing to report a vacuous pass (see assertion 9)")
inventory: list[str] = []
for svc, vol in corpus:
vname = vol.get("name", "<unnamed>")
label = f"{svc}/{vname}"
obj = f"{svc}-{vname}"
ro = bool(vol.get("readOnly"))
# `export` is what the schema requires of a bulk entry; without it there is nothing to
# attach and the rest of the assertions have no subject.
export = vol.get("export")
check(bool(export) and str(export).startswith("/"),
f"{label}: bulk entry has no absolute `export` path — a bulk volume attaches an "
f"EXISTING dataset, so it must say which one (schema requires it when class is bulk)")
pv = rendered.get(("PersistentVolume", obj))
pvc = rendered.get(("PersistentVolumeClaim", obj))
check(pv is not None,
f"{label}: no PersistentVolume '{obj}' in the committed apps/ goldens. A bulk "
f"entry that renders no PV has NO external storage behind it — the pod would get "
f"an unbound PVC, or (worse, with a dynamic class) a fresh EMPTY directory")
check(pvc is not None,
f"{label}: no PersistentVolumeClaim '{obj}' in the committed apps/ goldens")
if pv is None or pvc is None:
continue
pvs, pvcs = pv.get("spec", {}) or {}, pvc.get("spec", {}) or {}
# ------------------------------------------------------ (1) DISTINCT FROM REPLICATED
for what, spec in (("PV", pvs), ("PVC", pvcs)):
sc = spec.get("storageClassName")
check(sc not in REPLICATED_CLASSES,
f"{label}: bulk {what} storageClassName is '{sc}', which the repo declares "
f"with the Longhorn provisioner. That puts BULK data on the REPLICATED tier "
f"— 3x of a multi-terabyte library across three mini-PCs. Longhorn will not "
f"refuse it; it will accept the PVC and fill the tier (R-STATE-04)")
check(sc == "",
f"{label}: bulk {what} storageClassName is {sc!r}, not the explicitly EMPTY "
f"string. Empty is what disables dynamic provisioning AND the default-class "
f"annotation. Any named class risks a provisioner handing out a fresh EMPTY "
f"directory that mounts and binds cleanly while the real data is never attached")
check("nfs" in pvs,
f"{label}: bulk PV declares no external `nfs` source — the bulk tier's whole "
f"premise is that durability lives OUTSIDE the cluster (docs/04)")
# ------------------------------------------------------------ (4) RECLAIM POLICY
check(pvs.get("persistentVolumeReclaimPolicy") == "Retain",
f"{label}: bulk PV reclaim policy is "
f"{pvs.get('persistentVolumeReclaimPolicy')!r}, not Retain. This data pre-exists "
f"the cluster and has consumers outside it; anything but Retain makes `kubectl "
f"delete pvc` a data-loss operation")
# ------------------------------------------------- (3) PRE-BOUND FROM BOTH ENDS
check(pvcs.get("volumeName") == obj,
f"{label}: bulk PVC does not name its PV via volumeName (got "
f"{pvcs.get('volumeName')!r}, expected {obj!r}) — without it the claim can be "
f"satisfied by some OTHER volume of matching size and mode")
cref = pvs.get("claimRef") or {}
check(cref.get("name") == obj and cref.get("namespace") == svc,
f"{label}: bulk PV claimRef is {cref!r}, expected namespace={svc!r} name={obj!r} "
f"— pre-binding must be pinned from BOTH ends or another PVC can win the race")
# ------------------------------------------------------------- (5) MOUNT SAFETY
mopts = [str(o) for o in (pvs.get("mountOptions") or [])]
check(bool(mopts),
f"{label}: bulk PV declares no mountOptions — then `hard` vs `soft` and the NFS "
f"version are whatever the client happens to default to, and both are decisions "
f"with consequences (silent short reads; how many firewall ports must open)")
check("hard" in mopts,
f"{label}: bulk PV mountOptions do not include `hard` ({mopts}). A soft mount "
f"hands a server hiccup to the application as a short read or EIO — silent "
f"corruption for a media library. `hard` hangs instead, which is recoverable")
check("soft" not in mopts,
f"{label}: bulk PV mountOptions include `soft` ({mopts}) — see above; this is the "
f"one option that turns an outage into damaged data")
vers = [o for o in mopts if o.startswith("nfsvers=")]
check(len(vers) == 1 and re.fullmatch(r"nfsvers=4(\.[0-9]+)?", vers[0]),
f"{label}: bulk PV must pin exactly one NFSv4 `nfsvers=` option (got {vers}). "
f"NFSv4 carries mount and lock in-protocol, so the tier needs ONE reachable port "
f"(tcp/2049); v3 also needs rpcbind/111, mountd and statd, which changes what the "
f"operator has to open in pfSense for prod to reach this server at all")
# ---------------------------------------------- (7) READ-ONLY IS REAL + CONSISTENT
want_mode = "ReadOnlyMany" if ro else "ReadWriteMany"
for what, spec in (("PV", pvs), ("PVC", pvcs)):
modes = spec.get("accessModes") or []
check(modes == [want_mode],
f"{label}: bulk {what} accessModes {modes} != ['{want_mode}']. Every bulk "
f"volume is a `...Many` (one export, several namespaces, several nodes — "
f"ReadWriteOnce cannot express that), and WHICH `Many` is how the read/write "
f"posture stays legible in `kubectl get pv`")
nfs_ro = bool((pvs.get("nfs") or {}).get("readOnly"))
check(nfs_ro == ro,
f"{label}: ServiceDefinition readOnly={ro} but the PV's nfs.readOnly={nfs_ro}. "
f"The PV field is what makes the kernel mount `ro`; a mismatch means the declared "
f"posture is not the one the kernel enforces")
# The volumeMount half, read out of the rendered Deployment.
dep = rendered.get(("Deployment", svc))
check(dep is not None, f"{label}: no rendered Deployment '{svc}' to check the mount against")
if dep is not None:
containers = (((dep.get("spec") or {}).get("template") or {}).get("spec") or {}).get("containers") or []
# The app container is containers[0] by the renderer's contract (a forward-auth
# sidecar is appended AFTER it).
mounts = [m for m in (containers[0].get("volumeMounts") or []) if m.get("name") == vname] if containers else []
check(len(mounts) == 1,
f"{label}: the app container has {len(mounts)} volumeMount(s) named {vname!r}, "
f"expected exactly 1 — a bulk volume that is declared and never mounted is a "
f"PV/PVC pair doing nothing, and two mounts of one volume is assertion (6)")
if len(mounts) == 1:
m = mounts[0]
check(bool(m.get("readOnly")) == ro,
f"{label}: volumeMount readOnly={bool(m.get('readOnly'))} but the "
f"ServiceDefinition says readOnly={ro} — the PV and the mount must agree, "
f"so that dropping one still leaves the other")
check(m.get("subPath", vol.get("subPath")) == vol.get("subPath"),
f"{label}: volumeMount subPath {m.get('subPath')!r} does not match the "
f"declared subPath {vol.get('subPath')!r}")
# ------------------------------------------ (8) AUTHORED ONCE, SUPPLIED PER ENV
server = (pvs.get("nfs") or {}).get("server")
m = re.fullmatch(r"\$\{([A-Z0-9_]+)\}", str(server or ""))
check(m is not None,
f"{label}: bulk PV nfs.server is {server!r} — it must be a BARE cluster-vars "
f"token, not a literal address. dev and prod reach this storage host by DIFFERENT "
f"addresses (dev is on its LAN; prod is across pfSense), so a literal is correct "
f"in at most one environment and silently wrong in the other (R-ENV-02)")
cap = (pvs.get("capacity") or {}).get("storage")
req = ((pvcs.get("resources") or {}).get("requests") or {}).get("storage")
check(cap is not None and qty_to_bytes(cap) is not None,
f"{label}: bulk PV declares no parseable capacity ({cap!r}) — the API requires it, "
f"and it is what makes `kubectl get pv` report something true about this tier")
check(cap == req,
f"{label}: PV capacity {cap!r} != PVC request {req!r}; a request above the "
f"capacity never binds")
inventory.append(
f" {label:<28} export={export} {'ro' if ro else 'rw'} {want_mode:<14} "
f"cap={cap} server={server} opts={','.join(mopts)}"
)
# ---------------------------------- (2) CAPACITY STRATEGY: two tiers, two magnitudes
cap_b = qty_to_bytes(cap)
reps = replicated_bytes.get(svc) or []
if cap_b and reps:
biggest = max(reps)
# ONE ORDER OF MAGNITUDE. Calibrated against the real corpus rather than guessed:
# the media library is ~460x its service's config volume, and the Nextcloud archive
# ~13x its 20Gi Longhorn volume — so a floor above ~13x would fire on a legitimate
# bulk volume, and this check would be teaching people to weaken it. 10x cannot be
# met by accident and still catches the regression that matters: a "bulk" volume
# sized like a config PVC, i.e. the strategy named but never applied.
check(cap_b >= 10 * biggest,
f"{label}: bulk capacity {cap} is less than 10x this service's largest "
f"replicated volume ({biggest // 2 ** 30}Gi). R-STATE-04 exists because the "
f"two tiers hold data of two different MAGNITUDES — the replicated tier keeps "
f"the small config/DB and the bulk tier keeps what cannot be replicated 3x. A "
f"bulk volume sized like a config volume means the tier was named, not applied")
# ------------------------------------------------- (6) ONE EXPORT, AT MOST ONE MOUNT
# The EXDEV trap. Grouped per SERVICE, because the hazard is two mounts inside ONE pod.
by_service: dict[str, list[dict]] = {}
for svc, vol in corpus:
by_service.setdefault(svc, []).append(vol)
for svc, vols in sorted(by_service.items()):
seen: dict[str, str] = {}
for vol in vols:
exp = vol.get("export")
prev = seen.get(exp)
check(prev is None,
f"{svc}: mounts the export {exp} TWICE (volumes {prev!r} and "
f"{vol.get('name')!r}). Two mounts of one export are two FILESYSTEMS to the "
f"kernel, so link() between them fails EXDEV even though the server knows "
f"they are one dataset. Sonarr/Radarr do not fail loudly on that — they "
f"degrade to copy-then-delete, rewriting every imported file over NFS and "
f"destroying the seeding copy. Mount the common ancestor ONCE instead "
f"(docs/BULK-NFS-STORAGE.md §4)")
seen[exp] = vol.get("name")
# ------------------------------------- (8b) the knob has a base default AND every overlay
tokens = set()
for _svc, vol in corpus:
pv = rendered.get(("PersistentVolume", f"{_svc}-{vol.get('name')}"))
if pv:
mm = re.fullmatch(r"\$\{([A-Z0-9_]+)\}", str(((pv.get("spec") or {}).get("nfs") or {}).get("server") or ""))
if mm:
tokens.add(mm.group(1))
cv = ROOT / "clusters" / "base" / "cluster-vars.yaml"
base_keys: set[str] = set()
if cv.is_file():
doc = yaml.safe_load(cv.read_text()) or {}
base_keys = set((doc.get("data") or {}).keys())
overlays = sorted(
d for d in (ROOT / "clusters").iterdir()
if cv.is_file() and d.is_dir() and d.name != "base" and (d / "kustomization.yaml").is_file()
) if (ROOT / "clusters").is_dir() else []
for tok in sorted(tokens):
check(tok in base_keys,
f"bulk PVs reference ${{{tok}}} but clusters/base/cluster-vars.yaml declares no "
f"such knob. Under StrictPostBuildSubstitutions that fails the whole tier; without "
f"it the token ships into the cluster as a literal server name")
for ov in overlays:
text = (ov / "kustomization.yaml").read_text()
check(f"/data/{tok}" in text,
f"clusters/{ov.name} does not supply ${{{tok}}}. The base default is the DEV "
f"value by this repo's convention, so an overlay that omits it silently "
f"inherits an address on the WRONG SIDE of pfSense — a mount that either "
f"never connects or connects to something unintended (R-ENV-02)")
# --------------------------------------------------------------- (9) LOAD-BEARING
L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism.
# It used to assert `"bulk-tier-distinct-from-replicated/check.sh" in L0`: ci/l0.sh carried a hand-written
# "UNCONDITIONAL" stage per whole-repo check, hardcoding this path. Those stages are gone
# (they did not scale, and the next whole-repo invariant only got one if someone REMEMBERED
# to write it). The same property is now asserted two ways: this test DECLARES
# `conformance.home.k3s/scope: "whole-repo"`, and ci/l0.sh unions every declaring test into
# the run via `--include-scope`. Neither half alone is enough, so both are checked here; the
# mechanism ITSELF (the union really selects, and a scope-declared SKIP exits 3 rather than
# passing) is proven end-to-end and falsifiably by
# conformance/tests/l0-static/whole-repo-invariants-wired.
SELF_YAML = ROOT / "conformance" / "tests" / "l0-static" / "bulk-tier-distinct-from-replicated" / "chainsaw-test.yaml"
SELF = SELF_YAML.read_text() if SELF_YAML.is_file() else ""
check(re.search(r'^\s*conformance\.home\.k3s/scope:\s*["\']?whole-repo', SELF, re.M) is not None,
"bulk-tier-distinct-from-replicated/chainsaw-test.yaml does not declare "
'conformance.home.k3s/scope: "whole-repo", so the merge gate selects this check only '
"when a change happens to touch one of the requirements it is labelled with. "
"The path map keys NEITHER apps/* nor platform/longhorn/* to the bulk requirement, so moving a multi-terabyte bulk volume onto the 3-replica tier could attribute no requirement at all and be gated by nothing")
check("--include-scope" in L0,
"ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is "
"a purely requirement-scoped run again and the scope annotation is inert decoration — "
"this check would then run only by coincidence")
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
"ci/gate.sh does not set the merge block when the l0 stage fails")
# ------------------------------------------------------------------------- report
print("Bulk tier inventory (from catalog/ + the committed apps/ goldens):")
for line in inventory:
print(line)
print(f"\nreplicated tier StorageClasses (derived from platform/longhorn/): "
f"{', '.join(sorted(REPLICATED_CLASSES))}")
print(f"bulk volumes: {len(corpus)} distinct exports: "
f"{len({v.get('export') for _s, v in corpus})} "
f"server knob(s): {', '.join(sorted(tokens)) or 'NONE'}")
# A floor on the assertion count, for the same reason the corpus may not be empty: a pass
# must not be able to degenerate into "nothing was actually asked".
MIN_CHECKS = 40
check(CHECKS >= MIN_CHECKS,
f"only {CHECKS} assertions ran (floor {MIN_CHECKS}) — a pass this thin is not "
f"evidence; the corpus or the extraction has degraded")
print()
if FAILURES:
print(f"CHECK FAIL (bulk-tier-distinct-from-replicated): {len(FAILURES)} finding(s)",
file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
print(f"check OK: bulk-tier-distinct-from-replicated — {CHECKS} assertions (R-STATE-04)")
PY
@@ -0,0 +1,83 @@
# =============================================================================
# L0 STATIC — dev/test nodes are provisioned as code, end to end (R-ENV-01) [EXECUTABLE]
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
# Spec: docs/00 R-ENV-01 ("dev/test MUST be provisioned as local KVMs via IaC:
# created, baselined, cluster-joined, and managed as code"); the four stages
# are enumerated in docs/02 §"Node lifecycle stages".
#
# WHY L0: R-ENV-01 is a requirement about HOW the dev nodes came to exist, and a live
# Node object cannot answer it — a node built by OpenTofu and one built by hand in a
# hypervisor UI are indistinguishable through the Kubernetes API. What is provable, with
# no cluster, is that a COMPLETE and EXECUTABLE provisioning path exists for all four
# stages and covers EVERY dev node. For the Ansible half that is proven by DRIVING the
# real toolchain offline (`--syntax-check`, `--list-hosts` — neither opens a connection
# to any node), not by reading files: a chain that does not resolve fails the check.
#
# ENVIRONMENT-AGNOSTIC: no hostname, address, prefix or node count appears in the test.
# The dev group names are read from the playbooks' own default host patterns, the node
# names are generated from the IaC module's prefix/count variables, and the two sets are
# compared against each other — so the assertion survives a rename or a resize.
#
# GIVEN the repository as git tracks it,
# WHEN the provisioning chain is parsed and (where offline execution is possible) driven,
# THEN the IaC module creates the dev nodes on a LOCAL KVM hypervisor (allowlisted
# provider — a cloud provider fails), from a for_each map rather than a count index,
# each with CPU/RAM/OS disk/dedicated Longhorn data disk/NIC/cloud-init seed; the
# node set it creates is EXACTLY the set Ansible manages; the baseline stage applies
# the package floor, kernel modules, sysctls, data-disk mount, swap-off and iscsid
# declaratively; one k3s role founds the cluster with --cluster-init and joins the
# rest with --server while disabling flannel/kube-proxy/traefik/servicelb at
# install; and site.yml's chain (baseline -> k3s -> Cilium -> verify -> Flux)
# resolves and targets every dev node at every stage.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git, PyYAML, and —
# where present — ansible-playbook in no-connection modes; no cluster, no network);
# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test
# wraps the SAME script.
#
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
# * That the RUNNING dev VMs were created by this module — needs the OpenTofu state or
# hypervisor access, neither available offline.
# * `tofu validate` — needs `tofu init` (provider registry), i.e. network. The module is
# parsed, not evaluated; a network-allowed CI stage would close this.
# * "Never hand-edited on the box" (docs/02 stage 4's second half) — a live drift check.
# * REBUILD/re-converge (docs/02's own proof line) — exercised at L3 by
# tests/l3-fault-injection/node-loss (R-STATE-02).
# * docs/02 prose puts the baseline in cloud-init; the implementation moved it into the
# Ansible node_baseline role. R-ENV-01 requires the node to be "baselined ... as code",
# not that cloud-init be the tool, so the check asserts the STAGE and RECORDS the
# divergence rather than hiding it.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: dev-nodes-provisioned-as-code
labels:
conformance.home.k3s/R-ENV-01: "true"
annotations:
conformance.home.k3s/requirements: "R-ENV-01"
conformance.home.k3s/layer: "L0"
# NO `scope` ANNOTATION, DELIBERATELY — this invariant IS path-scoped, and was audited as
# such (2026-09-12). Its entire input surface is infra/ (OpenTofu + the Ansible roles that
# provision the dev nodes); it reads nothing else. The gap was that
# ci/touched-requirements.sh keyed infra/* to NOTHING, so an infra/ edit attributed no
# requirement and this check was filtered out. The right fix for a path-scoped invariant is
# a path-map entry, not an unconditional run: `infra/*) echo "R-ENV-01"` was added there in
# the same change. Keep the two in sync — if this check ever grows an assertion outside
# infra/, either widen that map entry or give this test a whole-repo scope.
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Given the tracked tree, when the dev provisioning chain is parsed and driven
offline, then every dev node is created by local-KVM IaC, baselined, cluster-joined
and handed to Flux by one code path that covers all of them — with no node outside
it and no stage missing.
spec:
steps:
- name: then-every-dev-node-is-created-baselined-joined-and-managed-as-code
description: >-
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
conformance/run.sh --layer l0 execute).
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,442 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — dev-nodes-provisioned-as-code
# Serves: R-ENV-01 ("dev/test MUST be provisioned as local KVMs via IaC: created,
# baselined, cluster-joined, and managed as code." — spec
# docs/00-vision-and-scope.md; the four stages are enumerated in
# docs/02-environments-and-provisioning.md §"Node lifecycle stages
# (R-ENV-01 — 'created, baselined, cluster-joined, managed as code')":
# 1. Create — OpenTofu instantiates the VM
# 2. Baseline — OS baseline and prerequisites
# 3. Cluster-join — Ansible installs/joins k3s with Cilium as CNI; disables
# k3s bundled flannel/traefik/servicelb/kube-proxy
# 4. Manage — GitOps-managed by Flux; node config changes are made via IaC
# and re-applied, never hand-edited on the box.)
#
# WHY L0 IS THE RIGHT LAYER, AND WHAT IT MEANS TO TEST THIS BEHAVIOURALLY
# ----------------------------------------------------------------------
# R-ENV-01 is a requirement about HOW the dev nodes come to exist. The live cluster
# cannot answer it: a Kubernetes node object looks identical whether it was built by
# OpenTofu or by hand in a hypervisor UI (there is no API field for "was I created by
# IaC"). What CAN be proven, with no cluster, is that a COMPLETE, EXECUTABLE
# provisioning path exists for all four stages and covers every dev node — and, for the
# Ansible half, that is proven by RUNNING the real toolchain in a no-connection mode
# (`--syntax-check`, `--list-hosts`) rather than by reading the files, so a chain that
# does not actually resolve fails here. The IaC half is asserted by parsing the module:
# `tofu init/validate` needs the provider registry, and L0 is offline by contract.
#
# GIVEN the repository as git tracks it,
# WHEN the provisioning chain is parsed and (where offline execution is possible)
# actually driven,
# THEN CREATE: the IaC module targets a LOCAL KVM hypervisor (every declared provider
# is a KVM-family hypervisor provider — no cloud provider may appear),
# instantiates its nodes from a for_each map (never a count index, so one
# node can be replaced without disturbing its siblings), and gives each
# node CPU, RAM, an OS disk, a DEDICATED Longhorn data disk, a NIC and a
# cloud-init seed carrying the login user + SSH key;
# AND the node set it creates is EXACTLY the dev node set Ansible manages —
# names derived from the module's own prefix/count, so no node can be
# hand-added to the inventory or hand-created in the hypervisor;
# BASELINE: a baseline stage exists as declarative Ansible (package floor, kernel
# modules persisted AND loaded, sysctls, the Longhorn data disk formatted
# and mounted, swap off, iscsid enabled), driven by variables with
# defaults rather than per-host special cases;
# JOIN: one role installs k3s for both roles of every node — `--cluster-init`
# on the primary, `--server` join on the secondaries — with flannel,
# kube-proxy, traefik and servicelb disabled AT INSTALL, and Cilium
# installed as the CNI afterwards;
# MANAGE: site.yml chains baseline -> k3s -> Cilium -> verify -> Flux bootstrap;
# `ansible-playbook --syntax-check` on that chain SUCCEEDS; and
# `--list-hosts` shows every one of those stages targeting EVERY dev
# node, so no node is outside the code path.
#
# ENVIRONMENT-AGNOSTIC: no hostname, address, prefix or node count is written down here.
# The dev group names come from the playbooks' own default host patterns, the node names
# from the IaC module's prefix/count variables, and the two sets are compared to each
# other. This check therefore keeps working when the estate is renamed or resized.
#
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them):
# - That the RUNNING dev VMs were in fact created by this module. Nothing offline can:
# it would need the OpenTofu state (a remote R2 backend + credentials) or hypervisor
# access. `tofu plan` against the live estate is the operator's own check.
# - That the module APPLIES cleanly. `tofu validate` needs `tofu init`, which downloads
# the provider from the registry; L0 is offline by contract, so the module is parsed,
# not evaluated. ci/lint.sh does not cover .tf either — a gap worth closing in CI
# with a network-allowed stage, not here.
# - "Never hand-edited on the box" (stage 4's second half) — a drift check against the
# live nodes, which is an operator/Ansible-check-mode exercise, not a repo property.
# - REBUILD/re-converge ("destroy a dev node VM and rebuild it", docs/02's own proof
# line) — that is exercised at L3 by tests/l3-fault-injection/node-loss (R-STATE-02).
# - That the baseline actually ran on a node (cluster/host state), and that the
# hypervisor is on the operator's own LAN rather than remote — "local" is asserted as
# "a KVM hypervisor provider driving a self-hosted endpoint variable", not as a
# network topology fact.
# - DEVIATION FROM docs/02 PROSE, ASSERTED DELIBERATELY: docs/02 describes stage 2 as
# cloud-init applying the OS baseline. The implementation moved the rich baseline into
# the Ansible `node_baseline` role (cloud-init keeps hostname/user/SSH/network),
# because Proxmox provisioning is API-only and a custom cloud-init snippet needs
# SSH/SFTP to the hypervisor. R-ENV-01's own wording requires the node to be
# "baselined ... as code", not that cloud-init be the tool, so this check asserts the
# STAGE, wherever it lives, and records the divergence rather than hiding it.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
import re
import shutil
import subprocess
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
FAILURES: list[str] = []
NOTES: list[str] = []
CHECKS = 0
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
def note(msg):
NOTES.append(msg)
# A LOCAL DEPENDENCY the repo deliberately does not carry — the vault passphrase (kept
# outside the repo, R-SEC-01) or the Galaxy collections (gitignored; installed from
# collections/requirements.yml). Their absence is an ENVIRONMENT gap, not a repo defect,
# and is the ONLY tolerated reason the driven sub-assertions may not run. Deliberately
# narrow: every other non-zero exit is a real failure. The parse assertions always run.
ENV_ONLY_FAILURE = re.compile(
r"vault password|vault-id|Attempting to decrypt"
r"|No module named 'ansible_collections|couldn't resolve module/action", re.I)
TOFU = ROOT / "infra" / "tofu"
ANS = ROOT / "infra" / "ansible"
check(TOFU.is_dir(), "infra/tofu/ is missing — the CREATE stage has no IaC at all (R-ENV-01)")
check(ANS.is_dir(), "infra/ansible/ is missing — the BASELINE/JOIN/MANAGE stages have no code")
if not TOFU.is_dir() or not ANS.is_dir():
print(f"CHECK FAIL (dev-nodes-provisioned-as-code): {len(FAILURES)} of {CHECKS} failed",
file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
TF = "\n".join((p.read_text(errors="replace") for p in sorted(TOFU.glob("*.tf"))))
check(len(TF) > 500, "infra/tofu/*.tf is empty or a stub — nothing creates the dev nodes")
# ============================================================ STAGE 1 — CREATE (IaC)
# A LOCAL KVM hypervisor, by allowlist: any provider outside this set fails, so moving
# dev onto a cloud (or anything that is not a KVM hypervisor the operator runs) is caught
# here rather than being noticed later. Each entry is justified, not just listed:
# bpg/proxmox Proxmox VE — a KVM/QEMU hypervisor the operator hosts.
# dmacvicar/libvirt libvirt/KVM directly (the pre-Proxmox substrate).
KVM_PROVIDERS = {"bpg/proxmox", "dmacvicar/libvirt"}
providers = dict(re.findall(r'source\s*=\s*"([^"]+)"', TF) and
[(m.group(1), m.group(2)) for m in
re.finditer(r'(\w[\w-]*)\s*=\s*\{[^}]*?source\s*=\s*"([^"]+)"', TF, re.S)])
check(len(providers) > 0,
"infra/tofu declares no required_providers — the substrate is unpinned, so 'via IaC' "
"cannot be checked at all")
for alias, source in sorted(providers.items()):
check(source.lower() in KVM_PROVIDERS,
f"infra/tofu declares provider {alias}={source!r}, which is not a local KVM hypervisor "
f"provider (allowed: {sorted(KVM_PROVIDERS)}) — R-ENV-01 requires dev/test to be LOCAL KVMs")
# The VM resource: for_each-keyed, and carrying the hardware + seed docs/02 enumerates.
vm_blocks = re.findall(r'resource\s+"([\w]*(?:_vm|_domain))"\s+"(\w+)"\s*\{(.*?)\n\}',
TF, re.S)
check(len(vm_blocks) >= 1,
"no VM/domain resource found in infra/tofu — nothing in IaC creates a dev node")
for rtype, rname, body in vm_blocks:
check("for_each" in body,
f'resource "{rtype}" "{rname}" is not for_each-keyed. A count index makes single-node '
"replace unsafe (a -target'ed destroy cascaded to sibling nodes — DECISIONS-LOG F1), so "
"'created ... as code' would not survive a rebuild")
check(not re.search(r"^\s*count\s*=", body, re.M),
f'resource "{rtype}" "{rname}" uses a positional count index')
for want, why in (
(r"\bcpu\b", "CPU"),
(r"\bmemory\b", "RAM"),
(r"network_device|network_interface", "a NIC"),
(r"initialization|cloud_init|cloudinit", "a cloud-init seed"),
):
check(re.search(want, body) is not None,
f'resource "{rtype}" "{rname}" declares no {why} — docs/02 stage 1 requires CPU, RAM, '
"disks, NICs and a cloud-init seed to come from IaC")
disks = len(re.findall(r"^\s*disk\s*\{", body, re.M))
check(disks >= 2,
f'resource "{rtype}" "{rname}" declares {disks} disk block(s); docs/02 requires the OS '
"disk PLUS a dedicated data disk for Longhorn")
check(re.search(r"user_account|ssh_authorized_key|ssh_keys", body) is not None,
f'resource "{rtype}" "{rname}" seeds no login user / SSH key — the node could not be '
"reached by the BASELINE stage without a hand step")
# Node identity comes from variables, and those variables define the whole node set.
def tf_default(var: str):
m = re.search(r'variable\s+"' + re.escape(var) + r'"\s*\{(.*?)\n\}', TF, re.S)
if not m:
return None
d = re.search(r'default\s*=\s*(.+)', m.group(1))
return d.group(1).strip() if d else None
prefix_raw = tf_default("node_name_prefix")
count_raw = tf_default("node_count")
ips_raw = tf_default("node_ips")
check(prefix_raw is not None and count_raw is not None,
"infra/tofu has no node_name_prefix/node_count variables — the node set is not derived "
"from code, so the created nodes cannot be compared with the managed ones")
prefix = (prefix_raw or "").strip('"')
try:
node_count = int((count_raw or "0").strip())
except ValueError:
node_count = 0
check(node_count >= 1, f"node_count default is {count_raw!r} — no node would be created")
if ips_raw is not None:
n_ips = len(re.findall(r'"[^"]+"', ips_raw))
check(n_ips >= node_count,
f"node_ips default has {n_ips} address(es) for node_count={node_count} — the module "
"cannot address every node it creates")
check((TOFU / "outputs.tf").is_file() and
re.search(r'output\s+"(ansible_inventory|nodes)"', (TOFU / "outputs.tf").read_text()) is not None,
"infra/tofu/outputs.tf surfaces no node inventory output — the JOIN stage would have to be "
"told the node facts by hand instead of consuming them from IaC")
# ------------------------------------------------- the dev node set, derived twice over
# Group names are taken from the PLAYBOOKS' own default host patterns, so nothing here
# restates the estate's naming.
PLAYBOOKS = ANS / "playbooks"
def default_patterns(playbook: str) -> list[str]:
p = PLAYBOOKS / playbook
if not p.is_file():
return []
return re.findall(r"default\('([^']+)'\)", p.read_text())
dev_groups = set(default_patterns("node-baseline.yml") + default_patterns("k3s.yml"))
check(len(dev_groups) > 0,
"no playbook declares a default host pattern — the dev target is not expressed in code")
INV = ANS / "inventory" / "hosts.ini"
check(INV.is_file(), "infra/ansible/inventory/hosts.ini is missing — no node is managed as code")
ini_groups: dict[str, list[str]] = {}
children: dict[str, list[str]] = {}
cur = None
for line in (INV.read_text().splitlines() if INV.is_file() else []):
s = line.strip()
if not s or s.startswith("#"):
continue
m = re.match(r"^\[([^\]]+)\]$", s)
if m:
cur = m.group(1)
(children if cur.endswith(":children") else ini_groups).setdefault(
cur.split(":")[0] if cur.endswith(":children") else cur, [])
continue
if cur is None or cur.endswith(":vars"):
continue
host = s.split()[0]
if cur.endswith(":children"):
children.setdefault(cur.split(":")[0], []).append(host)
else:
ini_groups.setdefault(cur, []).append(host)
def hosts_in(group: str, seen=None) -> set[str]:
seen = seen or set()
if group in seen:
return set()
seen.add(group)
out = set(ini_groups.get(group, []))
for child in children.get(group, []):
out |= hosts_in(child, seen)
return out
dev_hosts: set[str] = set()
for g in sorted(dev_groups):
dev_hosts |= hosts_in(g)
check(len(dev_hosts) > 0,
f"the dev groups {sorted(dev_groups)} resolve to NO hosts in hosts.ini — nothing is managed")
expected = {f"{prefix}{i + 1}" for i in range(node_count)}
check(dev_hosts == expected,
f"the dev nodes Ansible manages ({sorted(dev_hosts)}) are not exactly the nodes the IaC "
f"module creates ({sorted(expected)}) — a node outside IaC is a hand-built node (R-ENV-01)")
# ============================================================ STAGE 2 — BASELINE (code)
BASE_ROLE = ANS / "roles" / "node_baseline"
check(BASE_ROLE.is_dir(),
"no node_baseline role — the BASELINE stage has no code (docs/02 stage 2)")
base_tasks = "\n".join(p.read_text(errors="replace")
for p in sorted((BASE_ROLE / "tasks").glob("*.yml"))) if BASE_ROLE.is_dir() else ""
base_defaults = "\n".join(p.read_text(errors="replace")
for p in sorted((BASE_ROLE / "defaults").glob("*.yml"))) if BASE_ROLE.is_dir() else ""
# Each item docs/02 names for stage 2, matched on the ANSIBLE MODULE that would do it —
# a declarative, idempotent task, not a shell one-liner.
for pattern, what in (
(r"ansible\.builtin\.apt:", "a package baseline (apt module)"),
(r"modprobe:", "kernel modules loaded (modprobe module)"),
(r"modules-load|/etc/modules", "kernel modules PERSISTED across reboot"),
(r"sysctl", "sysctl tuning"),
(r"filesystem:|mkfs", "a filesystem on the Longhorn data disk"),
(r"mount:|ansible\.posix\.mount", "the Longhorn data disk MOUNTED"),
(r"swap", "swap disabled"),
(r"iscsid", "iscsid enabled (Longhorn prerequisite)"),
):
check(re.search(pattern, base_tasks) is not None,
f"node_baseline does not declare {what} — docs/02 stage 2 requires the OS baseline and "
"k3s/Longhorn/KubeVirt prerequisites to be applied as code")
check(re.search(r"^node_baseline_packages:", base_defaults, re.M) is not None and
re.search(r"^node_baseline_kernel_modules:", base_defaults, re.M) is not None,
"node_baseline's package/module sets are not variables with defaults — a per-host special "
"case is not 'managed as code' (and could not be scoped per environment, R-ENV-02)")
# ========================================================== STAGE 3 — CLUSTER-JOIN (k3s)
K3S_ROLE = ANS / "roles" / "k3s_server"
k3s_text = "\n".join(p.read_text(errors="replace")
for p in sorted((K3S_ROLE).rglob("*.yml"))) if K3S_ROLE.is_dir() else ""
check(K3S_ROLE.is_dir() and len(k3s_text) > 200,
"no k3s_server role — nothing joins the nodes into a cluster (docs/02 stage 3)")
check("--cluster-init" in k3s_text,
"the k3s role never passes --cluster-init — the HA embedded-etcd cluster is never founded")
check(re.search(r"--server\s+https", k3s_text) is not None,
"the k3s role never joins secondaries with --server — nodes would not form one cluster")
check(re.search(r"K3S_TOKEN|--token", k3s_text) is not None,
"the k3s role never supplies a join token — the join could only be completed by hand")
# The bundled components docs/02 requires to be disabled AT INSTALL, each checked by the
# flag that actually disables it.
for pattern, what in (
(r"--flannel-backend=none", "flannel (Cilium is the CNI)"),
(r"--disable-kube-proxy", "kube-proxy (Cilium kube-proxy replacement)"),
(r"traefik", "traefik (Gateway API via Cilium)"),
(r"servicelb", "servicelb"),
):
check(re.search(pattern, k3s_text) is not None,
f"the k3s install does not disable {what} — docs/02 stage 3 requires it disabled at "
"install, or the bundled component fights the platform's own")
check((ANS / "roles" / "cilium").is_dir(),
"no cilium role — stage 3 requires Cilium to be installed as the CNI at bootstrap")
check((ANS / "roles" / "flux_bootstrap").is_dir(),
"no flux_bootstrap role — stage 4's hand-off to GitOps has no code")
# ============================================== STAGE 4 — MANAGE + the chain is REAL
SITE = PLAYBOOKS / "site.yml"
check(SITE.is_file(), "no playbooks/site.yml — the provisioning chain is not expressed as one path")
site_txt = SITE.read_text() if SITE.is_file() else ""
order = [m for m in re.findall(r"import_playbook:\s*(\S+)", site_txt)]
check(order[:1] == ["node-baseline.yml"],
f"site.yml imports {order} — the BASELINE stage must come first (docs/02 sequencing)")
for want in ("k3s.yml", "cilium.yml", "flux-bootstrap.yml"):
check(want in order, f"site.yml does not import {want} — a lifecycle stage is missing from the chain")
if "k3s.yml" in order and "cilium.yml" in order:
check(order.index("k3s.yml") < order.index("cilium.yml"),
"site.yml installs Cilium before k3s — the CNI cannot precede the cluster")
# --- DRIVE THE REAL TOOLCHAIN, OFFLINE ------------------------------------------------
# --syntax-check and --list-hosts open NO connection to any node: they resolve the
# playbook, roles, collections and inventory only. Running them is the difference between
# "the chain is written down" and "the chain resolves".
if shutil.which("ansible-playbook") is None:
note("ansible-playbook is not installed here (e.g. the in-cluster chainsaw image), so the "
"chain was PARSED but not DRIVEN; the merge gate's container has it and does drive it")
else:
# cwd=infra/ansible so the tracked ansible.cfg (roles_path, collections_path, inventory)
# is the one in force — the same resolution a real run gets.
def ansible(*args) -> subprocess.CompletedProcess:
return subprocess.run(["ansible-playbook", *args], cwd=ANS,
capture_output=True, text=True)
syn = ansible("--syntax-check", "playbooks/site.yml")
syn_out = (syn.stderr or "") + (syn.stdout or "")
lh = None
if syn.returncode != 0 and ENV_ONLY_FAILURE.search(syn_out):
# NARROW, SELF-POLICING SKIP — see ENV_ONLY_FAILURE above.
note("ansible-playbook lacks a LOCAL dependency kept outside the repo by design (the vault "
"passphrase and/or the gitignored Galaxy collections), so --syntax-check/--list-hosts "
"were not driven; every parse assertion above still ran")
else:
check(syn.returncode == 0,
"`ansible-playbook --syntax-check playbooks/site.yml` FAILED — the provisioning chain "
f"does not even load: {syn_out.strip()[:400]}")
lh = ansible("--list-hosts", "playbooks/site.yml")
check(lh.returncode == 0,
"`ansible-playbook --list-hosts playbooks/site.yml` FAILED — the chain does not "
f"resolve against the tracked inventory: "
f"{((lh.stderr or '') + (lh.stdout or '')).strip()[:400]}")
if lh is not None and lh.returncode == 0:
# Parse plays -> hosts. NEVER `ansible-inventory --list`: it prints DECRYPTED vault values.
plays: list[tuple[str, set[str]]] = []
name, hosts, in_hosts = None, set(), False
for line in lh.stdout.splitlines():
m = re.match(r"\s*play #\d+ \([^)]*\):\s*(.*?)\s*TAGS:", line)
if m:
if name is not None:
plays.append((name, hosts))
name, hosts, in_hosts = m.group(1), set(), False
continue
if re.match(r"\s*hosts \(\d+\):", line):
in_hosts = True
continue
if in_hosts and line.strip() and not line.strip().startswith("pattern:"):
hosts.add(line.strip())
if name is not None:
plays.append((name, hosts))
check(len(plays) >= 4,
f"--list-hosts resolved {len(plays)} play(s) — the four-stage chain did not expand")
# Every lifecycle stage must cover EVERY dev node. A stage that reaches two of three
# nodes means the third was finished by hand.
STAGES = {
"baseline": r"baseline",
"cluster-join": r"^k3s (primary|secondar)",
"cni": r"cilium",
"manage (flux hand-off)": r"flux",
}
for stage, rx in STAGES.items():
covered: set[str] = set()
matched = 0
for pname, phosts in plays:
if re.search(rx, pname, re.I):
matched += 1
covered |= phosts
check(matched >= 1, f"no play in site.yml implements the {stage} stage")
check(covered == dev_hosts,
f"the {stage} stage targets {sorted(covered)}, not every dev node "
f"{sorted(dev_hosts)} — a node outside the code path is a hand-provisioned "
"node (R-ENV-01)")
# --------------------------------------------------------------------------- verdict
# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
check(CHECKS >= 30,
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
"would be meaningless")
if FAILURES:
print(f"CHECK FAIL (dev-nodes-provisioned-as-code): {len(FAILURES)} of {CHECKS} assertions failed",
file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
for n in NOTES:
print(f" note: {n}")
print(f"check OK: dev-nodes-provisioned-as-code — {CHECKS} assertions; "
f"{len(dev_hosts)} dev node(s) {sorted(dev_hosts)} created by IaC ({sorted(providers.values())}), "
f"baselined, cluster-joined and Flux-managed by the same chain. R-ENV-01's authored-path "
"clause only — the live VMs' provenance, `tofu validate`, and rebuild/re-converge are NOT "
"proven here (see the header's gap list; rebuild is L3 node-loss).")
PY
@@ -0,0 +1,99 @@
# =============================================================================
# L0 STATIC — two egress paths, and the rule set that selects between them (R-NET-04) [EXECUTABLE]
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
# Spec: docs/00 R-NET-04 ("The platform MUST support **multiple egress paths** selected by
# routing rules ... realized as two paths — the local `internet` path and the
# in-cluster `vpn` path — not multi-ISP routing"), elaborated by docs/03 §"Egress
# routing: two paths, one physical WAN": "the `internet` path is simply the default
# route out"; "**Default is the WAN; the VPN is strictly opt-in.** Every pod — and
# the nodes themselves — egress the local/private WAN by default and **never** the
# VPN. Only services labeled `egress-class=vpn` are steered to the tunnel, and the
# kill-switch blackholes **only** that class."
#
# WHAT THIS ADDS OVER R-NET-05/07 (already covered, and NOT re-tagged here)
# ------------------------------------------------------------------------
# R-NET-05/07 are about a SERVICE — it declares a class, enforcement is outside it, the class
# flips via GitOps — and are proven by tests/l1-rendering/egress-class-render and
# tests/l2-behavioral/egress-class. R-NET-04 is about the PLATFORM's PATH SET and the RULES
# that select among them, which nothing else asserts:
# * exactly TWO paths, the same closed set in the schema, the admission policy and the
# renderer (a third value would be a path nothing routes; "multi-ISP" is out of scope);
# * selection is by a ROUTING RULE and the rule set is TOTAL AND DISJOINT — exactly one
# Cilium steering rule, selecting only `egress-class=vpn`, and on the gateway node one
# `ip rule` scoped to the tunnel address. NOTHING selects the `internet` class, because
# absence of a matching rule IS the default path;
# * the two paths COEXIST per DESTINATION: private space is excluded from the tunnel, so a
# vpn-class pod's cluster/LAN traffic still takes the local path;
# * the vpn path never becomes the DEFAULT — its default route lives in a dedicated routing
# table, `route-noexec` is set and a pushed `redirect-gateway` is ignored;
# * the kill-switch blackholes ONLY that class — the nft chain's policy is accept and every
# DROP is scoped to the vpn class's own source set/fwmark, so `internet` fails OPEN.
# (R-NET-06's opposite claim, that `vpn` fails CLOSED, is L3's vpn-killswitch.)
#
# WHY L0: every clause above is a statement about ALL rules — including rules that must be
# ABSENT (nothing selects the default path; no unscoped rule; no unqualified drop; no default
# route in the main table). A cluster can show which path two pods took; it cannot show that
# no third rule exists or that the vocabulary is closed.
#
# ENVIRONMENT-AGNOSTIC: no address appears. The private ranges the tunnel must exclude are
# checked as CONTAINMENT of the addresses each environment's own Flux overlay declares (pod
# CIDR, Gateway VIP, LB pool, API host), so a new environment on a different LAN is checked
# against its own values.
#
# GIVEN the repository as git tracks it,
# WHEN the egress vocabulary, every CiliumEgressGatewayPolicy, the gateway node's policy
# routing / OpenVPN / nftables templates, and every environment's overlay addresses are
# parsed and cross-checked,
# THEN there are exactly two paths with `internet` as the default; exactly one steering rule
# selects exactly the vpn class by label; the gateway is selected by node label with a
# named egress interface; all RFC1918 space and every address any environment assigns
# itself is excluded from the tunnel; the tunnel's default route and its fail-closed
# blackhole live only in a dedicated routing table behind a scoped `ip rule`; OpenVPN
# cannot take the main table; and no nftables DROP is unscoped.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + PyYAML; no
# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
# Chainsaw Test wraps the SAME script.
#
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
# * BEHAVIOUR. That an unlabelled pod really takes the WAN path is asserted at L2 by
# tests/l2-behavioral/egress-default-path (same requirement label); the two labelled
# classes are R-NET-05's L2 test; the kill-switch firing is R-NET-06's L3 test.
# * The RUNNING gateway node's routing tables / nftables ruleset. This reads the templates
# that generate them; a hand-added rule on the box is invisible here.
# * PHYSICAL MULTI-WAN's absence is NOT asserted — it is out of scope per OQ-1 and no
# honest repository signal distinguishes "one WAN, correctly" from "one WAN, by default".
# * IPv6 egress.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: egress-paths-selected-by-rules
labels:
conformance.home.k3s/R-NET-04: "true"
annotations:
conformance.home.k3s/requirements: "R-NET-04"
conformance.home.k3s/layer: "L0"
# WHOLE-REPO SCOPE. Universally quantified over tracked YAML: every manifest that carries
# an egress-class label/annotation must use a value the policy set actually steers. The
# file most likely to break it is a NEW workload under apps/ — which attributes no
# requirement — not platform/vpn-egress/ (the only path the map keys to R-NET-04).
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Given the tracked tree, when the egress vocabulary, every steering rule and the gateway
node's policy-routing/OpenVPN/nftables templates are parsed, then the platform offers
exactly two egress paths with the WAN as the default, exactly one rule selects exactly
the vpn class, local destinations stay on the local path in every environment, the
tunnel never becomes the default route, and nothing unscoped can blackhole the default
path.
spec:
steps:
- name: then-two-paths-exist-and-the-rules-select-between-them-totally-and-disjointly
description: >-
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
conformance/run.sh --layer l0 execute).
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,521 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — egress-paths-selected-by-rules
# Serves: R-NET-04 ("The platform MUST support **multiple egress paths** selected by routing
# rules. Per OQ-1 the environment has **one physical commercial WAN and no
# multi-ISP plans**, so this is realized as two paths — the local `internet` path and
# the in-cluster `vpn` path — not multi-ISP routing. Physical multi-WAN (multiple ISP
# uplinks) is **out of scope** unless a future ADR reopens it."
# — spec docs/00-vision-and-scope.md).
#
# The elaborating spec text this check quotes and asserts against — docs/03
# §"Egress routing: two paths, one physical WAN (R-NET-04/05 — resolved by OQ-1)":
# | `internet` | Out the single commercial WAN | Normal pod egress via node -> SOHO edge
# router -> internet |
# | `vpn` | Routed through the in-cluster VPN tunnel | Steered to the shared VPN
# egress gateway (ADR-0013) |
# "**Physical multi-WAN (multi-ISP) is out of scope.** With a single WAN there is nothing
# to policy-route across; the `internet` path is simply the default route out."
# "**Default is the WAN; the VPN is strictly opt-in.** Every pod — and the nodes
# themselves — egress the local/private WAN by default and **never** the VPN. Only
# services labeled `egress-class=vpn` are steered to the tunnel, and the kill-switch
# blackholes **only** that class."
# "It is **split-tunnel**: it creates `tun0` but does **not** set the node's default
# route (`route-noexec` + policy routing), so the node's own traffic stays on the WAN —
# only what Cilium points at `tun0` uses the VPN."
#
# WHAT R-NET-04 ADDS OVER R-NET-05/07 (which are ALREADY covered) — READ THIS FIRST
# ---------------------------------------------------------------------------------
# R-NET-05/07 are about a SERVICE: it can declare its class, enforcement happens outside it,
# and the class can be flipped via GitOps. Those are proven by
# tests/l1-rendering/egress-class-render (the render) and tests/l2-behavioral/egress-class
# (two labelled pods observe two different public IPs, and a relabel re-routes the same pod).
# Re-tagging either of those with R-NET-04 would be a duplicate wearing a new label.
#
# R-NET-04 is about the PLATFORM's PATH SET and the RULES THAT SELECT AMONG THEM. Its
# distinctive content — none of which any existing test asserts — is:
#
# (1) THERE ARE EXACTLY TWO PATHS, and the same two everywhere. The class vocabulary is
# one closed set of two values agreed by the schema, the admission policy and the
# renderer. A third value appearing in one of them but not the others would be a path
# nothing routes (or a path nothing validates); "multi-ISP" reappearing as a third
# class would contradict OQ-1's resolution.
# (2) SELECTION IS BY A ROUTING RULE, and the rule set is TOTAL AND DISJOINT. The `vpn`
# path is entered only via rules that select the vpn class — a Cilium
# CiliumEgressGatewayPolicy pod-selector, and on the gateway node an `ip rule` scoped
# to the tunnel source address. Nothing selects the `internet` class, because
# "the `internet` path is simply the default route out": ABSENCE of a matching rule IS
# the default path. So an unlabelled pod, and an `internet`-class pod, take the WAN.
# (3) THE PATHS COEXIST — the choice is per-destination, not per-pod-lifetime. The vpn
# rule tunnels public destinations while EXCLUDING all private space, so a vpn-class
# pod's cluster/LAN traffic still takes the local path at the same time as its
# internet traffic takes the tunnel.
# (4) THE VPN PATH NEVER BECOMES THE DEFAULT. Split tunnel: the tunnel's default route
# lives in a DEDICATED routing table reached only through the scoped rule, the
# provider's pushed `redirect-gateway` is ignored, and `route-noexec` keeps OpenVPN
# from touching the main table. Otherwise there would be one path (the VPN), not two.
# (5) THE KILL-SWITCH BLACKHOLES ONLY THAT CLASS. The blackhole default is inside the
# dedicated table; the nftables chain's policy is ACCEPT and every DROP is qualified
# by the vpn class's own source set / fwmark. The `internet` path therefore fails
# OPEN and can never be blackholed. (R-NET-06's opposite claim — that `vpn` fails
# CLOSED — is proven at L3 by tests/l3-fault-injection/vpn-killswitch. This check
# asserts the OTHER direction, which that test does not.)
#
# WHY L0. Every clause above is a property of the RULE SET, i.e. of authored configuration:
# totality ("nothing selects the default path"), disjointness ("exactly one rule, selecting
# exactly one class"), and scoping ("no unqualified drop, no default route in the main
# table") are statements about ALL rules, including the ones that would have to be absent.
# A running cluster can show you which path two pods took; it cannot show you that no third
# rule exists, that no rule is unscoped, or that the vocabulary is closed. The behavioural
# complement — that an UNLABELLED pod really does take the WAN path — is at L2 in
# tests/l2-behavioral/egress-default-path (same requirement label).
#
# ENVIRONMENT-AGNOSTIC: no address is written down. The private ranges the vpn rule must
# exclude are checked as CONTAINMENT of the addresses each environment's own Flux overlay
# declares (pod CIDR, Gateway VIP, LB pool, API host) — so a new environment on a different
# LAN is checked against its own addresses, and the check keeps working when they change.
#
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them):
# - Behaviour. That a vpn-class pod's packets really leave via tun0, that an unlabelled
# pod's really do not, and that the tunnel is up at all. L2 covers the two labelled
# classes (tests/l2-behavioral/egress-class, R-NET-05) and the unlabelled default
# (tests/l2-behavioral/egress-default-path, R-NET-04); the kill-switch firing is L3.
# - THE RUNNING NODE's routing tables and nftables ruleset. This reads the templates that
# generate them, driven by no node. A hand-added `ip rule` on the gateway node is
# invisible here (and to every other test); drift on the box is an Ansible check-mode
# exercise for the operator.
# - PHYSICAL MULTI-WAN. Out of scope per OQ-1, so its absence is not asserted: there is no
# honest repository signal that would distinguish "one WAN, correctly" from "one WAN,
# because nobody tried" — and inventing one would be a hollow assertion.
# - IPv6. The policy's CIDRs and the routing rules are IPv4-only, matching the platform.
# If IPv6 egress is ever enabled, the `internet` path would need the same analysis and
# this check would NOT catch its absence.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
import ipaddress
import re
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
FAILURES: list[str] = []
CHECKS = 0
LBL = "homelab.keithjackson.dev"
CLASS_LABEL = f"{LBL}/egress-class"
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
def die():
print(f"CHECK FAIL (egress-paths-selected-by-rules): {len(FAILURES)} of {CHECKS} "
"assertions failed", file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
def docs(path: Path):
"""Every YAML document in a file, skipping empties."""
try:
return [d for d in yaml.safe_load_all(path.read_text()) if isinstance(d, dict)]
except yaml.YAMLError:
return []
# Skip dot-DIRECTORIES (.git, .claude, .cache) while still matching dot-FILES such as
# .sops.yaml. A bare rglob from ROOT also descends into NESTED WORKTREES —
# .claude/worktrees/agent-*/ created for isolated agent runs — which made this check see
# 5 CiliumEgressGatewayPolicy objects instead of the 1 authored, failing R-NET-04 for a
# reason that has nothing to do with egress. .claude/ is gitignored (.gitignore:43), so
# nothing under it is part of the authored surface by definition.
TRACKED_YAML = sorted(p for p in ROOT.rglob("*.yaml")
if "_artifacts" not in p.parts
and not any(part.startswith(".")
for part in p.relative_to(ROOT).parent.parts))
SCHEMA = ROOT / "schemas" / "servicedefinition.schema.json"
RENDER = ROOT / "catalog" / "render.py"
VPN_ROLE = ROOT / "infra" / "ansible" / "roles" / "vpn_egress_gateway"
CLUSTERS = ROOT / "clusters"
for p in (SCHEMA, RENDER, VPN_ROLE, CLUSTERS):
check(p.exists(), f"{p.relative_to(ROOT)} is missing — the egress path model cannot be "
"read at all (R-NET-04)")
if FAILURES:
die()
# ======================================================= (1) EXACTLY TWO PATHS, AGREED
import json # noqa: E402 — only needed for the schema
schema = json.loads(SCHEMA.read_text())
egress_prop = ((schema.get("properties") or {}).get("egress") or {})
enum = egress_prop.get("enum")
check(isinstance(enum, list) and len(enum) == 2,
f"schemas/servicedefinition.schema.json declares egress enum {enum!r} — R-NET-04 is "
"realized as EXACTLY TWO paths (OQ-1: one WAN, no multi-ISP). A third value is a path "
"nothing routes; one value is not 'multiple egress paths'")
PATHS = set(enum or [])
check(PATHS == {"internet", "vpn"},
f"the declared egress paths are {sorted(PATHS)} — docs/03's table fixes them as "
"`internet` (out the single commercial WAN) and `vpn` (the in-cluster tunnel)")
check(egress_prop.get("default") == "internet",
f"the schema's default egress class is {egress_prop.get('default')!r} — docs/03: "
"'Default is the WAN; the VPN is strictly opt-in', and 'the `internet` path is simply "
"the default route out'. A service that says nothing must get the WAN")
# The admission policy must constrain the pod label to the SAME closed set, or a workload
# could carry a class the schema never allowed (and that no rule routes).
pol_files = sorted((ROOT / "platform" / "kyverno").rglob("*egress*.yaml"))
check(len(pol_files) >= 1,
"no Kyverno policy file mentions egress under platform/kyverno — nothing stops a pod "
"carrying an egress class outside the declared path set")
pol_text = "\n".join(p.read_text(errors="replace") for p in pol_files)
for cls in sorted(PATHS):
check(re.search(rf"\b{re.escape(cls)}\b", pol_text) is not None,
f"the egress admission policy never names the {cls!r} path — a class the policy "
"does not know cannot be validated against the path set")
# No tracked object may carry a class outside the closed set — with ONE self-policing
# exemption. A Kyverno policy-test fixture deliberately carries an INVALID class so the
# admission test can assert the policy REJECTS it; that pod is exempt only while a
# kyverno-test.yaml in the same tests tree still expects `result: fail` for it. Delete the
# expectation (or flip it to pass) and the exemption evaporates — never a bare path allowlist.
def rejected_by_a_policy_test(resource_name: str, fixture: Path) -> bool:
tests_root = fixture.parent
while tests_root != ROOT and tests_root.name != "tests":
tests_root = tests_root.parent
if tests_root == ROOT:
return False
for tf in sorted(tests_root.rglob("kyverno-test.yaml")):
for d in docs(tf):
for case in (d.get("results") or []):
if case.get("result") == "fail" and resource_name in (case.get("resources") or []):
return True
return False
stray: list[str] = []
_STRAY_RE = re.compile(rf"{re.escape(CLASS_LABEL)}:\s*[\"']?([A-Za-z][A-Za-z0-9-]*)")
for f in TRACKED_YAML:
for d in docs(f):
labels = ((d.get("metadata") or {}).get("labels") or {})
cls = labels.get(CLASS_LABEL)
# Also catch pod-template labels (Deployments/DaemonSets/Jobs/VMs).
tmpl = (((d.get("spec") or {}).get("template") or {}).get("metadata") or {}).get("labels") or {}
for value, owner in ((cls, d), (tmpl.get(CLASS_LABEL), d)):
if value is None or value in PATHS or value == "CLASS":
continue
name = ((owner.get("metadata") or {}).get("name") or "?")
if rejected_by_a_policy_test(name, f):
continue
stray.append(f"{f.relative_to(ROOT)}:{name}={value}")
# Values written as raw text (templates, docs-as-code) are caught too.
for value in set(_STRAY_RE.findall(f.read_text(errors="replace"))):
if value in PATHS or value == "CLASS":
continue
if any(value in s for s in stray):
continue
# Raw-text hit with no parsed owner: exempt only if EVERY object in the file that
# carries this value is a policy-test fixture proven to be rejected.
owners = [((d.get("metadata") or {}).get("name") or "?") for d in docs(f)
if ((d.get("metadata") or {}).get("labels") or {}).get(CLASS_LABEL) == value]
if owners and all(rejected_by_a_policy_test(o, f) for o in owners):
continue
stray.append(f"{f.relative_to(ROOT)}:<text>={value}")
check(not stray,
f"tracked YAML assigns egress-class value(s) outside the declared path set "
f"{sorted(PATHS)}: {sorted(stray)} — a value no routing rule selects and no admission "
"policy allows is a workload on no path at all (the only exemption is a Kyverno "
"policy-test fixture that a kyverno-test.yaml still expects to FAIL)")
# The renderer must project the DECLARED class onto the pod label, unmodified: that label is
# what the routing rule selects on, so a renderer that rewrote it would break the coupling
# between the declaration and the path.
rtext = RENDER.read_text()
check(re.search(rf'f?"\{{LBL\}}/egress-class":\s*s\["egress"\]', rtext) is not None
or re.search(r'egress-class\"\]?\s*:\s*s\[\"egress\"\]', rtext) is not None,
"catalog/render.py does not project the ServiceDefinition's own `egress` value onto the "
f"{CLASS_LABEL} pod label — the declaration and the routing rule's selector would be "
"two independent facts that could disagree")
# ============================== (2) SELECTION IS BY RULE; THE RULE SET IS TOTAL & DISJOINT
policies: list[tuple[Path, dict]] = []
for p in TRACKED_YAML:
for d in docs(p):
if d.get("kind") == "CiliumEgressGatewayPolicy":
policies.append((p, d))
check(len(policies) == 1,
f"{len(policies)} CiliumEgressGatewayPolicy object(s) are authored "
f"({[str(p.relative_to(ROOT)) for p, _ in policies]}) — R-NET-04's steering must be ONE "
"rule. Two rules can overlap, and which one wins for an overlapping pod is not "
"something the repository states")
for path, pol in policies:
rel = str(path.relative_to(ROOT))
spec = pol.get("spec") or {}
selectors = spec.get("selectors") or []
check(len(selectors) == 1,
f"{rel}: the steering rule has {len(selectors)} selector(s) — every additional "
"selector widens the set of pods steered off the default path, and R-NET-04's "
"selection must be exactly 'the vpn class'")
for sel in selectors:
pod_sel = sel.get("podSelector")
check(isinstance(pod_sel, dict) and pod_sel,
f"{rel}: a selector has no podSelector — an empty selector in Cilium matches "
"EVERY pod, which would put the whole cluster on the VPN path and leave no "
"default path at all")
ml = (pod_sel or {}).get("matchLabels") or {}
me = (pod_sel or {}).get("matchExpressions") or []
check(not me,
f"{rel}: the podSelector uses matchExpressions {me!r}. The class contract is a "
"single label equality; expressions (In/NotIn/Exists) can select pods that "
"never declared a class, so the rule set would no longer be disjoint from the "
"default path")
check(set(ml) == {CLASS_LABEL},
f"{rel}: the podSelector matches on {sorted(ml)} — it must select on exactly "
f"{CLASS_LABEL}. Any other key makes path selection depend on something other "
"than the declared class (and a namespace/app key would silently exempt pods)")
check(ml.get(CLASS_LABEL) == "vpn",
f"{rel}: the steering rule selects {CLASS_LABEL}={ml.get(CLASS_LABEL)!r}. Only "
"the `vpn` class may be steered: docs/03 — 'Only services labeled "
"egress-class=vpn are steered to the tunnel'. A rule selecting `internet` would "
"route the default path through the tunnel, i.e. leave one path, not two")
# The gateway is chosen by a NODE LABEL, not a hostname: the path's endpoint is itself a
# rule, so the gateway can move (or gain a standby) with no rule rewrite.
gw = spec.get("egressGateway") or {}
node_sel = (gw.get("nodeSelector") or {}).get("matchLabels") or {}
check(bool(node_sel),
f"{rel}: the egressGateway selects no node by label — pinning the path's exit to a "
"named host means the vpn path cannot fail over (docs/03 requires two gateway "
"nodes + Egress Gateway HA)")
check("nodeName" not in gw,
f"{rel}: the egressGateway pins a nodeName — the exit node must be selected by "
"label so the path survives losing that node")
check(isinstance(gw.get("interface"), str) and gw.get("interface"),
f"{rel}: the egressGateway names no egress interface — without it the steered class "
"is not actually sent down a different path from the default")
# ------------------- (3) THE PATHS COEXIST: the choice is per DESTINATION -------------
dests = spec.get("destinationCIDRs") or []
excl = spec.get("excludedCIDRs") or []
check([str(c) for c in dests] == ["0.0.0.0/0"],
f"{rel}: destinationCIDRs is {dests!r}. The vpn path is for PUBLIC destinations "
"('Routed through the in-cluster VPN tunnel'), expressed as the whole internet "
"minus the exclusions below — narrowing it here would silently leave public "
"destinations on the WAN for a vpn-class service")
try:
excluded = [ipaddress.ip_network(str(c)) for c in excl]
except ValueError as e:
excluded = []
check(False, f"{rel}: excludedCIDRs contains an unparsable network ({e})")
check(len(excluded) >= 3,
f"{rel}: only {len(excluded)} excluded CIDR(s) — docs/03 requires intra-cluster, "
"pod->service and LAN traffic to stay on the LOCAL path, which is all of RFC1918")
def covered(net: ipaddress.IPv4Network | ipaddress.IPv6Network) -> bool:
return any(net.subnet_of(e) for e in excluded if e.version == net.version)
# RFC1918 in full — computed, not string-matched, so a typo'd mask is caught.
for rfc in ("10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"):
net = ipaddress.ip_network(rfc)
check(covered(net),
f"{rel}: the private range {rfc} is NOT excluded from the tunnel — a vpn-class "
"pod's cluster/LAN traffic would be SNAT'd out tun0, so the two paths would not "
"coexist for the same pod and east-west traffic would break")
# Every address each ENVIRONMENT declares for itself must fall in the excluded set: that
# is what makes 'local stays local' true per environment rather than by coincidence.
def overlay_addresses() -> list[tuple[str, str]]:
found: list[tuple[str, str]] = []
base = CLUSTERS / "base" / "cluster-vars.yaml"
data = {}
if base.is_file():
for d in docs(base):
if d.get("kind") == "ConfigMap":
data = {k: str(v) for k, v in (d.get("data") or {}).items()}
for f in sorted(CLUSTERS.glob("*/kustomization.yaml")):
env = {**data}
for m in re.finditer(
r"-\s+op:\s*replace\s*\n\s*path:\s*/data/(?P<k>[A-Z0-9_]+)\s*\n"
r"\s*value:\s*(?P<v>\S.*?)\s*$", f.read_text(), re.M):
env[m.group("k")] = m.group("v").strip().strip('"').strip("'")
label = f.parent.name
for key in ("CLUSTER_POD_CIDR", "GATEWAY_VIP", "CILIUM_LB_POOL_START",
"CILIUM_LB_POOL_STOP", "K3S_API_HOST"):
v = env.get(key, "")
if v and not v.startswith("TBD"):
found.append((f"{label}/{key}", v))
return found
addrs = overlay_addresses()
check(len(addrs) >= 5,
f"only {len(addrs)} environment address(es) could be read from the Flux overlays — "
"the 'local traffic stays local' assertion would be checking almost nothing")
for label, value in addrs:
try:
net = ipaddress.ip_network(value, strict=False)
except ValueError:
continue
check(covered(net),
f"{rel}: {label}={value}, an address THIS repository assigns to a cluster, is "
"NOT inside the tunnel's excludedCIDRs — a vpn-class pod in that environment "
"would try to reach its own cluster/LAN through the VPN")
# ============ (4) THE VPN PATH IS A SECOND TABLE, NEVER THE DEFAULT (split tunnel) ========
TPL = VPN_ROLE / "templates"
up = (TPL / "vpn-egress-up.sh.j2")
down = (TPL / "vpn-egress-down.sh.j2")
conf = (TPL / "vpn-egress.conf.j2")
nft = (TPL / "vpn-egress-killswitch.nft.j2")
for f in (up, down, conf, nft):
check(f.is_file(), f"{f.relative_to(ROOT)} is missing — the second egress path's routing "
"rules are not authored")
if FAILURES:
die()
defaults = yaml.safe_load((VPN_ROLE / "defaults" / "main.yml").read_text()) or {}
def render(text: str) -> str:
"""Substitute `{{ var }}` with the role's own default value, for every default.
The templates reference their knobs by NAME, so an analysis that looked for the VALUE
(e.g. the nft set name) would silently match nothing and pass vacuously — which is
exactly what an earlier draft of this check did. Rendering first means the assertions
below examine the text the node actually gets. Only simple `{{ name }}` expressions are
substituted; anything else is left as-is (and the assertions are written so an
unsubstituted expression fails rather than passes).
"""
out = text
for k, v in defaults.items():
if isinstance(v, (str, int)):
out = re.sub(r"\{\{\s*" + re.escape(str(k)) + r"\s*\}\}", str(v), out)
return out
up_t, conf_t, nft_t = render(up.read_text()), render(conf.read_text()), render(nft.read_text())
table_id = str(defaults.get("vpn_egress_gateway_route_table_id", ""))
check(table_id.isdigit() and table_id not in ("0", "253", "254", "255"),
f"the tunnel's routing table id is {table_id!r} — the second path must live in a "
"DEDICATED table. 254 (main), 255 (local) and 253 (default) are the kernel's own, and "
"putting the tunnel's default route in any of them makes the VPN the node's default "
"route instead of a selectable second path")
check(str(defaults.get("vpn_egress_gateway_rule_priority", "")).isdigit(),
"no ip-rule priority is declared for the tunnel table — 'selected by routing rules' "
"needs a rule, at a defined precedence")
# Every `ip route ... default` the up-script installs must name the dedicated table.
route_lines = [l.strip() for l in up_t.splitlines()
if re.search(r"^\s*ip\s+route\s+(replace|add)\b", l)]
check(len(route_lines) >= 2,
f"the up-script installs {len(route_lines)} route(s) — the vpn path needs its own "
"default route via the tunnel AND the fail-closed blackhole behind it")
for l in route_lines:
check(re.search(r'table\s+"?\$\{?TABLE_ID', l) is not None,
f"the up-script installs a route WITHOUT naming the dedicated table: {l!r}. A route "
"landing in the main table changes the path every other pod and the node itself "
"uses — the VPN would stop being opt-in")
check("blackhole" not in l or re.search(r'table\s+"?\$\{?TABLE_ID', l) is not None,
f"a blackhole route is installed outside the tunnel table: {l!r}. docs/03: the "
"kill-switch 'blackholes **only** that class'; `internet` fails OPEN and must never "
"be blackholed")
# Every `ip rule` must be SCOPED. An unscoped rule (`ip rule add lookup <table>`) sends
# everything into the tunnel table, which would leave one path instead of two.
rule_lines = [l.strip() for l in up_t.splitlines() if re.search(r"^\s*ip\s+rule\s+add\b", l)]
check(len(rule_lines) >= 1,
"the up-script adds no `ip rule` — nothing SELECTS the tunnel path, so the second "
"routing table would never be consulted (R-NET-04's 'selected by routing rules')")
for l in rule_lines:
check(re.search(r"\bfrom\s+\S|\bfwmark\s+\S|\biif\s+\S", l) is not None,
f"an unscoped ip rule is installed: {l!r}. Without a `from`/`fwmark`/`iif` selector "
"EVERY packet is looked up in the tunnel table, so the WAN path disappears and the "
"VPN is no longer opt-in")
check(re.search(r'lookup\s+"?\$\{?TABLE_ID', l) is not None,
f"an ip rule looks up a table other than the tunnel's: {l!r}")
check(re.search(r'ip rule add from\s+"\$\{SRC\}"', up_t) is not None,
"the steering rule is not scoped to ${SRC}, the tunnel's own local address — that "
"address is what Cilium SNATs vpn-class pods to, and it is the ONLY thing that "
"distinguishes the steered class from every other flow on the gateway node")
check(re.search(r"^\s*if \[ -z \"\$\{SRC\}\" \]", up_t, re.M) is not None,
"the up-script does not refuse to run when the tunnel address is unknown — installing "
"a partial rule set is how a path ends up half-selected")
# The provider must not be able to make the tunnel the default route either.
check(re.search(r"^\s*route-noexec\s*$", conf_t, re.M) is not None,
"the OpenVPN wrapper config lacks `route-noexec` — OpenVPN would install the pushed "
"routes itself, in the MAIN table, and the VPN would become the node's default route "
"(docs/03: 'does **not** set the node's default route (`route-noexec` + policy "
"routing)')")
check(re.search(r'pull-filter ignore "redirect-gateway"', conf_t) is not None,
"the OpenVPN wrapper config does not ignore a pushed `redirect-gateway` — the VPN "
"provider could redirect the node's whole default route into the tunnel, collapsing "
"the two paths into one")
check(re.search(r"^\s*redirect-gateway", conf_t, re.M) is None,
"the OpenVPN wrapper config itself sets `redirect-gateway` — that IS making the VPN the "
"default route")
# ================= (5) THE KILL-SWITCH BLACKHOLES ONLY THE VPN CLASS (internet fails OPEN)
src_set = str(defaults.get("vpn_egress_gateway_nft_src_set", ""))
fwmark = str(defaults.get("vpn_egress_gateway_fwmark", ""))
check(bool(src_set),
"the role declares no nftables source set — the kill-switch would have no way to name "
"'the steered class' and could only match broadly")
policies_lines = [l.strip() for l in nft_t.splitlines() if re.search(r"^\s*type filter hook", l)]
check(len(policies_lines) >= 1,
"the kill-switch nft template declares no base chain — nothing would be enforced")
for l in policies_lines:
check(re.search(r"policy\s+accept\s*;", l) is not None,
f"an nft base chain is not `policy accept`: {l!r}. A default-drop chain here "
"blackholes the `internet` path too, and docs/03 requires the two classes to fail "
"in OPPOSITE directions — 'internet fails **open** (WAN, never blackholed)'")
drop_lines = [l.strip() for l in nft_t.splitlines()
if re.search(r"\bdrop\b", l) and not l.lstrip().startswith("#")]
check(len(drop_lines) >= 1,
"the kill-switch nft template contains no drop rule at all — there is no kill-switch, "
"and R-NET-04's 'the kill-switch blackholes only that class' has nothing to scope")
for l in drop_lines:
scoped = (src_set and src_set in l) or (fwmark and "mark" in l)
check(scoped,
f"an nft DROP rule is not scoped to the vpn class: {l!r}. It must match the tunnel "
f"source set (@{src_set}) or the class's fwmark; an unqualified drop would blackhole "
"the default `internet` path, which must never be blackholed")
# --------------------------------------------------------------------------- verdict
# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
check(CHECKS >= 35,
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
"would be meaningless")
if FAILURES:
die()
print(f"check OK: egress-paths-selected-by-rules — {CHECKS} assertions; "
f"paths={sorted(PATHS)} (default={egress_prop.get('default')}), "
f"{len(policies)} steering rule selecting {CLASS_LABEL}=vpn only, tunnel default route "
f"in dedicated table {table_id} behind a scoped ip rule, blackhole and every nft DROP "
f"scoped to the vpn class. R-NET-04's RULE-SET half only: the paths' actual BEHAVIOUR "
"is L2 (egress-class for the two labelled classes, egress-default-path for the "
"unlabelled default) and the kill-switch firing is L3 (vpn-killswitch).")
PY
@@ -8,6 +8,18 @@
# the empty string while the Kustomization still reports Ready — which emptied Cilium's
# k8sServiceHost and broke the CNI on 2/3 dev nodes (DECISIONS-LOG 2026-09-01).
#
# It is also LOSSY IN THE OTHER DIRECTION, and that is the second incident on this record.
# envsubst consumes dollars in PAIRS and emits one per pair, unconditionally and everywhere —
# so no substituted path can carry a literal `$$` at all. On 2026-09-13 a ConfigMap under
# apps-wave-0 shipped a dollar-quoted PL/pgSQL `DO $$ ... $$;` self-check; Postgres received
# a half-quoted `DO $` and every Flux-reconciled run of the nextcloud-db grant reconciler died
# with a syntax error, nine Error pods deep before anyone looked (commit 16b7fcd). It passed
# review because it was hand-applied with `kubectl apply`, which runs no envsubst: the defect
# existed ONLY on the rendered path, and it landed on the assertion whose whole job was to
# make a silent regression visible. Nothing here failed — the manifest rendered
# "successfully" into something broken, because the check treated any doubled dollar as a
# deliberate escape and stopped looking.
#
# GIVEN every Flux Kustomization under clusters/ that declares postBuild.substituteFrom,
# WHEN each `spec.path` is resolved to the exact file set it renders (following
# kustomization.yaml `resources:` transitively, not globbing the repo),
@@ -15,7 +27,12 @@
# or deliberately ESCAPED `$${...}`; no unescaped expansion carries a default
# operator on an undefined name (the silent-empty class strict mode does NOT catch);
# no unescaped expansion is unparseable by envsubst (which fails the WHOLE
# Kustomization); every path rendering a knob actually substitutes cluster-vars;
# Kustomization); NO RUN OF TWO OR MORE DOLLARS IS SILENTLY REWRITTEN — the only shape
# that passes unremarked is exactly `$${`, the documented total escape, and every other
# run (a bare `$$` that cannot survive verbatim, an ODD `$$${...}` that only LOOKS
# escaped and in fact expands, an over-counted `$$$${...}`) is named and justified in
# `dollar-collapse.allow`, which is itself held to being exact and non-stale;
# every path rendering a knob actually substitutes cluster-vars;
# every overlay-patched knob has a base default; and the check is wired into the
# merge gate as a blocking, unconditionally-run stage.
#
@@ -26,6 +43,19 @@
# only); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw
# Test wraps the SAME script.
#
# NOT proven, and deliberately NOT proxied: the wider lesson of 2026-09-13 is that the bug
# survived review because verification used a path (`kubectl apply`) that runs no
# substitution. That property is NOT statically decidable from this repo — which command
# someone ran is a fact about a shell session, and git stores the manifest, never the
# invocation. Grepping commit prose or runbooks for `kubectl apply` would be trivially
# satisfiable by rewording and would fire on the break-glass runbooks whose whole content is
# that command, so no such proxy is attempted. The dollar-run assertion is the honest answer
# instead: it moves the proof off the verification PATH and onto the SOURCE, where it IS
# decidable, so the mangling is caught by reading the manifest however a human chose to try
# it. An end-to-end `kustomize build | flux envsubst --strict` per substituted path would be
# the real thing, and it is how the arithmetic in check.sh's header was measured — but the L0
# CI toolchain installs neither tool, so as a gate stage it would SKIP permanently.
#
# NOT proven (named in check.sh's header): this is a STATIC scan of the SOURCE manifests
# a path includes — a deliberate superset of the post-`kustomize build` stream. It does
# not render, does not prove the feature gate is enabled on the live kustomize-controller,
@@ -43,18 +73,36 @@ metadata:
annotations:
conformance.home.k3s/requirements: "R-ENV-02, R-GITOPS-01"
conformance.home.k3s/layer: "L0"
# WHOLE-REPO SCOPE — relocated verbatim from ci/l0.sh's hand-written "Stage 3c
# (UNCONDITIONAL)", which this annotation replaces:
#
# R-ENV-02's promote-once mechanism is `postBuild.substituteFrom cluster-vars`, and it
# FAILS OPEN: an undefined ${TOKEN} becomes the empty string and the Kustomization still
# reports Ready. That emptied Cilium's k8sServiceHost and broke the CNI on 2/3 dev nodes
# (DECISIONS-LOG 2026-09-01, "substitution race that fails OPEN").
#
# WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh's coarse path map
# keys only clusters/* to R-ENV-02 — an edit to a SUBSTITUTED manifest under apps/ or
# platform/ attributes no requirement, so the requirement-scoped selection would filter
# this check out of exactly the change that introduces the bug. The invariant spans every
# path any Flux Kustomization substitutes (apps/, platform/, platform-config/), so it runs
# on EVERY gate invocation.
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Every variable a substituted Flux path renders is defined in cluster-vars or
escaped, so post-build substitution cannot silently empty a value and enabling
StrictPostBuildSubstitutions cannot fail a legitimate manifest.
StrictPostBuildSubstitutions cannot fail a legitimate manifest; and no dollar run
envsubst rewrites reaches the cluster unreviewed, so content that must arrive
verbatim cannot be silently mangled on the Flux-rendered path (16b7fcd).
spec:
steps:
- name: audit-every-substituted-flux-path
description: >-
Resolve each postBuild.substituteFrom Kustomization to the files it renders and
assert every ${...} is defined or escaped, then assert the gate wires it blocking.
assert every ${...} is defined or escaped and no dollar run is silently collapsed,
then assert the gate wires it blocking.
try:
- script:
content: bash ./check.sh
@@ -18,6 +18,32 @@
# substituted path renders is either DEFINED or deliberately ESCAPED — and nothing in
# the repo asserted that. This check does.
#
# SECOND INCIDENT, 2026-09-13 — THE INVERSE HAZARD (commit 16b7fcd)
# ------------------------------------------------------------------
# The first incident was a variable the author wanted EXPANDED that rendered EMPTY. This one
# is its mirror: content the author wanted to reach the workload UNTOUCHED that substitution
# silently REWROTE. A CronJob reconciler in apps/wave-0/nextcloud-db.yaml asserted its own
# convergence in a dollar-quoted PL/pgSQL block —
#
# DO <dq> BEGIN ... END <dq>; where <dq> is two literal dollars: standard Postgres
#
# — inside a ConfigMap under apps-wave-0, which declares postBuild.substituteFrom. envsubst
# collapses a doubled dollar into ONE dollar, ALWAYS, so Postgres received a half-quoted
# `DO <one dollar>` and died with `syntax error at or near "$"`. Three scheduled runs left
# nine Error pods in nextcloud-db before anyone looked.
#
# It PASSED verification because it was hand-applied with `kubectl apply`, which runs NO
# envsubst. The defect existed ONLY on the Flux-rendered path. And the damage landed
# precisely on the self-check: the GRANT on line 1 still ran, so the privilege stayed
# correct and the only thing broken was the assertion meant to make a silent regression
# visible — the part whose whole job is to stop a blind spot being quiet.
#
# THE ASSERTION GAP THIS CLOSES: assertions (1)-(3) below ask whether every `${...}` is
# defined, escaped, or parseable. They said NOTHING about a doubled dollar that is not an
# escape, because the old classifier treated a `$` run of length >= 2 as "escaped, therefore
# fine" and stopped looking. Nothing failed the gate; the manifest rendered "successfully"
# into something broken. Assertion (7) is that hole, closed.
#
# ENVSUBST BEHAVIOUR ASSERTED HERE (empirically verified against the pinned toolchain,
# `flux envsubst`, flux 2.4.0 — this is measured behaviour, not assumption):
# ${DEFINED} -> substituted with the cluster-vars value.
@@ -33,6 +59,66 @@
# shell/cloud-init consumer. Escaping is total: it neutralises
# even the indexed form.
# $VAR, $(cmd) -> left alone by envsubst; they need no escaping.
#
# DOLLAR-RUN ARITHMETIC — the exact rule, MEASURED, not inferred (see assertion (7))
# ---------------------------------------------------------------------------------
# envsubst here is `github.com/fluxcd/pkg/envsubst` v1.1.0 — Flux's own maintained FORK of
# `github.com/drone/envsubst` ("This package is a fork of drone/envsubst", its README) — and it
# is exposed by the `flux envsubst` CLI ("replicate the behavior of the Flux Kustomization
# post-build substitutions", its own --help). Named from the binary, not from memory: the
# pinned flux 2.4.0 in the dev image embeds the Go module path
# `/home/runner/go/pkg/mod/github.com/fluxcd/pkg/[email protected]/parse`
# (`grep -ao 'github.com/fluxcd/pkg/envsubst[^ ]*' "$(command -v flux)"`). Its scanner walks
# the input left to right; on a `$` it looks at the NEXT byte:
# * another `$` -> BOTH are consumed and ONE `$` is emitted. This is UNCONDITIONAL. What
# follows the pair is irrelevant — `{`, a letter, a digit, a semicolon,
# a newline, end of input. `$$` is not "the escape for `${`"; it is the
# escape for `$` itself, everywhere.
# * `{` -> the braced expression is parsed and EXPANDED.
# * anything else -> the `$` is emitted literally (hence `$VAR` and `$(cmd)` survive).
# So a run of N consecutive dollars emits floor(N/2) literal dollars, and when N is ODD the
# final dollar is still LIVE. Measured with flux 2.4.0 `flux envsubst`, DEFINED=VALUE — the
# left column is authored bytes, the right is what reaches the cluster:
#
# $ -> $ ${DEFINED} -> VALUE
# $$ -> $ $${DEFINED} -> ${DEFINED} <- the escape
# $$$ -> $$ $$${DEFINED} -> $VALUE <- ODD: brace LIVE
# $$$$ -> $$ $$$${DEFINED} -> $${DEFINED}
# $$$$$ -> $$$ $$$$${DEFINED} -> $$VALUE
# DO $$ .. $$; -> DO $ .. $; $$${NOPE:-d} -> $d
# awk '{print $$2}' -> awk '{print $2}' $${ARR[0]} -> ${ARR[0]} (safe)
# $$@ $$< $$^ -> $@ $< $^ $$${ARR[0]} -> HARD PARSE ERROR, whole
# Kustomization dies
#
# TWO CONSEQUENCES THIS CHECK IS BUILT ON:
# (a) THERE IS NO WAY TO SHIP A LITERAL `$$` THROUGH A SUBSTITUTED PATH. To make two
# dollars arrive you must author FOUR. So any authored `$$` that is meant to reach the
# workload verbatim is a BUG, full stop — that is the 2026-09-13 incident, and Postgres
# dollar-quoting is its commonest shape (Makefile `$$@`, awk `$$2` and docker-compose
# `$${}` are the same hazard arriving from the other direction: there the collapse is
# what the author wanted).
# (b) `$$$` + brace is NOT "more escaped" than `$$` + brace — it is NOT escaped at all. An
# odd run leaves one live dollar, so the brace expands, or, for an inner envsubst
# cannot parse, fails the ENTIRE Kustomization (proven above with $$${ARR[0]}, which
# aborted the whole stream while $${ARR[0]} on the line before it was fine).
# `--strict` reports NONE of this: every row of that table except the parse error was
# accepted SILENTLY by `flux envsubst --strict`, exit 0, and its output was BYTE-IDENTICAL to
# non-strict (`cmp`). Strict mode cannot see this class, so a source-level assertion is the
# only available guard.
#
# TO REPRODUCE THE TABLE (30 seconds, no cluster) — it is a measurement, so re-measure it
# after any FLUX_VERSION bump in infra/devcontainer/versions.env. Open the toolbox with
# `./infra/devcontainer/dev.sh` and run these INSIDE it (do NOT pipe into `dev.sh run` from
# the host: it starts the container with `-it`, so a piped stdin hangs):
# printf '[$$] [$$$] [$${D}] [$$${D}] [DO $$ x $$;]\n' | D=VALUE flux envsubst
# -> [$] [$$] [${D}] [$VALUE] [DO $ x $;]
# And end-to-end, on the actual delivery path, with the pre-16b7fcd manifest checked out
# (`git show 16b7fcd^:apps/wave-0/nextcloud-db.yaml > apps/wave-0/nextcloud-db.yaml`):
# kubectl kustomize apps/wave-0 | grep 'DO \$' -> "DO $$" (the kubectl path)
# kubectl kustomize apps/wave-0 | flux envsubst --strict | grep 'DO \$'
# -> "DO $" (the FLUX path), exit 0
# (`kubectl kustomize` IS the kustomize build here; the image ships no standalone binary.)
# That two-command contrast IS the 2026-09-13 incident, and is why review missed it.
# ${...}, ${a b}, ${ARR[0]} (UNESCAPED) -> HARD PARSE ERROR ("unable to parse
# variable name" / "missing closing brace") which fails the
# WHOLE Kustomization — every resource in the path, not just
@@ -55,7 +141,14 @@
# (5) every knob an overlay patches (`/data/KEY`) exists in the BASE ConfigMap —
# the "add the knob with a base default BEFORE referencing it" rule that the
# incident's interim discipline depends on;
# AND (6) this check is LOAD-BEARING: ci/l0.sh runs it on every gate run and
# (7) NO DOLLAR RUN IS SILENTLY REWRITTEN: the only run of length >= 2 that passes
# unremarked is exactly `$$` immediately followed by `{` — the documented, total,
# already-inventoried escape. EVERY other run of >= 2 dollars (a bare `$$`, an odd
# `$$$`, an over-counted `$$$$`) is rewritten by envsubst between git and the
# container, so it must be named and justified in `dollar-collapse.allow`, a file
# this check also holds to being exact and non-stale;
# AND (6) this check is LOAD-BEARING: it DECLARES `conformance.home.k3s/scope: whole-repo`,
# so ci/l0.sh's `--include-scope whole-repo` runs it on every gate run, and
# ci/gate.sh sets the merge block when the l0 stage fails.
# Assertion (1)-(3) are applied to COMMENTS as well as content: `kustomize build`
# preserves body comments into the stream envsubst reads (verified: a `$`-carrying
@@ -78,6 +171,21 @@
# kustomize-controller. Both are cluster facts (L2), untested here. Tagging
# R-GITOPS-01 claims ONLY the static slice: what git says a Kustomization renders
# is fully determined, with no variable silently emptied on the way in.
# - It CANNOT prove the thing that actually let 16b7fcd's bug reach a cluster: that a
# change was VERIFIED through a substituting path. "It was hand-applied with kubectl" is
# a fact about a shell session, not about any tracked file — git records the manifest,
# never the command someone ran to try it, so the property is not statically decidable
# here and no honest formulation of it exists. Nor is a proxy worth having: grepping
# commit prose or runbooks for `kubectl apply` would be trivially satisfiable by
# rewording, and would fire on the many legitimate break-glass runbooks whose whole
# content is that command. So this check does not attempt it, and does not pretend to.
# Assertion (7) answers that incident a different way: it moves the proof off the
# verification PATH, which is unknowable, and onto the SOURCE, which is not — the
# mangling is now caught by reading the manifest, whichever way a human chose to try it.
# (The one decidable half-measure, `kustomize build <path> | flux envsubst --strict` per
# substituted path in CI, is already ruled out above: the L0 toolchain installs neither
# tool, so the stage would SKIP permanently. The dev image has both, and it is how the
# arithmetic table above was measured.)
# - It does NOT validate cluster-vars VALUES (that GATEWAY_VIP is inside the LB pool,
# that an overlay's LAN block is self-consistent). Presence, not correctness.
# - A cluster-vars key nothing references is reported as an ADVISORY, never a failure:
@@ -107,6 +215,7 @@ definition of "what a Flux path actually renders" in the harness.
"""
from __future__ import annotations
import bisect
import os
import re
import sys
@@ -230,9 +339,18 @@ LEN_RE = re.compile(r"^#([A-Za-z_][A-Za-z0-9_]*)$")
def classify(dollars: int, inner: str) -> tuple[str, str] | None:
"""Return (severity-tag, remediation) for a hazardous expansion, else None."""
if dollars >= 2:
return None # escaped: emitted as a literal
"""Return (severity-tag, remediation) for a hazardous expansion, else None.
PARITY, not ">= 2" (the 2026-09-13 correction). envsubst consumes dollars in PAIRS and
emits one per pair, so a run of N dollars in front of a brace leaves the brace ESCAPED
only when N is EVEN. An ODD run emits floor(N/2) literal dollars and its last dollar is
still LIVE, so `$$${X}` expands exactly as `${X}` does — and `$$${ARR[0]}` fails the
whole Kustomization where `$${ARR[0]}` is harmless. The old `dollars >= 2 -> return None`
called every such run "escaped" and stopped looking. Assertion (7) flags the run itself;
this function additionally classifies the still-live expansion an odd run carries.
"""
if dollars % 2 == 0:
return None # fully escaped: emitted as a literal
if NAME_RE.match(inner):
if inner in KEYS:
return None # defined knob — the happy path
@@ -267,6 +385,9 @@ def rendered_files(path_spec: str) -> tuple[list[str], bool]:
findings: list[str] = []
referenced: dict[str, set[str]] = {}
inventory: list[str] = []
# rel path -> the Kustomization name(s) that render it, for assertion (7)'s second pass.
# A file can be rendered by more than one Kustomization; all of them substitute.
subst_files: dict[str, set[str]] = {}
for k in kustomizations:
files, exists = rendered_files(k["path"])
@@ -280,13 +401,14 @@ for k in kustomizations:
text = f.read_text()
except (OSError, UnicodeDecodeError):
continue
subst_files.setdefault(rel, set()).add(k["name"])
for lineno, line in enumerate(text.splitlines(), 1):
for m in TOKEN_RE.finditer(line):
dollars, inner = len(m.group(1)), m.group(2)
verdict = classify(dollars, inner)
if verdict is None:
if dollars >= 2:
n_esc += 1
if dollars % 2 == 0: # PARITY, see classify(): an odd run is
n_esc += 1 # live, not escaped
else:
n_def += 1
referenced.setdefault(re.split(r"[:/#%^,]", inner)[0] or inner,
@@ -312,6 +434,156 @@ for k in kustomizations:
check(not findings,
f"{len(findings)} unescaped, undefined or unparseable expansion(s) in substituted paths")
# ----------------------------------- (7) no dollar run is silently rewritten on the way out
# THE 2026-09-13 CLASS. envsubst consumes dollars in PAIRS and emits one per pair,
# unconditionally and everywhere (header: DOLLAR-RUN ARITHMETIC, measured against flux 2.4.0).
# So every run of >= 2 dollars in a substituted manifest is REWRITTEN between git and the
# container, and from the text alone the two intents are indistinguishable:
#
# (a) a deliberate escape the author WANTS collapsed — `$${VAR}` -> literal `${VAR}`;
# (b) content that must arrive VERBATIM — `DO $$ ... $$;` -> `DO $ ... $;`
# i.e. the incident. A BUG.
#
# THE RULE CHOSEN, AND WHY IT IS NEITHER NOISY NOR VACUOUS.
# Exactly ONE shape passes unremarked: a run of EXACTLY TWO dollars immediately followed by
# `{`. That is not a guess about intent, it is the escape contract this repo already
# documents in clusters/base/apps-vms.yaml, that assertions (1)-(3) already validate the
# inside of, and that 18 reviewed occurrences already rely on (unifi cloud-init, the gitea
# and nextcloud OIDC bootstrap scripts, two longhorn comments). Failing those would be a
# permanently-firing alert on documented, correct intent, and this repo has already been
# bitten by one of those.
#
# EVERY OTHER run of >= 2 dollars FAILS unless it is named in `dollar-collapse.allow`:
# * DOLLAR-COLLAPSE — a run not followed by `{`. Cannot survive verbatim: `$$` -> `$`.
# This is the incident's exact shape and a blanket bug by default.
# * DOLLAR-HALF-ESCAPED — an ODD run >= 3 before `{`. Looks escaped, is not: the last
# dollar is live. Occasionally legitimate (`$$${PRICE}` to render a
# literal `$` before a value), so it is allowlistable, not banned.
# * DOLLAR-OVER-ESCAPED — an EVEN run >= 4 before `{`. Emits N/2 dollars then a literal
# brace; nearly always a miscount, sometimes deliberate.
#
# WHY AN ALLOWLIST AND NOT A BLANKET BAN: a blanket ban was the first choice and was
# rejected on evidence, not taste — main carries two real non-braced `$$` occurrences today
# (prose inside embedded shell comments), so a blanket ban would have shipped RED. Nor is
# the allowlist a hole: an entry is a hand-written edit to a file inside the conformance
# test, which is loud in review; it must quote a >= 24-character VERBATIM context, so it can
# only ever cover the occurrence it was written for and never a future one that drifts into
# the same file; it must carry a >= 40-character reason; and every entry must still MATCH
# something, so the allowlist cannot rot into a blanket exemption after the code it excused
# is deleted. A wildcard, a bare `$$` context and a stale entry each FAIL.
ALLOW_FILE = (ROOT / "conformance" / "tests" / "l0-static"
/ "flux-postbuild-substitution" / "dollar-collapse.allow")
MIN_CONTEXT = 24
MIN_REASON = 40
DOLLAR_RUN_RE = re.compile(r"\$+")
allow_entries: list[dict] = []
check(ALLOW_FILE.is_file(),
f"{ALLOW_FILE.relative_to(ROOT)} is missing — assertion (7) needs its allowlist to "
f"exist even when empty, because an absent file and an empty file are the same green "
f"and the next author would not know the mechanism is there")
if ALLOW_FILE.is_file():
for lineno, raw in enumerate(ALLOW_FILE.read_text().splitlines(), 1):
if not raw.strip() or raw.lstrip().startswith("#"):
continue
parts = [x.strip() for x in raw.split("|")]
if len(parts) != 3:
check(False, f"dollar-collapse.allow:{lineno}: expected exactly 3 '|'-separated "
f"fields (path | verbatim-context | reason), got {len(parts)}")
continue
allow_entries.append({"line": lineno, "path": parts[0], "ctx": parts[1],
"reason": parts[2], "used": 0})
for e in allow_entries:
tag = f"dollar-collapse.allow:{e['line']}"
check((ROOT / e["path"]).is_file(),
f"{tag}: names '{e['path']}', which is not a file in this repo — a stale entry is "
f"a standing exemption for a path someone could recreate")
check(len(e["ctx"]) >= MIN_CONTEXT,
f"{tag}: context {e['ctx']!r} is {len(e['ctx'])} chars; at least {MIN_CONTEXT} are "
f"required so an entry cannot be a near-wildcard that swallows future occurrences")
check(DOLLAR_RUN_RE.search(e["ctx"]) is not None
and max((len(m.group(0)) for m in DOLLAR_RUN_RE.finditer(e["ctx"])), default=0) >= 2,
f"{tag}: context {e['ctx']!r} contains no run of two or more dollars, so it excuses "
f"nothing and is either a typo or a decoy")
check(len(e["reason"]) >= MIN_REASON,
f"{tag}: reason is {len(e['reason'])} chars; at least {MIN_REASON} are required. "
f"State what envsubst will actually emit and why that is acceptable — an entry "
f"nobody had to explain is an accident wearing an approval")
RUN_TAGS = {
"collapse": ("DOLLAR-COLLAPSE",
"envsubst collapses each PAIR of dollars into one, everywhere and "
"unconditionally, so this CANNOT reach the workload verbatim (a literal $$ "
"needs $$$$ in source). This is the 2026-09-13 nextcloud-db shape: a "
"dollar-quoted PL/pgSQL DO block arrived half-quoted and every Flux-"
"reconciled run died. Rewrite without dollars, double them again if the "
"collapse is what you want, or justify it in dollar-collapse.allow"),
"half": ("DOLLAR-HALF-ESCAPED",
"an ODD run of dollars before a brace is NOT escaped: floor(N/2) literal "
"dollars are emitted and the LAST dollar is still live, so this expansion "
"EVALUATES at reconcile time (and if its inner is unparseable it fails the "
"entire Kustomization). Use an EVEN run to escape, or justify it in "
"dollar-collapse.allow if a literal dollar before a substituted value is "
"really what you meant"),
"over": ("DOLLAR-OVER-ESCAPED",
"an EVEN run of four or more dollars emits N/2 literal dollars followed by a "
"LITERAL brace — the value is never substituted and extra dollars appear in "
"the rendered manifest. Almost always a miscount of the escape; justify it in "
"dollar-collapse.allow if it is not"),
}
run_findings: list[str] = []
n_runs_ok = 0
for rel in sorted(subst_files):
text = (ROOT / rel).read_text(errors="replace")
# Absolute offsets, not per-line: the gitea/nextcloud bootstrap scripts are single
# 4000-character folded scalars, so "the line" is not a reviewable unit of context here.
nl = [i for i, ch in enumerate(text) if ch == "\n"]
spans = [] # allowlisted [start, end) regions
for e in allow_entries:
if e["path"] != rel:
continue
start = text.find(e["ctx"])
while start >= 0:
spans.append((start, start + len(e["ctx"]), e))
start = text.find(e["ctx"], start + 1)
for m in DOLLAR_RUN_RE.finditer(text):
n = len(m.group(0))
if n < 2:
continue
braced = text[m.end():m.end() + 1] == "{"
if n == 2 and braced:
n_runs_ok += 1 # the documented escape; (1)-(3) own it
continue
covering = [e for (a, b, e) in spans if a <= m.start() and m.end() <= b]
if covering:
for e in covering:
e["used"] += 1
continue
kind = "collapse" if not braced else ("half" if n % 2 else "over")
tag, fix = RUN_TAGS[kind]
lineno = bisect.bisect_right(nl, m.start()) + 1
ctx = text[max(0, m.start() - 60):m.end() + 60].replace("\n", "\\n")
run_findings.append(
f" [{tag}] {rel}:{lineno} a run of {n} dollar(s)"
f"{' before a brace' if braced else ''}\n"
f" rendered by Flux Kustomization(s) {', '.join(sorted(subst_files[rel]))}"
f" — postBuild substitution REWRITES it\n"
f" {fix}\n"
f" | ...{ctx}...")
check(not run_findings,
f"{len(run_findings)} dollar run(s) in substituted paths that envsubst rewrites and "
f"nothing justifies (the 2026-09-13 nextcloud-db class)")
for e in allow_entries:
check(e["used"] > 0,
f"dollar-collapse.allow:{e['line']}: STALE — its context is not present in "
f"{e['path']} any more (or no longer contains a rewritten dollar run), so the entry "
f"now excuses nothing. Delete it. A stale allowlist is how an opt-in decays into a "
f"blanket exemption for whatever lands in that file next")
# --------------------------------------------- (5) overlay knobs exist in the base
# Parse contract: overlays patch cluster-vars with JSON6902 ops (`- op: replace` /
# `path: /data/KEY`) inside an inline `patch: |-` block. The repo rule (cluster-vars.yaml
@@ -328,10 +600,27 @@ for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
# ------------------------------------------------------- (6) the check is load-bearing
L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
check("flux-postbuild-substitution/check.sh" in L0,
"ci/l0.sh does not run flux-postbuild-substitution/check.sh — the requirement-scoped "
"conformance selection can filter this check out of a gate run, so l0.sh must invoke "
"it directly and unconditionally or the gate is not actually enforcing it")
# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism.
# It used to assert `"flux-postbuild-substitution/check.sh" in L0`: ci/l0.sh carried a hand-written
# "UNCONDITIONAL" stage per whole-repo check, hardcoding this path. Those stages are gone
# (they did not scale, and the next whole-repo invariant only got one if someone REMEMBERED
# to write it). The same property is now asserted two ways: this test DECLARES
# `conformance.home.k3s/scope: "whole-repo"`, and ci/l0.sh unions every declaring test into
# the run via `--include-scope`. Neither half alone is enough, so both are checked here; the
# mechanism ITSELF (the union really selects, and a scope-declared SKIP exits 3 rather than
# passing) is proven end-to-end and falsifiably by
# conformance/tests/l0-static/whole-repo-invariants-wired.
SELF_YAML = ROOT / "conformance" / "tests" / "l0-static" / "flux-postbuild-substitution" / "chainsaw-test.yaml"
SELF = SELF_YAML.read_text() if SELF_YAML.is_file() else ""
check(re.search(r'^\s*conformance\.home\.k3s/scope:\s*["\']?whole-repo', SELF, re.M) is not None,
"flux-postbuild-substitution/chainsaw-test.yaml does not declare "
'conformance.home.k3s/scope: "whole-repo", so the merge gate selects this check only '
"when a change happens to touch one of the requirements it is labelled with. "
"The path map keys only clusters/* to R-ENV-02, so an edit to a SUBSTITUTED manifest under apps/ or platform/ attributes nothing and this check would be filtered out of exactly the change that empties a variable (the 2026-09-01 Cilium k8sServiceHost CNI outage)")
check("--include-scope" in L0,
"ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is "
"a purely requirement-scoped run again and the scope annotation is inert decoration — "
"this check would then run only by coincidence")
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
"ci/gate.sh does not set the merge block when the l0 stage fails")
@@ -345,6 +634,10 @@ print("Flux postBuild substitution inventory (per Kustomization):")
for line in inventory:
print(line)
print(f"\ncluster-vars knobs defined: {len(KEYS)} referenced: {len(referenced)}")
print(f"dollar runs >=2 in substituted files: {n_runs_ok} documented $${{...}} escapes, "
f"{sum(e['used'] for e in allow_entries)} justified by dollar-collapse.allow "
f"({len(allow_entries)} entr{'y' if len(allow_entries) == 1 else 'ies'}), "
f"{len(run_findings)} unjustified")
if ADVISORIES:
print("\nADVISORIES (not failures):")
@@ -356,6 +649,12 @@ if findings:
for f in findings:
print(f, file=sys.stderr)
if run_findings:
print("\nHAZARDS — dollar runs envsubst rewrites between git and the container:",
file=sys.stderr)
for f in run_findings:
print(f, file=sys.stderr)
print()
if not FAILURES:
print(f"check OK: flux-postbuild-substitution — {CHECKS} assertions "
Loaded 100 of 319 files, more files were not shown because too many files have changed in this diff. Show more