The drift detector merged in #11 was inert and would have stayed inert silently.
WHAT HAPPENED. infra/gitea/set-ci-token.sh minted a token and PUT it to the repo Actions
secret GITEA_SETTINGS_TOKEN. Gitea RESERVES the `GITEA_` prefix for secret names -- exactly
as GitHub reserves `GITHUB_` -- and rejects that PUT with:
HTTP 400 {"message":"invalid secret name"}
The script sent the PUT to /dev/null and printed "==> stored." unconditionally, and
`curl -sS` exits 0 on an HTTP error, so it reported success for a store that never happened.
Measured afterwards: the token existed in the user's token list, and the repo's secret list
was EMPTY. ci/gitea-settings.sh would have skipped on every CI run forever, and with
REQUIRE_GITEA_SETTINGS=true it would instead have BLOCKED EVERY PR -- including the one
fixing it.
Three changes:
* the secret is SETTINGS_API_TOKEN; ci/gitea-settings.sh reads it, keeping
GITEA_SETTINGS_TOKEN as a local-run fallback;
* set-ci-token.sh checks the PUT's HTTP status AND reads the secret back, refusing to claim
success otherwise. That missing verification is the actual defect -- the wrong name was
just what it hid;
* gate.yml maps the secret and ENABLES REQUIRE_GITEA_SETTINGS.
WHY ENABLING IT HERE IS SAFE: this PR's own CI run is the proof. If the stage were still
skipping, REQUIRE turns that into a failure and this PR goes RED rather than merging. Green
means the token resolved and live Gitea matched infra/gitea/expected-branch-protection.json.
Verified locally before pushing:
REQUIRE=true, token present -> MERGE GATE: PASS
REQUIRE=true, no token -> gitea-settings SKIPPED -> BLOCKED (not a silent skip)
The secret is already stored and read back as present, so CI has what it needs.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Every check in this repo reads the repository, the rendered manifests, or the cluster.
NONE could see Gitea's own database, and that is where branch protection and runner
registration live. Three failures came out of that blind spot on 2026-09-29, all the same
shape -- a setting that READ as enforced while not being, with nothing able to notice:
* main had NO branch protection for ~a month while DECISIONS-LOG recorded it as a
required status check;
* ZERO Actions runners were registered, so PR #1 -- the first PR this repo ever had --
sat pending for 30.6 minutes with nothing to execute it;
* after enable_status_check and required_approvals were turned on they STILL did not
bite, because block_admin_merge_override was false: PRs #5-#9 merged with zero
approvals recorded against required_approvals: 1.
infra/gitea/*.sh ASSERT the desired state. Nothing DETECTED drift from it. This is the
detector.
ci/gitea-settings.sh runs as a merge-gate stage: it diffs the live rule at
/branch_protections/main against infra/gitea/expected-branch-protection.json and asserts at
least one runner is online. It reads the RULE object, not /branches/main -- the branch view
does not expose block_admin_merge_override and showed all-green while the rule was
overridable by the only account that merges.
Verified by test, not assumed:
no token -> exit 3, skips loudly, does NOT block
live == declaration -> exit 0
injected drift -> exit 1, blocks, naming both the drifted key and the runner count
The skip is DELIBERATELY not in gate.sh's strict-skips loop. Making an absent credential
block every PR is precisely the lock-yourself-out failure this repo keeps walking into, so
a skip denies a PASS (exit 3) and says loudly what is unverified. REQUIRE_GITEA_SETTINGS=true
upgrades it to a block once the token is wired.
gate.yml's `|| [ "$?" -eq 3 ]` tolerance now covers a second stage. That widening is
declared in the file rather than left to be discovered, and it narrows back the moment
REQUIRE_GITEA_SETTINGS is set.
infra/gitea/set-ci-token.sh mints the token and stores it as the repo Actions secret without
ever printing the value.
UNTIL THAT TOKEN EXISTS THIS STAGE SKIPS, AND A SKIPPED DETECTOR DETECTS NOTHING.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
enable_status_check and required_approvals were set earlier today and recorded as done.
They did not bite. block_admin_merge_override was false, and Gitea lets repo admins
override merge requirements without it. jakman85 holds owner, so both requirements were
bypassable by exactly the account that performs every merge.
The evidence was already on the record: PRs #5-#9 all merged while required_approvals: 1
was in force, with ZERO approvals recorded on any of them.
Two process failures, both the same shape as the defect this whole session started with:
1. Verified with the wrong view. /branches/main reports protected, enable_status_check and
required_approvals but NOT block_admin_merge_override -- it showed everything green while
the rule was overridable. /branch_protections/main is authoritative.
2. Probed for a field name assumed to exist. This Gitea (1.24.3) has no `apply_to_admins`;
asking jq for it returns null, which reads like "set to false" but means "key absent", so
a PATCH that silently did nothing produced a convincing before/after. jq returning null
is not a value -- it may be an absence. Enumerate with to_entries.
require-gate.sh now sets block_admin_merge_override: true; block-admin-override.sh fixes it
in isolation. Both carry the field-name warning.
This PR is the test. If it merges WITHOUT an approval, the fix did not work.
Also records the governance question: with only jakman85 (human) and gitea-admin (agent),
a human-authored PR now needs the automation account to approve it. That is ceremony, not
review, and the choice between keeping it, dropping approvals in favour of the status
check, or adding a second human is the operator's to make.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The branch-protection + PR-gate + sidecar sequence is complete and verified. This records
the finished state and fixes what the sequence itself proved wrong.
CORRECTS A FALSE CLAIM I SHIPPED. PR #7 said the sidecar conversion would remove the
one-per-start dockerd line "http: TLS handshake error from 127.0.0.1: EOF". It does NOT.
Exactly one still appears, immediately after "API listen on [::]:2376" -- it is now the
startupProbe's FIRST attempt, landing microseconds after dockerd begins listening and cut
off by timeoutSeconds: 1; the next poll succeeds. Cosmetic, and deliberately not chased:
raising the timeout to hide one benign startup line would slow every subsequent poll. The
manifest comment now states this instead of the wrong prediction.
Lands infra/gitea/require-gate.sh and verify-then-enforce.sh, which drove the rollout from
a session temp dir. verify-then-enforce.sh merges nothing -- merging is a human's reviewed
act -- but forces the reconcile and ABORTS unless the pod is healthy, a runner is online,
and a real workflow goes green on it, before enabling enforcement and probing the push.
Re-run it after any future act-runner change.
Trims the now-obsolete feat/ rationale from claim-ship.sh: per-branch push triggers were
deleted, so no prefix produces a duplicate run any more.
CLAUDE.md's enforcement table flips from "weaker than this rule" to machine-enforced, with
all four rows marked verified and the push rejection noted as PROVOKED rather than assumed.
It also now states what a green PR does NOT prove: CLUSTER_AVAILABLE is unset, so L2/L3
never runs in CI and only the L0/L1 offline subset is covered.
infra/gitea/README.md marks the sequence done and names the root cause that remains
unfixed: Gitea repo settings live in its own database, invisible to Flux and unreadable by
the conformance gate, so NOTHING detects drift from what this directory declares. These
scripts assert desired state; they are not a detector. The check that reads the live API
and fails on divergence is the highest-value work left, and it is the difference between
this being fixed and this being unable to recur.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The 2026-09-29 session found the merge gate did not exist -- no branch protection, no
runner -- while the tree asserted both. This corrects the documents that carried, and in
several cases CAUSED, that belief.
CLAUDE.md gains section 2b: never infer platform state from this repository. It carries
the specific API and kubectl checks for protection, runners, CI status, workloads and
Flux revisions, and the rule that a green tick is not proof anything ran (a skipped Gitea
job reports success). This is the rule whose absence produced every other finding.
.gitea/workflows/README.md and merge-gate.yaml both said "there is no remote git host,
this file is dormant". Gitea had been live for weeks. Both were written as plans and never
revisited once the plan completed. The Transition section is now marked COMPLETE and says
what is genuinely still outstanding (CLUSTER_AVAILABLE unset, so L2/L3 never runs in CI).
DECISIONS-LOG 2026-08-31 is annotated IN PLACE. Anyone reading chronologically previously
hit "required status check; runner incluster-dind online" with nothing marking it false
until a month later. The correction now sits on the bullet itself.
ci/README.md, clusters/README.md and both conformance READMEs carry a status banner: they
speak as though no cluster exists, which makes a live L2/L3 coverage gap read as an
expected phase. That framing is why the gap went unexamined.
apps/wave-1/gitea-actions-runner-secret.sops.yaml: fixed a wrong path, replaced advice to
`sops -d` into a terminal (which CLAUDE.md forbids) with the rotation script, and recorded
that a fresh token alone does not re-register a runner holding a stale /data/.runner.
apps/wave-0/plex-claim-secret.sops.yaml: the header claimed the file ships the placeholder.
It has not since the claim -- it holds the spent token, which is inert and deliberately
left in place.
catalog/plex.yaml and the plex secret both asserted a "~4-minute TTL" as fact. It was
uncited, upstream documents no expiry, and it was the basis for a since-disproven claim
that the pipeline was too slow to fit inside it. Measured: ~2m45s.
Both .sops.yaml files were verified to still decrypt after the comment edits
(`sops -d > /dev/null`), and render==golden confirms the catalog comment does not
propagate into the rendered manifest.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
A systematic grep for the TTL assertion found it still in runbooks/README.md and in
runbooks/kustomization.yaml, outside the page being rewritten. Same correction: the
countdown on plex.tv/claim is the authority, not a number in our tree.
README.md also now records that the server is CLAIMED and points at claim-ship.sh with
the measured ~2m45s loop, rather than describing a push-based sequence that no longer
exists.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Executes the FOLLOW-UP that has been carried in this manifest since the readiness fix,
now that k3s is v1.36 and sidecar containers are GA.
dind moves into initContainers as the first entry with restartPolicy: Always -- the one
field that makes an initContainer a sidecar: it starts before the regular containers,
runs alongside them, and is terminated after them. kubelet therefore enforces the
ordering that the runner used to hand-roll, so the runner drops its command: override
and runs the image's own entrypoint.
The startupProbe execs a full TLS-verified `docker ... info`, NOT a tcpSocket probe. A
tcpSocket probe would re-open the exact gap the old Stage 1 existed to close: dockerd
writes the TLS certs and only THEN starts listening, so the port can accept a connection
while the certs are absent, and act_runner hard-exits on a missing
/certs/client/ca.pem. `docker info` proves both that the certs are usable and that the
daemon answers. periodSeconds 1 x failureThreshold 60 preserves the previous 60s cap.
It also removes the one-per-start dockerd log line
http: TLS handshake error from 127.0.0.1: EOF
which was the old `nc -z` probe opening a TCP connection and closing it without a TLS
handshake -- an error we were generating ourselves and then reading in the logs.
SECONDARY WIN: deleting the shell deletes an envsubst hazard. apps-wave-1 runs postBuild
substitution over this ENTIRE manifest, comments included, and those loops depended on
bare dollar-names passing through untouched -- a braced or double-dollar spelling would
have silently unbounded the waits or failed the whole Kustomization. The only dollar left
in the file is the intended ${BASE_DOMAIN}.
LANDS ALONE, DELIBERATELY. This Deployment is the merge gate for the entire repository:
if the conversion is wrong, nothing can merge. Merge this BEFORE enabling the required
status check, and watch a real workflow run green on the converted pod before doing so --
otherwise a bad conversion locks the repo with no way to revert through the gate.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The tracked next-step said "align gate.yml's push trigger with this repo's feat/**
prefix". Doing that literally would have made things worse, so this does the opposite.
act_runner runs at capacity 1. A branch push and its PR fire the SAME gate over the SAME
tree, and the redundant run SERIALISES AHEAD of the PR run -- doubling the wall clock on
the one result that gates the merge. Adding another prefix widens that, it does not fix
it. infra/plex/claim-ship.sh currently has to pick a feat/ branch name specifically to
DODGE this, during a loop that is racing a short-lived token.
The prefix list also silently failed to match this repo's own convention: branches here
are feat/**, which was absent, so the "fast feedback" never fired for the most common
prefix. Nobody noticed for a month because the only symptom of a wrong entry is a MISSING
run. Any hand-maintained prefix list has that failure mode. Deleting the list removes the
whole class rather than correcting one entry.
It was redundant anyway: CLAUDE.md already requires ./ci/gate.sh in the pinned toolbox
before pushing (74s locally), and the PR gate re-runs it authoritatively.
push: [main] is KEPT -- post-merge main must still be verified, and it is the only run for
which no PR context exists.
FOLLOW-UP once this and #5 are both merged: the feat/ rationale in
infra/plex/claim-ship.sh becomes obsolete and its comment should be trimmed. The script
keeps working either way.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks
instructed is REJECTED. The wording was merely stale before; it is now actively wrong,
and it is the line an operator reads mid-incident.
Every occurrence now describes the real path: push a branch, open a PR, merge once the
gate is green -- that merge is the deploy. 35 files shared one identical block;
flux-reconcile.md, README.md and plex-claim.md each said it their own way and were
corrected individually.
plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW
CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the
page only applies after a /config wipe.
Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate ->
merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose
under time pressure and two are SILENT when skipped: --with-source (without it the
Kustomization applies the previously-fetched revision), the mandatory rollout restart
(only the ciphertext changes, so the pod template is byte-identical and nothing restarts
itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside
gate.yml's push trigger fires a redundant run that serialises ahead of the PR run).
Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream
pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority.
Measured loop: ~2m45s.
DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false,
the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction
that a skipped Gitea job reports success rather than hanging.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The runner came up on 2026-09-29 and the first CI jobs this repo has ever run went
green, which falsified a claim merged in #2.
MEASURED:
gate / gate (pull_request) success 1m58s (PR 1), 1m34s (PR 2)
gate / gate (push) success 1m32s (PR 2 only)
gate / conformance (pull_request) success "Has been skipped"
#2 said listing the conformance context as a required check would deadlock every PR,
because the job is `if: false` and never runs. That is wrong. Gitea reports a skipped
job as SUCCESS. Requiring it would not deadlock anything -- it would pass VACUOUSLY,
forever, while nothing ran.
The conclusion is unchanged (never require it) but the reason inverts, and the corrected
reason matters more: under HARD RULE 4 a permanently-green check that verifies nothing is
worse than a deadlock. A deadlock is loud. This is the exact fail-open shape the
2026-09-18 audit and this one both keep finding.
Also records that the (push) contexts only appear for branches matching gate.yml's push
trigger -- feature/**, fix/**, docs/**, chore/** -- and NOT feat/**, which is why PR 1's
feat/ branch produced only pull_request contexts. Require the pull_request context alone,
or PRs from non-matching prefixes never satisfy the rule.
CLAUDE.md's enforcement table is updated: the runner half is genuinely fixed and observed
green, the branch-protection half is not. Adds the warning that a green tick is not by
itself evidence that anything ran.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
No runner is registered at repo, org or site level, so gate.yml has never executed.
PR #1 sat pending for 30.6 minutes across 122 polls and never reported. Requiring the
gate as a status check in that state would lock the repo permanently.
Adds runner-token-rotate.sh: mints a fresh instance-level registration token from the
Gitea API and writes it into the SOPS secret via sops set, without ever printing the
value.
Records the likely root cause, which a new token alone does NOT fix. act_runner
registers once and persists its identity in /data/.runner on the act-runner-data PVC.
If Gitea no longer holds a matching runner row, the pod starts, sees .runner, SKIPS
registration entirely, and polls with an identity Gitea does not recognise. Symptom:
zero runners, pod running, nothing obviously broken. Clearing the PVC is what forces a
genuine re-registration.
The README now sequences runner-first, enforcement-second, and requires CONFIRMING the
runner reports online rather than assuming it. That assumption is the whole defect:
DECISIONS-LOG recorded the runner as online on 2026-08-31 and nobody checked since.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Gitea repo settings live in its own database, not in a manifest Flux reconciles, so
they are drift-prone and invisible to the conformance gate -- which is how main came
to accept direct pushes while the log recorded it as protected.
Adds infra/gitea/branch-protection.sh (blocks direct pushes to main) and a README
carrying the measured before-state and the ORDER the remaining steps must happen in.
The script deliberately leaves enable_status_check:false and required_approvals:0.
Both would deadlock today: there is no runner to satisfy a required context, and the
only human account is not yet a repo collaborator, so no one could supply an approval.
The README sequences both.
It prints a push probe to run afterwards. The API response is not proof -- Gitea may
let site admins bypass protection, and that has to be observed, not assumed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Going to land the plex claim wiring through the documented PR flow surfaced that
the flow has never run. PR #1 is the first pull request this repo has ever had.
Verified against the live Gitea, not inferred from the docs:
* branch_protections is [] -- main is protected:false, enable_status_check:false,
status_check_contexts:[], required_approvals:0, user_can_push:true
* ZERO Actions runners registered at repo, org and site level
DECISIONS-LOG 2026-08-31 asserts a "required status check" and a runner
"incluster-dind online". Both are false of the live system. The gate is neither
enforced nor able to run, and that went unnoticed because the log claimed
otherwise -- the same fail-open class as the two holes found on 2026-09-18.
Adds CLAUDE.md, the first agent-facing delivery doc here: the
branch -> PR -> gate -> approval -> merge rule, stated alongside the honest fact
that enforcement is currently weaker than the rule, so agents follow the rule
rather than what they can get away with. Carries the toolbox gate invocation, the
two git-worktree workarounds, the rc 0/1/3 contract, and secret handling.
Records the ordering trap: enabling a required status check before a runner exists
deadlocks every PR permanently, including the one that would fix it.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Plex cannot create libraries over the already-mounted 8.33T NFS bulk share while
the server is UNCLAIMED. catalog/plex.yaml carried a long argument for leaving
PLEX_CLAIM unwired; the operator has decided to override it. This implements the
wiring in the way that respects the half of that argument which is still true.
WHAT WAS TRUE AND STILL IS: a https://plex.tv/claim token has a ~4-minute TTL,
far shorter than commit -> push -> Flux reconcile -> pod-roll on its own timers.
So the committed value is a documented PLACEHOLDER and the live token is injected
by the operator inside that window, with the reconcile FORCED rather than waited
for. runbooks/plex-claim.md is that timed procedure.
WHAT WAS NOT TRUE: the old comment claimed "a placeholder/absent claim Secret
would crashloop the pod". Half of that is wrong, and the difference is the whole
design. Established from plexinc/pms-docker's own first-run script (branch
`master`, root/etc/cont-init.d/40-plex-first-run, s6-overlay v2), cross-checked
byte-identical against the same file inside the running :latest image:
* the claim exchange (line 106) is gated on PLEX_CLAIM being non-empty AND the
PlexOnlineToken pref in /config being empty. Once the server is claimed the
block is skipped entirely, so the env is a NO-OP on every later start
whatever it holds ("If server is already logged in, this parameter is
ignored" -- upstream README).
* with an invalid/expired token on a still-unclaimed /config, plex.tv answers
403 with an empty body; curl's status is never checked, the token sed yields
empty, the success branch at line 120 does not fire, there is no else branch
and no non-zero exit, and the script (no `set -e`) exits 0. The pod comes up
UNCLAIMED AND SERVING. It does not crashloop.
* the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS (default 0 = continue even
if a cont-init script failed), so even a non-zero exit would not halt
startup.
An ABSENT Secret is the real hard failure (secretKeyRef has no `optional`, so it
is CreateContainerConfigError), which is why the Secret is committed, encrypted
and listed in the wave kustomization rather than hand-created at claim time.
NO SCHEMA CHANGE. `optional` on secretKeyRef would only be needed if the pod had
to tolerate a missing Secret or could not start without a VALID token. Neither
holds: the Secret always exists in the Flux graph, and an invalid token is
non-fatal. schemas/servicedefinition.schema.json is untouched, and so are
catalog/render.py's env emission and the renderer contract.
Changes:
* apps/wave-0/plex-claim-secret.sops.yaml -- new Opaque Secret plex-claim/plex,
key PLEX_CLAIM, SOPS-encrypted to the committed age recipient. Its header
carries NO dollar-brace expressions on purpose: kustomize preserves body
comments into the stream envsubst reads (per
conformance/tests/l0-static/flux-postbuild-substitution), so quoting the
upstream shell conditions verbatim would have been a live substitution
hazard on apps-wave-0, not prose.
* catalog/plex.yaml -- the PLEX_CLAIM env via valueFrom.secretKeyRef, the
"deliberately NOT set" block rewritten around the TTL + one-shot design, and
the stale crashloop claim in the /config storage comment corrected.
* apps/wave-0/plex.yaml -- re-rendered with the canonical command
(`--all catalog -o apps --wave-layout --no-image-automation`). No L1 golden
under conformance/tests/l1-rendering covers plex, so this is the only
rendered surface that moves.
* catalog/render.py -- plan_apps_waves places `<svc>-claim-secret.sops.yaml`
in the app's OWN wave, like the bootstrap secret, plus a selftest assertion.
Without this, the next `--emit-waves` would have silently dropped the Secret
from apps/wave-0/kustomization.yaml and put the pod into
CreateContainerConfigError. apps/wave-0/kustomization.yaml is the hand-pruned
subset (vm-demo/keycloak are rendered but deliberately unlisted), so its one
new line was applied without re-emitting the rest.
* runbooks/plex-claim.md + its kustomization.yaml generator entry + README
index -- the operator procedure that fits in 4 minutes, including the single
forced-reconcile command and the mandatory pod roll (a changed Secret does
NOT restart anything: PLEX_CLAIM is read only at container start and the pod
template is byte-identical across this change).
Touched requirements: R-SEC-01 (secrets encrypted at rest), R-GITOPS-02
(ServiceDefinition onboarding contract).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
alert-rules-unit-tested decided whether an alert had a unit test with a bare word
match over the whole test file. Any mention counted — including prose in a `#`
comment.
Hit for real while adding platform/monitoring/tests/vpn-egress.test.yaml. Its header
explains that VPNTunnelDown stays quiet during an egress-steering failure, and that
sentence alone made this check declare VPNTunnelDown "asserted on" and demand its
removal from UNTESTED_ALERTS. Removing the entry would have recorded coverage that did
not exist — the precise "looks fixed" outcome this check exists to prevent, produced
by the check itself. (In that instance I made the coverage real instead, which also
exposed that the fixtures were missing tun0's receive counter entirely.)
Now an alert counts only when a test file names it in an assertion position, with
comment lines stripped first so a commented-out assertion does not count either.
BOTH promtool idioms are accepted, because both are real assertions:
alert_rule_test: - alertname: CNPGPrimariesCoLocated
promql_expr_test: - expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
Matching only the first is not a stricter check, it is a WRONG one: it reported
CNPGPrimariesCoLocated as untested when platform/cnpg/alerts.promtool-test.yaml pins
it across four scenarios entirely through the second form. Believing that would have
produced a redundant duplicate test for an alert already covered better than most —
so the first version of this very fix had to be corrected before it landed.
MEASURED BEFORE LANDING, because the registers are shrink-only and a prose-only alert
would have required an entry this check forbids adding: of 61 alerts, 32 are asserted
in one of the two idioms, 29 are not mentioned at all (already declared), and ZERO were
prose-only. So this tightening failed nothing on the day it landed.
My first measurement said 31/30/0 and was wrong in a way worth recording: it scanned
only tests/*.test.yaml and missed the <manifest>.promtool-test.yaml companion
convention, which is exactly what hid the CNPGPrimariesCoLocated false negative.
Verified in both directions: 134 assertions pass with the real coverage intact, and a
temporary prose-only mention of NodeExporterDown (a registered untested alert) leaves
it correctly listed as untested instead of triggering the stale-entry failure.
Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips,
0 unwired.
Co-Authored-By: Claude Opus 5 <[email protected]>
Records the three fixes and, in each case, the thing that was reporting green or red
over something other than what it claimed to measure. Includes corrections to my own
numbers (the '192 unmapped files' figure was my audit script's regex matching across a
newline, not the repo) and the two check defects the egress branch exposed.
PM HANDOFF lists open items in priority order. R-STATE-02 remains the Phase 7 exit
item and is blocked on the permission classifier, not on readiness.
Co-Authored-By: Claude Opus 5 <[email protected]>
Brings the two alerts for the silent egress-gateway failure: VPNEgressSteeringStale
(critical) and VPNEgressUplinkLabelMissing (warning), plus the kube-state-metrics
allowlist entry that makes them non-inert.
WHY THE BRANCH SAT IN THE BACKLOG: it could not pass the merge gate. Three
whole-repo checks failed on it, and each one was a real gap rather than a nuisance:
* alert-runbooks-resolvable — both alerts link to /runbooks/vpn-egress-stale,
which did not exist. An alert firing now would hand the operator a 404.
* alert-rules-unit-tested — alerts.yaml is already a covered file, so new rules
in it must be tested or a "tested" file quietly grows untested rules.
* alert-metrics-have-scrape-targets — see the guard bug below.
WHAT THE ALERTS CATCH. egress-class=vpn pods leave through one gateway node's tun0;
Cilium resolves it to an ifindex and caches it. An OpenVPN reconnect destroys and
re-creates tun0 with a NEW ifindex, and the node label
homelab.keithjackson.dev/egress-uplink=<addr>-<ifindex> is what forces Cilium to
re-resolve. If the label does not follow the tunnel, the cache points at a destroyed
interface and every vpn-class pod is silently dropped — while tun0 stays up and
VPNTunnelDown says nothing. That ran ~80 minutes undetected on the dev gateway.
Verified this is live on prod, not inert: k3s-prod-3 carries egress-gateway=true and
egress-uplink=10.150.192.86-139, and vpn-egress-steering has been in place 14 days.
The label was exported on 0 series before this change, so the allowlist entry is
what switches the alerts on.
runbooks/vpn-egress-stale.md — written and wired into the configMapGenerator, with
the four required operator sections. It leads with proving egress from a pod the
policy selects, because the symptom is "outbound traffic silently disappears", and
it says plainly not to "fix" this by deleting the policy: that does not restore
egress, it sends vpn-class traffic out of the real ISP connection, which is the leak
the class exists to prevent.
platform/monitoring/tests/vpn-egress.test.yaml — four promtool fixtures. 1 and 2 are
the SAME tunnel reconnect with and without the label following it, which isolates the
mechanism the rule actually keys on: a label VALUE change is not a changing sample,
it retires one series and starts another. 3 is the inert case (gateway node, no uplink
label) where the critical rule CANNOT fire, which is why the warning exists.
Two fixes the branch needed that were defects in the checks themselves:
1. alert-metrics-have-scrape-targets asserted the KSM allowlist matched
`nodes=[...egress-gateway]` with a literal closing bracket — i.e. that the list
contained EXACTLY that one label. Adding a second label correctly (egress-uplink,
which these alerts need) therefore FAILED the check while the property it guards
was perfectly satisfied. Loosened to `[,\]]`: in the list, not the whole list. A
guard that breaks when you correctly extend the thing it guards teaches people to
edit the guard, which is how it stops guarding.
2. UNTESTED_ALERTS listed VPNTunnelDown, and the check flagged it as stale because
"asserted on" is a bare word match over the test file — my header COMMENT
mentioning VPNTunnelDown was enough to count as coverage. Rather than delete the
register entry and claim coverage that did not exist, the fixtures now assert
VPNTunnelDown QUIET in all four cases. That required making them faithful: they
had no node_network_receive_bytes_total{device="tun0"} at all, so VPNTunnelDown
would have been firing throughout unnoticed, because promtool only checks the
alertnames you name. A healthy tun0 passes traffic, and that silence IS the claim
these rules exist for.
NOTE FOR LATER, not fixed here: that word-match means any alert named in prose counts
as tested. Tightening it to `alertname:` positions could reveal other prose-only
"coverage", and the register is shrink-only, so the cascade wants doing deliberately
rather than at 01:00 unattended.
Conflicts resolved: kept HEAD's Velero custom-resource-state block and took only the
branch's one-line allowlist change (14 lines down to 1); inserted the branch's
2026-09-12 DECISIONS-LOG entry in date order rather than appending it out of sequence.
Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips,
0 unwired. The new unit tests pass under the pinned promtool 3.14.0.
Co-Authored-By: Claude Opus 5 <[email protected]>
ci/touched-requirements.sh's coarse path map had no arm for apps/*. A change adding
or editing ANY workload on the cluster — 49 tracked manifests across apps/wave-0,
-1, -2 and apps/vms — attributed zero requirements, so the merge gate selected only
the whole-repo-scoped checks and none of the requirement-scoped L2 behaviour tests.
The tests that enforce default-deny, egress class and auth mode did not run on
exactly the changes that can violate them. Kyverno admission and the L0 egress-path
check are real backstops, but admission is post-merge and the gate exists to catch
it first.
platform/homepage and platform/headlamp were unmapped too — the C2 UI. That gap was
already written down in changed_diff()'s rule-3 comment ("the ONLY route to
R-C2-01/02 was rule 3", i.e. editing the C2 test itself) and had sat there since.
Twelve arms added: apps/vms/* (ordered BEFORE apps/*, because `case` takes the first
match), apps/*, platform/homepage|headlamp, platform/kubevirt, platform/gateway,
platform/coredns, platform/crafty, platform/external-snapshotter|backup, runbooks/*,
platform-config/*, platform/kustomization.yaml and .sops.yaml.
platform/kustomization.yaml earns its own arm: it is the assembly list, so one
deleted line disables an entire platform component while every file it points at
stays perfectly valid on disk. That is the most consequential single-line edit in
the repo and it attributed nothing.
Measured effect for an apps/ change, via --print-selection:
before: 0 requirement-scoped tests
after: 13 (3 L1, 8 L2, 2 L3)
Coverage stays 51/51, so the gate's "attributed IDs must be covered" assertion does
not false-block.
Ten new assertions in ci/tests/touched-requirements.test.sh (26 total, all passing),
including the ORDER property — apps/vms/* must win over apps/* — because arm order
is a real failure mode of `case`, not merely a question of both arms existing. Each
fixture cites no R-ID of its own, so any ID in the output came from the PATH and not
the content scan.
A correction on numbers I reported earlier in this session: I claimed "192 unmapped
files", then "182". Both were inflated by my own audit script rather than the repo.
Its regex used `[^)]*`, which matches across a newline, so the first arm parsed as
`case "$1" in\n catalog/*` and never matched anything; I also mis-parsed
alternation arms like `.gitea/*|ci/*`, which made ci/ look unmapped when it is
mapped. True remainder after this change is 18 files, all prose or captured
artifacts (docs/plans/, root READMEs, .standup-dns/*.txt).
Gated: offline l0,l1,meta --include-scope whole-repo, exit 0.
Co-Authored-By: Claude Opus 5 <[email protected]>
Supersedes 05a8094, which fixed the right bug in the wrong layer.
That commit excluded ns velero's volume data because Velero's own transient kopia
maintenance Jobs (emptyDir `plugins` + `scratch`) were failing their
PodVolumeBackups and turning the nightly Backup PartiallyFailed. It deferred the
general fix — skip emptyDir by type — because Velero's `volumeTypes` policy
condition was unproven here and an unparseable policies.yaml breaks EVERY backup.
It said "revisit if the shape shows up elsewhere."
It showed up elsewhere within the hour. A verification Backup came back
PartiallyFailed with 4 errors, all in ns runbooks: the nginx Deployment's emptyDir
`tmp` + `cache`, failing with
error to expose PVB: error getting client pod runbooks-...: pods "..." not found
because pushing a runbook edit rolled that Deployment while the backup was
snapshotting it. Cause worth stating plainly: I triggered that rollout myself, so
those four failures were self-inflicted rather than a standing defect — but the
mechanism is not. A PVB exposes the volume through its pod, so ANY rollout landing
in the backup window fails one, and one failed PVB makes the whole Backup
PartiallyFailed. On a GitOps cluster that is routine.
So the property is what matters, not the namespace: an emptyDir is created empty
with its pod and destroyed with it, so backing one up cannot protect anything
while attempting it can break the backup that protects everything else.
The `volumeTypes` risk was retired by testing, not by hoping. The new policy went
into a THROWAWAY ConfigMap first, leaving the live one untouched, and was proven
on prod (velero 1.18.1) with a control:
control, ns runbooks, NO policy -> 4 PodVolumeBackups (2x tmp, 2x cache), Completed
same namespace, WITH the policy -> 0 PodVolumeBackups, Backup Completed, 0 errors
The control is the point — "zero PVBs" alone is equally consistent with nothing
having been attempted, which is the kind of empty green I have shipped before.
The ns velero exclusion is now reverted as redundant: `plugins` and `scratch` are
emptyDir (verified on a live maintain pod), so the general rule covers them, and
reverting keeps Velero's own resources in schedule (1) where they belong instead
of moving them between schedules for no reason.
Gated: L0 whole-repo 21 cases, 0 failures, 0 skips, 0 unwired.
Co-Authored-By: Claude Opus 5 <[email protected]>
They do not, and the correction changes the operator's plan rather than just a
detail.
Verified live: all three CNPG clusters run 3 instances spread one per node by
required anti-affinity, and k3s-prod-1 holds a REPLICA of each (keycloak-db-2,
nextcloud-db-2, semaphore-db-2). No primary is on node 1, so draining it forces
no failover at all — the best state this maintenance could ask for. The original
sentence was carried over from earlier session state and was simply stale; the
runbook now prints the command to re-derive it, because primaries move on their
own after any restart or failover.
Adds the warning that matters more. CNPGPrimariesCoLocated is currently firing
for k3s-prod-2, which holds both the keycloak-db and semaphore-db primaries. The
instinct is to tidy that before a maintenance window. Doing so would be actively
harmful here: rebalancing means promoting a replica, the only node with spare
primary capacity is node 1, and so "fixing" the alert would put a primary on the
very node about to be powered off — converting a no-failover drain into a
two-database failover. Rebalance after node 1 is back and caught up.
Gated: L0 whole-repo 21 cases, 0 failures, 0 skips, 0 unwired.
Co-Authored-By: Claude Opus 5 <[email protected]>
VeleroNoRecentBackup has been firing CRITICAL on prod over a backup that was
actually protecting everything.
velero-daily-20260919030041: PartiallyFailed, 3079/3079 items backed up, exactly
2 errors — both PodVolumeBackups for the `plugins` and `scratch` volumes of
homepage-default-kopia-maintain-job-1789786837269-qhsvh. Four such PVB failures
had accumulated.
Velero runs transient kopia repository-maintenance Jobs in its own namespace, one
per BackupRepository, each with emptyDir `plugins` and `scratch` volumes. Schedule
(1) runs defaultVolumesToFsBackup:true — the opt-out model — so it attempts
fs-backup on those volumes too. The Job finishes and its pod disappears
mid-backup, and the PodVolumeBackup fails.
Nothing real was ever missed. But PartiallyFailed does not advance
velero_backup_last_successful_timestamp, so the alert that exists to catch a
genuine backup outage fires every night over Velero cleaning up after itself. An
alert that cries wolf nightly is how a real failure gets ignored — and this one
guards R-STATE-03.
Fix uses the pattern already established for monitoring+unifi: exclude the
namespace's volume DATA from schedule (1), keep its RESOURCES in schedule (2).
Safe by inspection — ns velero holds no PVCs, so there is no volume data to lose,
while the BSL, credentials and Backup CRs that DR actually needs are still
captured.
Deliberately narrower than the general rule. The general fix is a volume policy
skipping emptyDir by type, which would also cover a transient Job in any other
namespace, but it needs Velero's `volumeTypes` policy condition — unproven in this
repo on 1.18.1, and a policies.yaml Velero cannot parse breaks EVERY backup. That
is a much worse failure than the one being fixed, and all four observed failures
were kopia-maintain pods in ns velero. Rationale recorded inline so the next
person can widen it on purpose rather than by accident.
Gated: L0 whole-repo 21 cases, 0 failures, 0 skips, 0 unwired.
Co-Authored-By: Claude Opus 5 <[email protected]>
Records the session's three pieces of work and, more importantly, the theme
joining them: the gate was reporting green over things it had not checked.
- runbooks/node-memory-swap.md, with the three claims I wrote and then removed
because they were false (EtcdIsVoter does not exist here; node-drain-policy is
not pinned so the runbook reads it live; the 5-vs-1 rebuild-limit confusion).
- The nested-worktree scan: six discovery sites auditing 18 copies of the repo.
- L1 rendering never running on a requirement-scoped gate, with the external-dns
incident reproduced via --print-selection (0 tests before, 3 after).
- The two defects that surfaced in the meta-check guarding all of it.
PM HANDOFF lists the open items in priority order. The first is a judgement call
worth flagging rather than burying: the planned memory swap on k3s-prod-1 IS a
supervised single-node loss on prod, which is exactly the event R-STATE-02
describes and which is still proven only on the decommissioned dev cluster.
Capturing evidence during the swap costs nothing; manufacturing the event later
costs a second maintenance window.
Co-Authored-By: Claude Opus 5 <[email protected]>
Declaring whole-repo scope on the L1 tests made whole-repo-invariants-wired
(R-TEST-03) fail. It was right to change behaviour and wrong about why, and it
could not tell anyone either way. Two independent defects.
1. LAYER-AGNOSTIC WHERE THE DESIGN IS LAYER-BOUNDED
The "every scope-declared test is selected despite a requirement filter matching
none" assertion compared EVERY scope-declaring dir against an `--layer l0`
selection. That silently assumed only L0 tests would ever declare the scope —
while ci/l1.sh was written from the opposite premise, that a whole-repo invariant
could appear at any layer. So the first L1 declaration made the check blame the
union for correctly excluding another layer.
Now partitioned by layer, and the L1 half of the property is asserted too:
every scope-declared L1 test must be selected by `--layer l1 --include-scope`,
and that selection must not leak non-L1 tests. That is what makes the L1
declaration load-bearing rather than decorative — previously nothing proved the
union reached L1 at all, because nothing at L1 declared the scope.
2. EVERY ASSERTION FAILED SILENTLY
The idiom throughout was a bare test on one line and `chk $?` on the next, under
`set -euo pipefail`. A false test aborts the script BEFORE chk can print, so the
diagnostic — which is the entire value of this check — was never reached. The
real failure above emitted 26 `ok` lines, rc=1, and no message at all; it read
as a tooling glitch rather than a finding.
All 12 assertion sites now capture the status (`chk $arc`) so the check reports
and continues. Proven by forcing MIN_DECLARED=999 against the real check: the
failure prints its full explanation and the run proceeds to the remaining
assertions, where before it died mute.
A meta-check that cannot say why it failed is worse than no meta-check.
Verified: 35 assertions, rc=0; full offline l0,l1,meta 25 tests / 0 failures.
Co-Authored-By: Claude Opus 5 <[email protected]>
All three L1 tests now declare `conformance.home.k3s/scope: "whole-repo"`.
ci/l1.sh has passed --include-scope whole-repo since it was written, with a
comment saying no L1 test declared the scope yet and it was wired ahead of time
so the next whole-repo invariant "at any layer" would be picked up by annotation
alone. Nothing ever declared it. The consequence is that the entire L1 rendering
layer only ran when a touched path happened to attribute R-PLAT-02, R-NET-01 or
R-NET-05/06/07 — a fail-open hole for every other change in the repo.
The coupling requirement-scoped selection cannot see is that all three tests
exercise ONE code path: catalog/render.py against
schemas/servicedefinition.schema.json. Anything that changes rendered output
invalidates these goldens no matter which requirement the touched path maps to.
The 2026-09-04 external-dns merge is the proof, measured rather than asserted.
ci/touched-requirements.sh attributes platform/externaldns/helmrelease.yaml to
R-ENV-02, R-OBS-01/02/04, R-STATE-03 and R-TEST-01/02/03/04 — none of the
rendering requirements — yet that change added
external-dns.alpha.kubernetes.io/controller: none
to the rendered output of all three goldens. Via --print-selection:
requirement-scoped only (the 2026-09-04 behaviour): 0 L1 tests selected
with --include-scope whole-repo (how ci/l1.sh runs): 3 L1 tests selected
So the merge gated GREEN while leaving every rendering golden stale, and the
staleness surfaced only on a later full run.
Also corrects the now-false comment in ci/l1.sh that said no L1 test declares the
scope, and records why the annotation is load-bearing there.
Verified: offline l0,l1,meta --include-scope whole-repo in the pinned toolbox —
25 tests, 0 failures, 0 errors, exit 0.
Co-Authored-By: Claude Opus 5 <[email protected]>
The operator has a matched 32 GB pair to install. k3s-prod-1 is the node that
needs it: DIMM1 is a Micron 8ATF2G64HZ-3G2E1 (16 GB, DDR4-3200, 1Rx8) and DIMM3
a 16ATF2G64HZ-2G6E1 (16 GB, DDR4-2667, 2Rx8), so capacity is correct at 32 GB
but the pair runs at the slower 2667. Nothing is faulty. k3s-prod-3 is a matched
Crucial pair.
The procedure exists because this node is the awkward one to take down: it held
24 Longhorn replicas, it is one of three etcd members (so the drain leaves
quorum at 2 of 3 with no further margin), and its rebuilds are the load that
starved etcd once already. So the runbook front-loads the checks that decide
whether to start at all, and treats a BLOCKED drain as the safety property
working rather than an obstacle to force past.
Two things it deliberately does not assert:
- node-drain-policy is not pinned in platform/longhorn/helmrelease.yaml, so the
runbook has the operator READ the live Setting and set it before draining if
it is not block-if-contains-last-replica, instead of assuming the default.
- Configured Memory Speed, not Speed, is named as the field that proves the
swap worked — a matched pair being clocked down still reports the rated Speed.
It also flags the capacity decision as a decision: 2x16 GB keeps the three nodes
symmetric as PROD-PROVISIONING.md §2 assumes, while 2x32 GB makes node 1 the
asymmetric one on memory as well as disk.
Recorded caveats: the node-health artifacts de-duplicate DIMM part numbers, so
they can prove "two different modules" but cannot distinguish one module from
two identical ones — verification therefore collects dmidecode live. And node 2's
inventory is stale and self-contradictory (documented as the only single-DIMM
node at 16 GB, yet reporting ~31.1 GiB), so it needs re-inventory before anyone
claims three-way symmetry. This is NOT the §0b "do not buy RAM" case, which was
an EFI memattr kernel bug on node 2.
Wired into the configMapGenerator so it is served at /runbooks/node-memory-swap
rather than existing only in git.
Co-Authored-By: Claude Opus 5 <[email protected]>
Six discovery sites walked ROOT without pruning dot-directories, so they
descended into .claude/worktrees/agent-*/ — up to 18 full copies of the repo,
created for isolated agent runs and gitignored (.gitignore:43).
The visible half was false FAILs: four ServiceMonitors reported as selecting no
Service, and nine PrometheusRule manifests reported as "under test but not in
the repo", all of them worktree paths. The whole-repo L0 gate was red for
reasons that had nothing to do with authored surface.
The dangerous half is quieter. A check that asks "does anything in the repo
declare X" can be satisfied by a worktree copy instead of real source, so the
same bug that invents failures can also manufacture a pass. That is the reason
to fix all six sites rather than only the ones that happened to be failing.
egress-paths-selected-by-rules already had this fix and documents it (it once
saw 5 CiliumEgressGatewayPolicy objects instead of the 1 authored); this applies
the same dot-directory predicate everywhere else, which skips dot-DIRECTORIES
while still matching dot-FILES such as .sops.yaml.
platform/monitoring/tests/run.sh is the other side of the same failure: its
`--list` reported each manifest once per worktree, and the L0 check compares
that listing against the tree. Both find calls now prune .claude alongside .git.
`--list` returns 21 real pairs where it previously returned them ~19x over.
Verified in the pinned toolbox: L0 --include-scope whole-repo is 21 cases,
0 failures, 0 skips, 0 unwired (was 4 failures, then 1). L1 3/3.
The 18 stale worktrees are all on already-merged branches and remain on disk;
cleaning them up is a separate change and is no longer load-bearing for the gate.
Co-Authored-By: Claude Opus 5 <[email protected]>
A kubectl jsonpath with {range .data}{@}{end} dumped the unifi Secret's data
map, putting the base64 console-password in the session transcript. Exactly the
constraint I put in every agent brief, broken by my own careless verification
command.
Exposure bounded: 0 commits contain it (git log -S), the capturing scratch file
was shredded, and a repo-wide grep finds nothing on disk. Critically it is not
yet a live credential — the Secret was created two days after the running VMI,
whose volume list is rootdisk and cloudinit only, so the UOSDATA disk was never
attached, install-uos never ran, and the ubuntu account is still locked.
Action recorded with the ordering that matters: rotate console-password BEFORE
recreating the VMI, so the leaked value never becomes valid.
Lesson: checking a Secret's existence never needs its values. Listing .data's
keys or .metadata.name answers the question; dumping .data answers one nobody
asked.
Co-Authored-By: Claude Opus 5 <[email protected]>
Operator set console-password in the SOPS secret (installer-url deliberately
left as REPLACE-ME, which install-uos handles by printing 'not set yet' and
exiting 0; the uosserver guard exits before it is even read).
Root cause of the lost address, from evidence rather than inference. KubeVirt
generates a fresh MAC per VMI when none is pinned. cloud-init writes
/etc/netplan/50-cloud-init.yaml matching the MAC it saw, and will not rewrite
it while the NoCloud instance-id is unchanged — and that instance-id is
domain.firmware.uuid, stable across restarts by design. So every VMI
recreation handed the guest a MAC its retained netplan did not match, leaving
enp1s0 administratively DOWN, never even attempting DHCP.
What the 2026-09-12 boot log shows, and why it fits: cloud-init's own device
table reads 'enp1s0 | Up: False | Address: .'; there are no apt failures
because the per-instance modules were skipped; yet UOS is installed (6.3 GB of
40 GB) and qemu-guest-agent answers — both of which required working network on
an EARLIER VMI. The MAC is identical between the current VMI status and that
boot log, so it did not change on this boot; the mismatch is against a netplan
written under a previous incarnation.
Pinned to the MAC the current VMI already holds, so nothing changes for this
guest. Combined with the instance-id bump already merged, the next restart
re-runs cloud-init, rewrites netplan against a MAC that is now permanent, and
the interface should come up — and the console password is in place as the
fallback if it does not.
Still unverified from outside: the netplan file's actual contents. That needs
the console the operator can now reach.
Co-Authored-By: Claude Opus 5 <[email protected]>
b184a42 made the renderer stamp external-dns.alpha.kubernetes.io/controller:
none on internal services' HTTPRoutes, but three L1 golden fixtures were never
regenerated, so container-workload-render (R-PLAT-02) and egress-class-render
(R-NET-05/06/07) failed. Exactly one line differs in each — the annotation and
nothing else.
WORTH RECORDING: the merge that caused this GATED GREEN. The requirement-scoped
selection did not include R-PLAT-02 or R-NET-05/06/07, so neither check ran. A
later docs-only commit shifted the touched set, selected them, and exposed a
failure the earlier merge had introduced — meaning the gate reported PASS on a
tree it had not actually checked in the relevant place.
That is the same fail-open the whole-repo scope mechanism was built for, except
that mechanism covers L0 only; L1 is still purely requirement-scoped. A renderer
change can therefore drift any L1 golden whose requirements the change does not
happen to attribute. Both affected checks are pure offline renders with no
cluster dependency, so they are cheap to run unconditionally — declaring
scope: whole-repo on the l1-rendering tests looks like the right follow-up.
Also note the checks diff fail-fast: only egress-vpn was reported, while
egress-internet had drifted identically and was never reached.
Co-Authored-By: Claude Opus 5 <[email protected]>
Confirmed at the authority, not a cache: all six internal names absent
publicly, all five public names untouched, and Pi-hole answering for the six
internally. The R-DNS-03 disclosure is closed for that tier.
Records a correction to my own sequencing: I insisted this could not land
before the DHCP repoint, but the router returns nothing for those six names, so
the records served nobody and the dependency I asserted did not exist. One
query would have shown it.
Also records the operator's clarifications — Pi-hole is DNS only with the
router still doing DHCP, reverse proxy by hostname rather than a second IP, and
Minecraft staying open — plus the unencrypted-secret fix.
Co-Authored-By: Claude Opus 5 <[email protected]>
Every other *.sops.yaml in this repo carries sops metadata and ENC[] values;
this one shipped with neither and both fields as literal REPLACE-ME. Two
consequences: 'sops <file>' fails with 'sops metadata not found', so the
documented operator step could not work; and Flux would pass the file through
undecrypted and create the Secret with the literal placeholder strings.
Encrypted in place with the repo's age recipient, placeholders intact.
Round-trip verified: sops -d returns installer-url and console-password as
REPLACE-ME, so the operator's workflow is now the normal one — sops the file,
set both values, commit.
Both fields need setting, not just the password: installer-url is also
REPLACE-ME, which I had omitted from the operator instructions.
Co-Authored-By: Claude Opus 5 <[email protected]>
Six exposure:internal services published A records into the public Cloudflare
zone answering 192.168.2.20, disclosing the internal subnet, the Gateway VIP
and a full service inventory, and forming a DNS-rebind primitive that is why
the SOHO router strips these answers in the first place.
The renderer now stamps external-dns.alpha.kubernetes.io/controller: none on
every HTTPRoute it renders for an internal service, derived from the exposure
field alone — so a service that omits exposure defaults to internal and is
likewise unpublishable. Established against the deployed v0.21.0's own source,
where the controller check runs before hostname resolution in both configured
sources, then proven live on a throwaway hostname including that adding the
annotation to an existing record triggers the DELETE.
Deletes exactly 12 records: the A and TXT pair for jackett, pihole,
qbittorrent, radarr, semaphore and sonarr. A 39-record recovery baseline is
committed alongside.
Safe to land before the DHCP change, contrary to my earlier caution: the
router returns NOTHING for all six names today, so these records serve nobody
on the LAN, while Pi-hole at 192.168.2.23 already answers for anyone pointed
at it.
The five exposure:public services are untouched. They remain unreachable
publicly, which is a separate problem: the legacy edge at 136.51.115.252
returns HTTP 403 for our hostnames and serves a placeholder certificate, so
nothing is reverse-proxying them today.
Third confident claim of mine corrected in two days, and this one I took from
a stale README rather than the machine. uosserver.service starts on the
current boot and the root filesystem holds 6.3 of 40 GB, while enp1s0 is up
with no address and does not answer ARP — so the controller runs and nothing
can reach it.
Records the near-miss: proceeding on my premise would have reinstalled over a
working controller, which the new install-uos guard now prevents. Also records
that there is no way into the guest at all, which is why the deliverable became
console access, and the cloud-init instance-id discovery that explains why an
earlier unattended install could never have taken effect.
Co-Authored-By: Claude Opus 5 <[email protected]>
The premise I briefed was wrong and the agent checked before building on it.
UniFi OS Server IS installed: the current boot starts uosserver.service and
uosserver-updater.service, and the root filesystem holds 6.3 GB of 40 GB where
bare Ubuntu plus our packages is about 2 GB. Verified independently.
The actual fault is that the guest has no address — .status.interfaces shows
enp1s0 link up with no ipAddress, and ip neigh inside virt-launcher reports
10.0.2.2 FAILED, so the guest does not answer ARP on its masquerade address.
Probes from virt-launcher straight to the pod IP and to 10.0.2.2 RST on
11443/8443/8080/22, which exonerates the VIP, the traffic policy and the
network policy. The cause cannot be determined from outside the guest.
That made one guard non-negotiable: install-uos now bails the moment systemd
knows a uosserver unit. Without it this merge would have reinstalled over a
working controller.
Second finding: there is no way into the guest at all — no SSH key, no
password, and SSH cannot work by construction because masquerade forwards only
declared ports and 22 is not one. So the deliverable is console access: a SOPS
secret carrying installer-url AND console-password, mounted as a NoCloud disk
and re-read every boot.
The discovery that makes it work: NoCloud instance-id is
spec.template.spec.domain.firmware.uuid, stamped on the VM object. Cloud-init
compares it to the DMI product UUID, so editing userData alone changes nothing
on a second boot — the cached datasource is restored and write_files/runcmd are
skipped. That is why a previously shipped unattended install could never take
effect. The UUID is now pinned in git and bumped once.
Deliberately did NOT put the whole cloud-config behind secretRef: that would
hide the shell in ciphertext, out of reach of the L0 dollar-collapse guard,
while Flux's envsubst still runs over the decrypted resource — the nextcloud-db
shape from 2026-09-13. Script in plaintext, operator values in SOPS.
Unattended install is reported honestly as third-party practice, not verified
here, and the README's stale dev-era addressing is corrected throughout since
it doubles as the rebuild runbook.
The operator's UOS bring-up becomes one field. apps/vms/unifi/unifi-uos-installer.sops.yaml
(SOPS/age, with a .example sibling per the crafty/headlamp convention) carries `installer-url`
and `console-password`. KubeVirt mounts it as a read-only iso9660 disk labelled UOSDATA and an
in-guest `uos-install.service` reads it on EVERY boot, so a later URL change needs only a commit
and a VM restart.
Mechanism verified against the DEPLOYED KubeVirt v1.9.0, not from memory:
* inline userData cap is exactly 2048 bytes — server-side dry-run apply accepted 2048 and
rejected 2049; the payload here is 1454.
* `cloudInitNoCloud.secretRef` and `secret.volumeLabel` both exist in the live CRD; the secret
disk + third disk were accepted by a server-side dry-run of the real manifest.
* the NoCloud seed's instance-id IS spec.template.spec.domain.firmware.uuid (read out of
noCloud.iso inside virt-launcher), stamped on the VM OBJECT, so it survives restarts. Editing
userData alone therefore changes NOTHING — cloud-init restores its cached datasource and skips
write_files/runcmd. The UUID is now pinned in git and bumped once so the new payload lands.
secretRef was deliberately NOT used for the whole cloud-config: it would put the shell in
ciphertext, out of reach of the L0 dollar-collapse guard, while Flux's envsubst still runs over
decrypted resources — the 2026-09-13 nextcloud-db shape. Script in plaintext, operator values in
SOPS. apps-vms had no `decryption` stanza (first encrypted content under apps/vms/), so one was
added to both overlays.
CORRECTION, and it is the headline: UNIFI OS SERVER IS ALREADY INSTALLED on the live guest. The
serial console log starts uosserver.service and uosserver-updater.service, and / is 6.3 GB of
40 GB used. The controller is unreachable because THE GUEST HAS NO IP: .status.interfaces shows
enp1s0 up with no address, and `ip neigh` in virt-launcher reports 10.0.2.2 FAILED. 11443/8443/
8080 and 22 all RST, from the pod IP and from 10.0.2.2 directly, so the VIP and the CNP are
exonerated. install-uos therefore bails out the moment systemd knows a uosserver.service —
without that guard this change would have reinstalled over a working controller.
There was also no way into the guest at all: no key, no password, and 22 is not a declared
masquerade port so SSH cannot work by construction. Hence console-password.
README corrections (it doubles as the rebuild runbook and was misdirecting):
* VIP 192.168.1.54 -> 192.168.2.21; pool 192.168.1.53-.60/.53 -> 192.168.2.20-.30 with the
Gateway on .20; device subnet 192.168.1.0/24 -> 192.168.2.0/24; node k3s-dev-1 -> k3s-prod-1.
All re-read live (pool blocks, cluster-vars, the Service EXTERNAL-IPs, LAN_CIDR).
* "UOS is NOT installed" -> it is; the real fault is the lost IP, with the evidence table.
* "apps-vms must NOT use postBuild.substituteFrom" -> it has since 2026-08-31 and the escape
contract is in force.
* "install-uos is no longer baked into the guest" -> it was baked in, and it was broken twice
over (an unverified firmware API, and no `install` subcommand).
* "SSH in, cloud-init installs the operator's key" -> it installs none, and 22 is not forwarded.
* externalTrafficPolicy Local -> Cluster (the 8-day outage fix already in 50-service-l4.yaml).
* 60 GB -> 40Gi (the DataVolume and the Bound PVC); "10 GB minimum" -> ~25 GB.
* userDataSecretRef -> the YAML key is `secretRef`.
* 90-httproute-optional.yaml referenced as if it existed; it was dropped.
Unattended install, honestly: Ubiquiti documents no silent flag, no answer file, no env vars.
`yes y | <installer> install` is the convergent community practice and is what ships here, with a
bare-invocation fallback. It has NOT been run end-to-end on our guest. The installer URL also
does not look signed or time-limited — it is account-derived and per-release; it is in SOPS
anyway, but nothing should depend on expiry.
The running VM was not touched: no restart, no migration, no re-import. Every cluster call was a
read or an explicit --dry-run=server; the live VM still carries firmware uuid ...c07 and the
34h-old virt-launcher pod is untouched. The change takes effect on the operator's next VM restart.
Co-Authored-By: Claude Opus 5 <[email protected]>
Records the EULA acceptance prominently with revocation instructions, the
Server List Ping proof, and Crafty's autostart trap where the create API
accepts 'autostart' but stores 'auto_start'.
Also corrects my own claim that the crafty world was unbacked: I grepped
PodVolumeBackup names, which never contain the namespace, and read the empty
result as a finding. Crafty had 2 PVBs in the last run. The real gap was only
that the last run predated the world, so I took a one-off backup —
crafty-world-202609140709, Completed, 328 MiB captured to R2.
Co-Authored-By: Claude Opus 5 <[email protected]>
Crafty was healthy but had never had a server created — /crafty/servers was
empty, which is why the port refused. Paper 26.2 (Mojang's latest.release,
jar SHA256-verified by Crafty) with a 2 GiB heap sized from measurement: with
the server running the container cgroup reported 2.86Gi against a 4Gi limit,
so a 3 GiB heap would have risked OOMKill.
EULA ACCEPTED ON THE OPERATOR'S BEHALF — https://aka.ms/MinecraftEULA — via
Crafty's agree_to_eula, writing eula=true. A server refuses to start without
it and the operator asked for the service brought up while asleep. No account,
key or payment involved. To revoke: stop the server and set eula=false, or
delete the server.
Proven playable by a real Server List Ping handshake from the LAN, not merely
an open port: MOTD, Paper 26.2, protocol 776, 0/20 players, 12.3 ms RTT.
Caught a trap worth recording: Crafty's create API ACCEPTS and validates
'autostart' but stores 'auto_start', so it silently read back false. Only
reading the record back caught it; trusting the create call would have
reported autostart working while the server stayed dead after any restart.
Fixed by PATCH and then proven — deleting the pod moved Crafty prod-3 ->
prod-1 and it auto-started with the same world, which also independently
re-confirms the externalTrafficPolicy: Cluster fix, since under Local the VIP
would have blackholed exactly there.
Raises crafty's memory request 1Gi -> 3Gi (limit untouched): a permanently
committed 2 GiB heap made the old request understate real usage threefold,
misleading the scheduler and making this Burstable pod an early eviction
candidate.
ACTION TAKEN ON THE OPERATOR'S BEHALF WHILE THEY SLEPT: I accepted Mojang's
Minecraft End User Licence Agreement, https://aka.ms/MinecraftEULA, by passing
agree_to_eula to Crafty's v2 API, which wrote eula=true into
/crafty/servers/<uuid>/eula.txt. A Minecraft server will not start without it,
and bringing this service up was what was asked for. Please review. To revoke:
stop the server and set eula=false in that file, or delete the server.
Crafty was healthy but had never had a server created, so 192.168.2.22:25565
refused connections. Created "homelab" via the API (not UI automation):
Paper 26.2, TCP 25565, heap -Xms2000M -Xmx2000M, autostart on
uuid 13c7b85d-00bb-4a58-8c86-2433ca47891c
Paper over Vanilla for tick performance and plugins, with no cost to players
(vanilla clients connect unchanged). 26.2 is Mojang's latest.release, out
2026-06-16; the 26.3-rc-2 snapshot and 26.2-rc-2 were deliberately avoided.
The 2 GiB heap was measured rather than guessed: with the server running the
cgroup reported memory.current 2.86Gi against the 4Gi limit, so a 3 GiB heap
would have risked an OOMKill.
Proven playable, all real output: Crafty reports running with 'Done (12.737s)!';
Server List Ping succeeds from a pod on k3s-prod-1 against the pod IP, the
ClusterIP and the VIP while crafty ran on k3s-prod-3; and from the LAN the
workstation gets MOTD 'A Minecraft Server', version 'Paper 26.2', protocol 776,
players 0/20 over a full handshake plus ping/pong. Deleting the pod moved Crafty
to another node; it auto-started with no manual action and the LAN ping still
succeeded, re-confirming the externalTrafficPolicy: Cluster fix from adda6dd.
Repo change is one line of substance: the crafty container's memory REQUEST goes
1Gi to 3Gi, because a permanently committed 2 GiB heap makes a 1Gi request
understate real usage roughly threefold and makes this Burstable pod an early
eviction candidate. Limit stays 4Gi. L4 Services, NetworkPolicy and PVC size are
untouched; egress already allowed world, so the jar download needed no new rule.
Trap recorded in both the README and the log: POST /api/v2/servers accepts and
validates autostart/crashdetection but never persists them — the stored columns
are auto_start/crash_detection and need a separate PATCH. Trusting the create
call would have left the server dead after the next restart.
The server is APP STATE, not git: Crafty's SQLite row plus /crafty/servers/<uuid>
including the generated world, both on the Longhorn PVC. A rebuild from git alone
gives an empty Crafty and a refused 25565. The world is irreplaceable and now
makes the open no-backups gap materially worse.
Co-Authored-By: Claude Opus 5 <[email protected]>
I reported the UniFi controller and Minecraft server as unreachable for up to 8
days. Probing the backend pods directly, bypassing VIP and policy, returns
REFUSED on all four ports while crafty's own UI and pihole's DNS answer as
controls. /crafty/servers is empty and the unifi README still lists installing
UOS in the guest as the open blocker. Nothing regressed.
The externalTrafficPolicy defect is real and is fixed, but it was not the cause.
I observed unreachable ports, found a mechanism that produces unreachable ports,
and concluded it was the cause without running the one control probe that
separates them — the same error shape as nearly blaming the drive for the I/O
pressure spike when the local disks were idle.
Co-Authored-By: Claude Opus 5 <[email protected]>
Cilium's L2 announcer elects a lease holder without regard to which nodes hold
a backend, so externalTrafficPolicy: Local leaves the frontend absent from the
announcing node's service table entirely and the VIP answers ICMP unreachable.
Proven causally with cilium-dbg service list: under Local the frontend is
missing from prod-2; flipped to Cluster it appears as an active backend entry;
flipped back it vanishes. Durability proven too — cordon prod-1, crafty
reschedules to prod-3, and prod-1's UDP lease re-points to the new backend with
no intervention, which under Local is exactly the failure mode.
IMPORTANT CORRECTION to my own framing: this is NOT why unifi and crafty do not
answer, and fixing it does not make them answer. There is no application behind
any of those ports. /crafty/servers is empty — no Minecraft server has ever been
created — and apps/vms/unifi/README.md step 5 still reads 'NEXT: install UOS in
the guest', with the unattended install flagged as the open blocker. Probing the
pods directly, bypassing VIP and policy, returns REFUSED on all four ports while
crafty's own UI and pihole's DNS answer as controls. Nothing regressed; the
8-day-old leases are just 8-day-old Services. I called it a live outage and it
was not.
What this commit does is remove a landmine that would have blackholed both VIPs
the moment those applications exist.
Client-IP cost assessed per service rather than assumed: zero for UniFi, whose
KubeVirt masquerade binding already NATs everything into the guest, so Local was
protecting an address the controller could never observe; real but acceptable for
Crafty, where player identity is a Mojang UUID so bans and whitelists are
unaffected, with /ban-ip documented as do-not-use. loadBalancerSourceRanges
deliberately NOT copied from the Pi-hole pattern: those Services never had a LAN
restriction, and adding one during a fix could cut off off-LAN players.
Cilium DSR mode is recommended separately, not bundled — it carries an Ansible
parity contract and rolls the CNI across all three nodes.
Cilium's L2 announcer elects one node per Service to answer ARP for its
LoadBalancer VIP, and that election is NOT backend-aware. Combined with
externalTrafficPolicy: Local -- which lets a node deliver only to a backend on
itself -- an elected announcer holding no backend answers ARP for the address
and then has nowhere to send the packet, so the client gets ICMP port
unreachable. It never self-heals: the lease holder keeps renewing after the
backend moves away.
Measured on prod 2026-09-14: unifi's VM ran on k3s-prod-1 with both leases on
k3s-prod-2, and crafty's pod ran on k3s-prod-1 with the -tcp lease on
k3s-prod-2 while its identical -udp twin happened to draw k3s-prod-1 and
worked. Confirmed causally on the announcing node with `cilium-dbg service
list`: under Local the frontend 192.168.2.22:25565 is absent from that node's
service table entirely; under Cluster it appears as `1 => 10.0.0.11:25565
(active)`; flipped back, it vanishes again. Durability confirmed by cordoning
k3s-prod-1 and moving the crafty pod to k3s-prod-3, after which the announcing
node re-pointed to the new backend with no intervention.
Both files also carried a comment claiming Local "pins the VIP to the node
running the pod", which is the precise thing that is false here; both are
rewritten. apps/wave-0/pihole-dns.yaml is the reference implementation.
The client-IP cost is stated per service rather than waved away. For unifi it
is ZERO: the VM's KubeVirt `masquerade` binding already SNATs everything into
the guest, so Local was never preserving a client address the controller could
see. For crafty it is real and accepted -- player identity is a Mojang/Xbox
UUID so /ban and the whitelist are unaffected, but /ban-ip would hit every
player at once and must not be used; no IP ban or per-IP rate limit is
configured today.
No loadBalancerSourceRanges here, deliberately departing from pihole-dns.yaml:
these Services never had a source restriction (their CNPs admit `world`), so
Cluster removes no protection, and adding a LAN-only filter during an outage
fix could cut off off-LAN players or a remote site's device informing.
No CiliumNetworkPolicy change is needed, and that was verified rather than
assumed: Cilium's `cluster` entity subsumes `remote-node`, confirmed by
dumping the crafty endpoint's datapath policy map with the CNP unmodified
(`reserved:remote-node` was already an Allow on the game ports). Unlike
pihole, whose policy omits `cluster` and must name remote-node explicitly.
Adds an L0 whole-repo guard for the class. It fails on the pre-fix tree with
exactly these four Services named and passes after. whole-repo scope is
necessary: ci/touched-requirements.sh has no arm for apps/* or
platform/crafty/*, so the change most likely to reintroduce this would
attribute no requirement and be gated by nothing.
HONEST LIMIT -- this does NOT end the reported outage. All four ports also
have no listening backend: UniFi OS was never installed in the VM (its README
flags this as the open blocker) and /crafty/servers is empty, so no Minecraft
server exists. Probed directly from the node, bypassing the VIP entirely,
10.0.0.74:8443/:8080/:11443 and 10.0.0.11:25565 all refuse, while the two
controls answer. This commit fixes a real latent defect that would have
blackholed those VIPs once the applications exist; it cannot make an absent
application answer.
Co-Authored-By: Claude Opus 5 <[email protected]>
The spec and README both claim a LAN split-horizon via Pi-hole; it never
existed. Pi-hole's Service was ClusterIP with ports 80 and 4180 only, so LAN
clients depended on hand-maintained per-client dnsmasq files — one of which
went stale after the dev decommission and cost access to sonarr.
Now DNS on udp/53 + tcp/53 on a pinned pool VIP 192.168.2.23, with the zone
answered as a wildcard to the Gateway VIP plus longest-match overrides for the
two hostname-publishing L4 Services. Declared via Pi-hole v6 FTLCONF env vars,
which FTL treats as env-forced and applies at cold start — a real declarative
entry point, unlike the Keycloak secret and CNPG grant whose reconcilers exist
precisely because no such knob is available.
Two defects found on the way. The live pod carried hand-set upstreams marked
'### CHANGED' that no manifest knew about, so any restart would have left
Pi-hole with upstreams = [] and no recursion at all; FTLCONF_dns_upstreams now
declares them. And /etc/pihole is not on the config PVC, which mounts at
/data/config, so that volume has been holding nothing.
Deviates from the unifi/crafty pattern deliberately: externalTrafficPolicy
Cluster, not Local. Cilium's L2 announcer does not filter by which nodes hold a
backend, so with Local an announcing node without the pod answers ICMP port
unreachable — measured 1-in-3 per protocol, and it does NOT self-heal: a
cordoned node kept renewing its lease for over three minutes with LAN DNS dead.
The LAN restriction moves to loadBalancerSourceRanges, enforced in eBPF on the
VIP, and rate limiting is disabled because SNAT makes the whole LAN look like
one client.
Proven from the workstation in the worst case — pod and both protocol leases on
three different nodes: wildcard, both overrides, recursion over udp and tcp,
NXDOMAIN outside the zone, NODATA for AAAA, source-range drop when excluded,
and survival of a cross-node reschedule with no intervention.
platform/coredns/coredns-custom.yaml has always said its split-horizon is for
PODS and deferred LAN clients to "PiHole split-horizon". PiHole served no DNS:
its Service was ClusterIP with ports 80 and 4180 only. The LAN half of R-DNS-03
that docs/09 and the README's Phase 2 row both claim did not exist, so LAN
clients ran on hand-maintained per-client dnsmasq files - one went stale after
the dev decommission and cost the operator sonarr.home.keithjackson.dev. The
SOHO router that owns 192.168.2.0/24 cannot cover for it either: it strips
RFC1918 answers for public domains as rebind protection and returns NOERROR
with no answer.
EXPOSURE (apps/wave-0/pihole-dns.yaml, hand-written, placed into wave 0 by
render.py's EXTRA_UNITS - the schema has no field for a LoadBalancer Service, a
pinned lbipam VIP, a sharing key, source ranges or UDP, the same reason
apps/vms/unifi and platform/crafty hand-write theirs):
- two Services, udp/53 and tcp/53, sharing one pinned VIP
${PIHOLE_DNS_VIP} = 192.168.2.23 on prod, the next free address in the
.20-.30 pool (pool status IPsUsed 3 of 11; .20 Gateway, .21 unifi, .22
crafty; arping from a LAN host got no reply).
- no external-dns hostname annotation: nothing public is created, and
platform/externaldns/ is untouched.
ANSWERS (catalog/pihole.yaml env - declarative, no reconciler): PiHole v6 takes
every FTL setting as an FTLCONF_ environment variable and treats it as
env-forced, applied ahead of pihole.toml on every cold start and read-only in
the UI. That is a real declarative entry point, which the Keycloak and CNPG
CronJob reconcilers exist precisely because their state lacks, so a reconciler
here would be ceremony. misc.dnsmasq_lines carries a wildcard
address=/<zone>/<GATEWAY_VIP> plus one longer-domain override per L4 service
(unifi, crafty-mc, pihole-dns); dnsmasq resolves overlaps by longest matching
domain, so precedence is structural and a newly onboarded Gateway-fronted
service needs no DNS change. dns.upstreams is declared too - the live pod
carried hand-set upstreams that no manifest knew about and that every restart
discarded.
externalTrafficPolicy Cluster, NOT Local, and loadBalancerSourceRanges for the
LAN restriction. Measured on prod, Cilium 1.20.1: the L2 announcer does not
filter lease candidates by which nodes hold a backend - the two Services landed
on k3s-prod-3 and k3s-prod-2 while the pod ran on k3s-prod-1 - and it keeps
renewing after the pod moves away, so with Local a reschedule leaves LAN DNS
dead indefinitely. Cluster serves the VIP from any node; the LAN restriction
moves to loadBalancerSourceRanges, which Cilium enforces in eBPF on the VIP
before load-balancing (verified by narrowing the range to exclude the test
client and back). The cost is SNAT: forwarded queries reach PiHole from the
ingress node, so PiHole's per-client rate limit is disabled in the catalog or
the whole LAN would count as one client against its 1000-per-60s default.
Proven from a LAN host (192.168.2.132) against 192.168.2.23 with the pod and
both leases on three DIFFERENT nodes: sonarr/grafana/nextcloud/keycloak/apex
-> 192.168.2.20, unifi -> .21, crafty-mc -> .22, pihole-dns -> .23, recursion
(cloudflare.com, github.com, debian.org) intact over both udp and tcp, AAAA
NODATA, NXDOMAIN outside the zone, and still answering after a forced
prod-1 -> prod-2 reschedule.
Co-Authored-By: Claude Opus 5 <[email protected]>
docs/09 R-DNS-03 and schemas/renderer-contract.md both said an `internal`
service renders no public DNS record. Nothing enforced either sentence.
ExternalDNS's `gateway-httproute` source admitted EVERY HTTPRoute on the shared
Gateway, so six `exposure: internal` services — jackett, pihole, qbittorrent,
radarr, semaphore, sonarr — published `<name>.home.keithjackson.dev` into the
public authoritative Cloudflare zone answering with the Gateway VIP
192.168.2.20. That published the internal subnet, the Gateway VIP and a full
service inventory to anyone who could guess a label, and made each of those
names a DNS-rebind primitive — which is why the SOHO router strips RFC1918
answers and LAN clients need per-client dnsmasq overrides to reach their own
services.
THE MECHANISM, established against the DEPLOYED version rather than from
memory. The live prod Deployment runs
registry.k8s.io/external-dns/external-dns:v0.21.0. In v0.21.0,
source/annotations/processors.go IsControllerMismatch() skips a source object
whose `external-dns.alpha.kubernetes.io/controller` annotation is present and
!= its own `dns-controller`, and source/gateway.go applies that check inside
Endpoints() BEFORE any hostname is resolved; source/service.go applies the
identical check, covering the second configured source. catalog/render.py now
emits that annotation on every HTTPRoute it renders for an `internal` service,
derived from `exposure` alone — no per-service list — so a service that omits
`exposure` inherits the internal default and is unpublishable too. Verified
LIVE on prod on a throwaway hostname with no existing record at risk: the
annotated route published nothing, and adding the annotation to a route that
already had a record produced action=DELETE for both its A record and its `a-`
TXT registry entry one ~60s reconcile later, moving nothing else in the zone.
WHY PER-RESOURCE AND NOT A CLUSTER-WIDE OPT-IN FILTER. `labelFilter` is the
stronger default posture and v0.21.0 supports it for both configured sources,
but nine publishing objects here are hand-authored platform/apps manifests with
no `exposure` field, seven of which hold live records (auth, crafty, crafty-mc,
grafana, headlamp, the apex, unifi). An opt-in filter deletes all seven at
once, and whether each should be publicly resolvable is an operator decision
per name. Per-resource exclusion has exactly the blast radius of the defect;
the seven are enumerated in a shrink-only register so the unadjudicated set
cannot grow silently, and the filter stays available as hardening afterwards.
`policy: sync`, `txtOwnerId` and `domainFilters` are UNTOUCHED. Per-resource
exclusion only ever removes an object from this instance's view, so nothing
here makes prod capable of deleting a record it does not own.
WHAT THIS DELETES, and it is not reversible by a revert alone: 12 records — the
A record and the `a-`-prefixed TXT registry entry for each of jackett, pihole,
qbittorrent, radarr, semaphore and sonarr. The zone has no
`*.home.keithjackson.dev` wildcard and `*.keithjackson.dev` does not match two
labels, so each becomes a public NXDOMAIN with no fallback. The complete
pre-state is captured at the authority and committed as a recovery baseline
(.standup-dns/dns-20260914-pre-externaldns-scope.txt), with the restore
procedure for both halves of a record.
MERGE ORDER IS NOT OPTIONAL: feat/pihole-lan-dns-authority must land and be
verified first. Removing these records while the LAN still depends on them,
with the router stripping RFC1918 and no internal authority yet, breaks name
resolution outright.
The five `exposure: public` services are LEFT UNCHANGED and still answer with
the Gateway VIP, so they remain publicly unreachable. That is two open operator
decisions, both written up in platform/externaldns/helmrelease.yaml: what
address they should answer with (136.51.115.252 is the legacy non-k3s stack
serving a placeholder cert, not established as correct for this cluster), and
how to separate them from the non-public names that share the Gateway —
because in v0.21.0 the target annotation is read only from the GATEWAY and a
ListenerSet, never from an HTTPRoute, so a per-service override is not
expressible while they all attach to `shared`. No knob is added for it yet: one
that cannot take effect invites someone to set it and expect one.
Guard: conformance/tests/l0-static/internal-services-publish-no-public-dns, a
whole-repo-scoped L0 static check (no ci/l0.sh stage — the
whole-repo-invariants-wired meta-test forbids one). It asserts both directions
(present on internal, absent on public), the `service`-source half, that the
renderer DERIVES the annotation from `exposure` and cannot emit external-dns's
own claiming value, proves the shipped renderer end-to-end on
internal/public/omitted-field fixtures, and ratchets the non-catalog register.
Proven to fail on the pre-fix tree (9 of 53 assertions, naming all six
services) and pass after (56/56).
Schema 1.6.0 -> 1.7.0: no field changes, a mapping change to `exposure`.
Co-Authored-By: Claude Opus 5 <[email protected]>
Operator-reported, verified. external-dns runs provider cloudflare with
gateway-httproute + service sources and publishes the Cilium Gateway VIP
192.168.2.20, so ~16 hostnames in a public authoritative zone answer with
RFC1918 space. The spec says the opposite: external resolves to the public IP,
Pi-hole serves the internal split-horizon answers, internal services are
unreachable from the internet.
Provenance stated plainly: the pattern predates me — external-dns always
published the Gateway address, and dev did the same under owner=default. My
apex migration changed which names carry it, propagating an existing flaw onto
the apex rather than inventing it.
Four consequences recorded: topology and service-inventory disclosure; a
DNS-rebind primitive, which is why the SOHO router strips these answers and
LAN clients need per-client overrides; the five services declaring exposure
public are not actually publicly reachable; and seven internal services are
publicly advertised.
Also records why deleting the records is not a safe first step: the
*.home.keithjackson.dev wildcard is gone, so removal yields NXDOMAIN rather
than a fallback. Remediation sequenced, nothing started.
Co-Authored-By: Claude Opus 5 <[email protected]>
Handed an agent a baseline of 4.0-4.8 ms/op response, 0 ms/op queue, 0 major
timeouts over 24h. Re-measurement found queue time is 0.013-0.067 ms/op (I read
rounded output as zero); the 24h claim was really ten minutes, because the
DaemonSet had just rolled and the range query returned 18 samples rather than
~2,880; and the baseline was idle, so 4.4 ms is a cached STATFS rather than real
media I/O and thresholds could not be multiples of it.
The 24h error is the same class as the label-values mistake recorded above:
trusting a query whose window was never valid.
Records two real discoveries: mounts sharing an NFS superblock share kernel RPC
statistics, so prod-1's two plex mounts are one signal reported twice; and
maximum_rpc_slots high-water is 586 on prod-1 against 2 on prod-2, an
independent indication that prod-1 carries far more concurrent NFS work.
Co-Authored-By: Claude Opus 5 <[email protected]>
Closes the gap the previous agent correctly left open: it could not write a
rule against node_mountstats_* because the series did not exist until the
collector rolled out, and the L0 guard rightly rejects a rule on an unscraped
metric. 1,842 series are now live, so five rules land.
The discriminator is the point. response_time is how long the SERVER held the
request; queue_time is how long it waited on our node for an RPC slot. Both
server rules carry 'unless queue_time/op > 5ms', so a queueing client can never
be reported as a slow server, and the two server rules then partition the same
slow-server state by our own op rate at 20/s — exactly one fires, no gap and no
overlap. Plus major timeouts as critical, since that is hanging rather than slow.
Corrects three figures I had supplied from bad measurements: client queue time
is 0.013-0.067 ms/op, not zero; 'zero major timeouts over 24h' was really zero
over about ten minutes, because the DaemonSet had just rolled and the range
query returned 18 samples rather than ~2,880; and the baseline was an IDLE one
whose 4.4 ms is a cached STATFS, so thresholds are argued from the link and the
array rather than as a multiple of it.
Also found that mounts sharing an NFS superblock share the kernel's RPC
statistics — prod-1's plex and plex/downloads report identical rates to every
digit — so one slow export raises two alerts. Documented rather than aggregated
away, since aggregating would hide which path is affected.
14 mutations, all killed. Runbook triage table 10 rows -> 15, with 'not
collected' stated plainly for both past incidents because the collector
postdates them and that cannot be backfilled.
Closes the gap `feat/io-pressure-diagnostic-context` deliberately left open. That
change enabled node-exporter's `--collector.mountstats` and recorded its own honest
limit: "an NFS export that is merely SLOW cannot be distinguished today", because
node_mountstats_* did not exist on prod until the HelmRelease rolled and the L0 guard
alert-metrics-have-scrape-targets rightly refuses a rule on an unscraped metric.
The rollout has landed. Verified on prod 2026-09-15: 1,842 node_mountstats_* series
(~0.4% of a 451,650-series head), six mounts, 426 per-operation series in each of the
five operation families and 6 in each transport family. Every metric referenced below
returns series under `promtool query instant`.
FIVE RULES in platform-health.node-io-cause, each naming its cause in `summary`:
NFSExportServerSlow response_time > 50 ms/op, our op rate > 20/s,
queue_time < 5 ms/op -> the server is slow BECAUSE OF US
NFSExportServerSlowNotOurLoad same, but our op rate 1-20/s -> the server is slow and
the load is NOT OURS. This is the operator's Proxmox
hypothesis made testable.
NFSClientQueueTimeHigh queue_time > 5 ms/op -> WE are saturating the transport
NFSTransportBacklogDeep backlog depth > 8 per send -> even the grown slot table is full
NFSExportMajorTimeouts any major timeout in 15m -> the export is HANGING (critical)
THE DISCRIMINATOR. mountstats splits a slow NFS op into the half the server owns
(response_time) and the half we own (queue_time). High response with low queue means the
SERVER; high queue means US. The two server rules then partition that state by our own op
rate at 20/s, so exactly one of them can fire and it names which reading applies. The
`unless queue_time > 5ms` clause is what licenses reading response_time as a statement
about 192.168.2.102 rather than about us, and it is pinned by fixture 5 (a client queueing
30 ms/op while the server appears to take 60 ms/op: both server rules must stay quiet).
MEASURED BASELINE, prod 2026-09-15, re-measured for this change:
server response time 4.40-4.81 ms/op (30m window), uniform across all six mounts
client queue time 0.013-0.067 ms/op; snapshot maximum 0.333 ms/op
our own op rate 0.019-0.163 ops/s — STATFS keepalive and nothing else
major timeouts 0 on every mount
backlog depth 0 on every mount
slot-table high-water 586 (prod-1), 162 (prod-3), 2 (prod-2)
THREE LIMITS ON THAT BASELINE, recorded in the rules and the runbook rather than papered
over:
(a) the history is MINUTES, not days — the series begin when the DaemonSet rolled
(count_over_time(...[24h]) returned 18, and time() - process_start_time_seconds read
558-561 s on all three nodes), so no rule uses a range longer than 15m;
(b) it is an IDLE baseline — STATFS keepalive only, no READ/WRITE traffic, so the 50 ms
threshold is argued from the link and the array (one outstanding 1 MB read at 50 ms is
20 MB/s) and must be re-cut after the first real media event. The runbook carries the
query;
(c) prod-1's /mnt/Storage/plex and /mnt/Storage/plex/downloads report byte-identical
per-operation counters (0.12443893333333335 on both, to every digit), as do prod-3's two
nextcloud mounts — mounts sharing one NFS superblock share the kernel's RPC statistics.
One slow export on prod-1 therefore raises two alerts. Documented, not aggregated away.
PROOF, not assertion. platform/monitoring/tests/nfs-export-latency.test.yaml: seven fixtures
(measured healthy; healthy under 60 ops/s of real load; server-slow-ours; server-slow-not-ours;
client-backed-up; backlog-deep; export-failing), and every fixture asserts all five alertnames
in the fires or the quiet direction. The measured baseline is pinned as a band, the
shared-superblock duplication is pinned, and both hazards the sibling recorded are asserted:
the PromQL precedence trap (`(ratio) and rate(ops) > 1 > 0.050` returns a series on a healthy
60 ops/s mount; the shipped form returns none) and the op-rate guards (a mount doing half an
operation a second at 1 s/op response, 1 s/op queue and 20 deep in the backlog — 20x, 200x and
2.5x the three thresholds — must stay silent).
MUTATION-CHECKED, 14 mutations, every one of them making the test file FAIL: each of the five
thresholds moved in both directions where the fixtures allow it, both `unless` clauses deleted,
and all four rate guards loosened. The two clause deletions fail with an exp/got alert diff,
not a parse error. No threshold here is pinned only by a promql_expr_test restating it.
Live on prod, all five shipped expressions return NO series while all four of their
sub-expressions return six each — so the silence is a verdict, not missing data.
runbooks/node-io.md: the triage table grows from 10 rows to 15 (per-export server response
time, client queue time, our own demand, major timeouts, backlog depth), with real measured
HEALTHY values and an explicit "not collected" in both incident columns, because the collector
postdates both events and that can never be backfilled. A new four-way verdict table reads rows
11-13 together and maps each combination to the alert that says it.
HONEST REMAINING GAP: these rules rule US out; they cannot rule Proxmox IN. The per-client view
of an export exists only on the server. What would settle it is one read on 192.168.2.102 while
NFSExportServerSlowNotOurLoad is firing — `nfsstat -s` and /proc/fs/nfsd/clients/*/info — which
is out of scope from this repo. Recorded in the runbook.
Co-Authored-By: Claude Opus 5 <[email protected]>
I recorded that NFS was 'entirely unmonitored' with 'zero NFS observability',
based on a __name__ label-values query that came back empty. The query was at
fault: count({__name__=~"node_nfs.*"}) returns six live families on all three
nodes, and the nfs collector is on by default.
The real gap is narrower — node_mountstats_* is absent, and that is the one
carrying per-mount RPC round-trip time, so NFS latency specifically is
invisible while volume and retransmissions are not.
Two further signals already existed and had simply never been queried:
container_pressure_io_stalled_seconds_total, which names the stalling workload
and for tonight names sonarr; and etcd_request_duration_seconds via the
apiserver job, which the node-io runbook had explicitly claimed did not exist.
Also tempers the Proxmox hypothesis I floated: prod-1 was doing 114 NFS reads/s
of its own during the spike, so our traffic is substantial and the stall is not
obviously unexplained by us.
Co-Authored-By: Claude Opus 5 <[email protected]>
NodeIOPressureCritical fired on prod-1 and said nothing about why; establishing
'not a failing drive' took eight ad-hoc queries after the fact. Eight new rules
each name a class in the summary — device saturation, device latency, NFS
transport, etcd, not-local-disk, and which container is stalled — with the
companion PromQL in a diagnostic_query annotation, a WHY row on the existing
dashboard, and a 10-row triage table in runbooks/node-io.md carrying real
HEALTHY / 2026-09-14 / 2026-09-07 columns.
The discriminator is NodeIOPressureNotLocalDisk, written with 'unless' rather
than 'and ... < T' on purpose: an empty right-hand side removes nothing, so it
still fires when devices are too idle to have a measurable per-op latency —
'and <' would go silent in exactly the state it exists to detect.
Acceptance is checked against both real profiles. Tonight: NotLocalDisk fires
and ContainerIOStallHigh names sonarr. 2026-09-07 re-queried live at offset
164h: 1,039 ms per write on prod-1 sda, so the latency rules fire and name the
device while NotLocalDisk correctly stays silent. Healthy state: all quiet.
10 mutations applied, 10 killed — one initially survived and exposed a
threshold pinned only by a test that restated it.
Corrects two claims: the runbook said no etcd fsync metric exists, but
etcd_request_duration_seconds is present via the apiserver job and attributes
per node; and container_pressure_io_stalled_seconds_total was already scraped
with ~400 series and unqueried.
Also fixes two bugs found in the work itself: a PromQL precedence error where
'ratio and rate(ops) > 10 > T' compares T against the op rate rather than the
ratio, and node-exporter extraArgs replacing rather than appending, which would
have silently dropped the chart's filesystem excludes.
Verified against the chart (helm show values prometheus-community/prometheus-node-exporter)
and against the live DaemonSet's argv on prod: the subchart's extraArgs default is
NOT empty — it ships the two filesystem excludes
(mount-points-exclude for /dev,/proc,/sys,containerd,docker,kubelet and the
fs-types-exclude list). Setting extraArgs to just --collector.mountstats would have
silently dropped both, so the filesystem collector would have begun reporting every
container overlay and every kubelet bind mount: a large cardinality jump, and a
behaviour change in NodeDiskFillingUp, which alerts per mountpoint. Both defaults
are now restated verbatim alongside the new collector, with a note to re-reconcile
them on a chart upgrade.
Co-Authored-By: Claude Opus 5 <[email protected]>
node-io-cause.test.yaml's healthy fixture asserts NodeIOPressureHigh and
NodeIOPressureCritical quiet on the same measured prod state the eight new cause
rules are pinned against — the point of that fixture being that the detectors and
the cause rules must agree about what "healthy" is. The shrink-only register
entries are therefore stale, and the gate correctly refused them.
Co-Authored-By: Claude Opus 5 <[email protected]>
NodeIOPressureCritical has now fired twice on k3s-prod-1 for opposite causes and
said the same thing both times. On 2026-09-14 02:40-02:45Z it fired at PSI(full)
20.98% — twice the threshold, four times the 2026-09-07 level — while every local
device was idle (0.09-0.27 utilisation) and fast (1.4-4.8 ms per write), NFS
retransmissions were zero and no Longhorn volume was degraded. Establishing that
took eight ad-hoc PromQL queries after the event had already recovered.
Adds the group platform-health.node-io-cause: eight rules, one per cause class,
each naming its cause in its own summary.
NodeDiskDeviceSaturated device saturation >0.85 busy, 10m
NodeDiskWriteLatencyHigh device latency >20ms/write, 10m
NodeDiskWriteLatencyCritical device latency >50ms/write, 5m
NodeDiskReadLatencyHigh device latency >250ms/read, 15m
NodeNFSRetransmissionRateHigh NFS transport >1% retransmitted, 10m
NodeIOPressureNotLocalDisk not local disk PSI>10% with all of the above absent
NodeEtcdRequestsSlow etcd >0.5% of etcd requests over 1s
ContainerIOStallHigh which workload per-container PSI >15%
Every threshold is measured on prod over 24-48h, not guessed, and each rule
records its headroom. The PSI thresholds themselves are deliberately unchanged:
the 21% stall was real, it fired twice in 24h for two real events on the one node
that has a problem, and raising the bar would also have raised it above the 11.36%
at which prod-1 was already losing etcd on 2026-09-07.
Context reaches the operator at page time three ways: the summary states the cause
class; every rule carries a diagnostic_query annotation with the exact companion
PromQL; and node-pressure-dashboard.yaml gains a WHY row (seven panels) plotting
saturation, write/read service time, NFS retransmissions, our own NFS op rate,
the etcd slow-request fraction and per-container PSI side by side. runbooks/node-io.md
gains a 10-row triage table with the real healthy / 2026-09-14 / 2026-09-07 columns,
which turns those eight queries into a two-minute read.
etcd and NFS metric availability, verified on prod rather than assumed:
* etcd SERVER disk metrics do NOT exist. etcd_disk_wal_fsync_duration_seconds and
etcd_disk_backend_commit_duration_seconds are absent (kubeEtcd disabled for k3s);
a query for etcd_disk_*/etcd_server_*/etcd_mvcc_* returns nothing at all.
NodeEtcdRequestsSlow uses the apiserver's client-side view of its own local
member instead, and says in its own description that it is a proxy.
* node_nfs_* DOES exist — the nfs collector is on by default, host-wide via
/proc/net/rpc/nfs, confirmed on all three nodes. It gives op counts and
retransmissions and NO latency, so it can see a dropping export but not a slow
one. The mountstats collector is enabled here to close that gap; NO rule
references node_mountstats_* yet, because the series do not exist until the
HelmRelease rolls out and the collector's visibility through the node-exporter
mount namespace must be verified on prod first. The runbook records the check
and says to revert if it comes back empty.
Proof, per the repo's two shipped alerting failure modes:
* platform/monitoring/tests/node-io-cause.test.yaml — 7 fixtures, all from
measured profiles: healthy (all 8 quiet), 2026-09-14 (NotLocalDisk fires and
names sonarr), 2026-09-07 re-queried at offset 164h (write-latency and etcd
fire, NotLocalDisk correctly silent), rebuild storm, NFS transport, read
latency, and the `unless` semantics.
* 10 mutations of the shipped rules, all killed. The mutation run found a real
gap: the etcd threshold could be changed without failing any test, because it
was only pinned by a promql_expr_test that restated the threshold in the test
file. An alert_rule_test now pins it.
* All 11 referenced metrics verified to return series on prod with
`promtool query instant`, and all 8 expressions verified quiet on today's state.
Also extends the alert-metrics-have-scrape-targets owner table for container_*
(the kubelet ServiceMonitor's cAdvisor path, which this entry already claimed in
its title) and for etcd_request_duration_seconds_* by EXACT NAME, deliberately not
by an etcd_ prefix, so a future rule on the unscraped etcd_disk_* names still fails
the gate.
Co-Authored-By: Claude Opus 5 <[email protected]>
Records three owner decisions and the finding that makes the third tractable.
The /downloads malware stays: the Proxmox-hosted stack still has operational
control of those files so deletion just triggers a re-download, nothing but the
arr suite touches the directory, and there are no Windows machines anywhere —
which removes the exposure I was concerned about. Written down so nobody
'discovers' and deletes them later.
Anti-affinity for the co-located NFS pods declined; the concentration is
accepted rather than overlooked.
And the enabling finding: there are NO NFS metrics in the TSDB at all —
node-exporter's nfs and mountstats collectors are off by default. A cluster
with four NFS mounts carrying the media library and legacy Nextcloud data has
zero NFS visibility, while PSI counts NFS stalls as I/O stalls. That is
precisely why a 20.98% stall alongside idle local disks could not be explained.
Co-Authored-By: Claude Opus 5 <[email protected]>
NodeIOPressureCritical fired on prod-1 (peak 20.98% PSI, recovered
unattended) — the first time this alert has fired on prod, and only because
the scrape gap was closed yesterday.
Evidence rules out the 2026-09-07 drive repeat: 3.5 MB/s of writes, device
utilisation 0.09-0.21, write latency 1.4-4.8 ms, zero NFS retransmissions,
21.4 MB/s NFS throughput, no degraded Longhorn volumes, and KSM never observed
NotReady. A 21% stall on trivial volume with healthy latency is transient
blocking I/O, not hardware — and PSI counts NFS too.
Structural finding: sonarr, qbittorrent and jackett all run on prod-1, the
node with the I/O history. Same concentration class as the CNPG primaries.
Separately, the /downloads files are confirmed malware: two PE executables
('MZ') sharing one MD5, padded to ~926 MB and named as two different shows,
plus a ZIP masquerading as video. Not executed, cluster not compromised; the
risk is a Windows client opening them.
Co-Authored-By: Claude Opus 5 <[email protected]>
I wrote that pfSense was stripping the RFC1918 answers. The operator corrected
it, and this repo already documented the topology I failed to read:
192.168.2.0/24 is pfSense's WAN (clusters/prod/kustomization.yaml:458,
cluster-vars.yaml:190, DECISIONS-LOG 1772), pfSense lives on 192.168.1.0/24,
and 192.168.2.1 was already recorded here as 'both gateway and only IPv4
resolver' and flagged as a SPOF (entries 1318, 1468).
The measurement is unchanged — that resolver returns NOERROR/ANSWER:0 for this
zone while resolving cloudflare.com — but the attribution invalidated my
preferred fix: configuring pfSense cannot help, because prod's resolver path
never reaches it.
Fixes reordered accordingly. Exposing Pi-hole's DNS to the LAN is now first: it
is the architecture the repo already claims, it does not depend on a consumer
router's feature set, and it retires the single-resolver SPOF this log flagged
months ago. Configuring the SOHO router drops to second, since many such
devices cannot express a domain override and it is outside GitOps regardless.
Co-Authored-By: Claude Opus 5 <[email protected]>
The workstation could not reach sonarr.home.keithjackson.dev. The stale
dnsmasq override pinning the zone to dev's dead 192.168.2.50 was the obvious
cause, but removing it — which is what I was about to recommend — would not
have fixed it: pfSense returns NOERROR/ANSWER:0 for this zone while resolving
cloudflare.com fine, the signature of private-address rebind filtering, since
the zone legitimately points at RFC1918 space.
The underlying gap: coredns-custom.yaml defers LAN clients to 'PiHole
split-horizon', but Pi-hole's Service is ClusterIP with ports 80/4180 — it
serves no DNS at all. So the LAN half of split-horizon does not exist and each
client silently depends on a hand-maintained file.
Records the corrected override (staged for the operator's sudo) and the three
proper fixes, none applied.
Co-Authored-By: Claude Opus 5 <[email protected]>
R-STATE-03 reads proven in --status as of 2026-09-14, the first requirement in
this repo to reach that state, with both verifiers recorded on prod.
Also states why the earlier PROVEN: 0 was the honest answer rather than a bug:
a proof is a claim about a specific tree on a specific cluster, so coverage
(51/51) and proof are two different questions.
Co-Authored-By: Claude Opus 5 <[email protected]>
A requirement reads PROVEN only when EVERY verifying test has a fresh ledger
row — the state is the minimum across them, so a test with no row contributes
not-derivable and one recorded verifier cannot manufacture green.
R-STATE-03 carries two executable verifiers (L0 + L3). The L3 dr-restore proof
landed in bd17b65; this records the L0 side, placeholder-guard-coupling, which
also carries R-ENV-02. Worth noting it is the test that was permanently
SKIPPED until earlier today — it was hiding a red, and now it runs and is
recorded.
Co-Authored-By: Claude Opus 5 <[email protected]>
Phase 7's outstanding exit criterion. The only previous dr-restore proof was
recorded on the dev cluster (decommissioned 2026-09-12) against MinIO, so it
could not speak for prod or for today's Cloudflare R2 target.
Run 20260914T021505Z-24b8111, env=prod, kube v1.36.4+k3s1, chainsaw v0.2.12,
150.70s, 1 passed / 0 failed / 0 skipped. Sentinel
DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK.
What makes it a real proof rather than a green check: the test seeds a canary
on the replicated longhorn class, backs the namespace AND volume data up to
the off-cluster store, then DESTROYS every in-cluster copy — namespace, PVC
and PV — before restoring into a clean, previously-nonexistent namespace. It
asserts the restored PVC binds a DIFFERENT PersistentVolume
(pvc-f0c061e1 != pvc-4107d59d) and fails explicitly if it rebinds the
original, because that would be an in-cluster survival masquerading as a
restore. The canary came back byte-identical.
It touches no real service's namespace, PVC or PV — the scratch namespaces are
named after its own Chainsaw namespace and are destroyed by design.
This is the run whose first attempt was discarded for recording env=dev; the
recorder fix is 24b8111, which this proof is recorded against.
Co-Authored-By: Claude Opus 5 <[email protected]>
record_run.py defaulted --env to the literal 'dev'
(os.environ.get("CONF_ENV", "dev")) and run.sh hardcoded the same fallback
at both call sites, while CONF_ENV is set nowhere in this repo. So every
recorded proof claimed dev no matter which cluster it ran against.
Harmless while dev was the only cluster. Dev was decommissioned 2026-09-12,
and today a genuine DR-restore run against k3s-prod was recorded as
env=dev — caught before it was committed, and the record discarded rather
than hand-edited. A proof that lies about WHERE it was proven is worse than
no proof, because locating the demonstration is the ledger's whole job; it
would also have looked like a sixth dev-only row while actually being the
prod proof Phase 7 is waiting on.
env is now derived from k3s-prod (k3s-prod -> prod),
which cannot drift from the cluster the run actually talked to. An
undeterminable context records 'unknown', never a cluster name, and an
unexpected context is recorded verbatim so it is visible rather than coerced.
CONF_ENV still overrides. Recorder selftest passes.
Co-Authored-By: Claude Opus 5 <[email protected]>
The Status section still said phases 0-6 were deployed on the DEV cluster and
that Phase 7 was not open with clusters/prod 'a design-only overlay'. Dev was
decommissioned on 2026-09-12 and prod has been the live cluster serving the
apex since. It also claimed 36/51 requirements covered with a safety-critical
gap, and that Velero was blocked on OQ-BKP.
Now states Phase 7 as open with its three exit criteria and which one is
actually outstanding; corrects the node inventory from the Proxmox dev IPs to
the bare-metal prod nodes; records Velero as unblocked to R2; and adds the
distinction the old text elided — coverage (51/51) is not proof (PROVEN: 0),
because the only dr-restore and node-loss proofs were recorded on a cluster
that no longer exists, against MinIO rather than R2.
Also records the owner's decision that Velero+kopia to R2 is the accepted
volume backup, with no Longhorn backup target, and that Longhorn config
belongs in source.
Co-Authored-By: Claude Opus 5 <[email protected]>
runbooks/longhorn-degraded.md asserted concurrentVolumeBackupRestorePerNode-
Limit and snapshotHeavyTaskConcurrentLimit were tightened 5 -> 1 after the
2026-09-07 etcd I/O starvation. Both read 5 on prod and neither name existed
anywhere in the repo. Now declared in the helmrelease defaultSettings, the
path Flux actually reconciles.
Both go to 1 rather than 2, and the cost is smaller than I assumed: velero's
schedules run defaultVolumesToFsBackup, i.e. kopia through the node-agent, so
they are not Longhorn volume backups and this setting does not bind them;
CNPG's base backups are barman-level; and Longhorn has no backup target
configured at all. Purge work comes from rebuild cleanup, which is already
serialised to 1/node, so 1 keeps pace with its own generator while 2 would add
a second heavy writer during exactly the rebuild that starved etcd.
Also declares nodeDrainPolicy, which was only ever Longhorn's own default —
present in no manifest and no rendered ConfigMap, so an upstream default
change would have moved it silently while two runbooks reason about drain
safety from it. Value unchanged.
The Ansible etcd_io_priority role is deliberately NOT ported. The pathological
drive it was written for has been replaced (I/O PSI 11.36 -> 3.79 under the
same load class), the role was never run against any cluster, and its own
README concedes it cannot separate the tenants because they share one ext4
filesystem. Swapping the elevator and halving nr_requests on the live device
holding etcd's WAL is a node-wide blast radius for a low ceiling. Kept as a
named operator option; the real fix remains giving etcd its own device.
Removes orphanAutoDeletion, re-verified present-and-inert two ways: helm
template emits no orphan key for it, and the live setting is NotFound.
`runbooks/longhorn-degraded.md` told operators that, after the 2026-09-07 etcd
I/O-starvation incident, `concurrentVolumeBackupRestorePerNodeLimit` and
`snapshotHeavyTaskConcurrentLimit` had been tightened 5 -> 1. Measured live on
prod today: both read **5**, and neither name appeared anywhere in this
repository — so nothing declared them and nothing would have restored them.
Fourth instance of config that looked applied and was not.
Landed in `platform/longhorn/helmrelease.yaml` `defaultSettings` (the path Flux
reconciles; `values.yaml` is the reference mirror), NOT by `kubectl patch` —
a hand-applied Setting is exactly how this gap arose and Helm would silently
rewrite it on its next upgrade.
concurrentVolumeBackupRestorePerNodeLimit 5 -> 1
snapshotHeavyTaskConcurrentLimit 5 -> 1
replicaReplenishmentWaitInterval 600 PINNED, unchanged
backupConcurrentLimit / restoreConcurrentLimit 2 / 2 PINNED, unchanged
concurrentAutomaticEngineUpgradePerNodeLimit 0 PINNED, unchanged
nodeDrainPolicy block-if-contains-last-replica PINNED — it was in
force only as Longhorn's OWN DEFAULT, declared in no manifest and in no
rendered ConfigMap, so an upstream default change would have moved it
silently while two runbooks reasoned about drain safety from it.
EVERY KEY PROVEN TO LAND, not assumed. `helm template longhorn/longhorn
--version 1.12.1` (the version this chart range resolves to, per the
HelmRelease's own lastAttemptedRevision) on this branch's spec.values,
substituted as clusters/prod substitutes them, emits all eight into the rendered
`longhorn-default-setting` ConfigMap in kebab-case. Diffed against the LIVE
ConfigMap the change is purely ADDITIVE: 8 keys added, none altered, none lost.
The chain to the Settings is verified read-only on prod — all 115
settings.longhorn.io carry `longhorn.io/configmap-resource-version: 7917`
(= that ConfigMap's resourceVersion) with `status.applied: true`, and every key
in the ConfigMap equals its Setting's value. The two targets sit at 5 with the
key absent, i.e. Longhorn defaults, as measured.
`orphanAutoDeletion: true` REMOVED, verified inert two independent ways against
the deployed 1.12.1: the render emits no orphan key at all (Longhorn 1.9 renamed
it `orphanResourceAutoDeletion`, a list, and the chart drops unknown
defaultSettings keys with no error), and live, `setting/orphan-auto-deletion` is
NotFound while `orphan-resource-auto-deletion` reads "". Deliberately NOT
replaced — orphan deletion is extra writes on the constrained partition and buys
only space. Orphan cleanup stays manual by decision; the two runbooks that cite
the key by name are corrected rather than left to go stale.
BOTH NEW CAPS ARE 1, AND 1 COSTS SOMETHING — stated, not waved at.
`concurrentVolumeBackupRestorePerNodeLimit: 1` costs nothing TODAY: Longhorn has
no backup target (the `backup-target` Setting does not exist; still the
WS-STORAGE Phase-4 TODO), and velero's nightly run is not a Longhorn backup —
`snapshotVolumes: false` + `defaultVolumesToFsBackup: true` is kopia
file-system backup through velero's node-agent, which this Setting does not
gate; CNPG base backups are barman-level and ungated too. Its only consumer is
an R-STATE-03 restore, where serialising ~24 volumes beats losing etcd
mid-restore. Revisit when a Longhorn backup target lands.
`snapshotHeavyTaskConcurrentLimit: 1` rather than 2 because purge work is
generated by rebuild cleanup (auto-cleanup-system-generated-snapshot true, NO
RecurringJobs, 24 volumes / 24 snapshots) and rebuild is already serialised to
1 per node — so 1 keeps pace with its own generator, where 2 buys backlog drain
at the price of a second heavy writer during the exact rebuild that starved
etcd. The cost: space reclaims serially after a rebuild storm — real here (a
37.7 GiB snapshot once pinned a volume at 97.3% allocated), but instrumented by
LonghornDiskAllocationHigh and recoverable by hand, where a wedged etcd is not.
Headroom on the record: I/O PSI full is 0.30/0.62/0.37 % now, and 3.79 under a
full 18-replica rebuild after node 1's drive replacement, against a 5% warning
threshold. These caps are a BOUND, not a fix — etcd and Longhorn still share one
ext4 filesystem and nothing arbitrates the jbd2 journal fdatasync waits on.
NOT PORTED, deliberately: the same unmerged branch's node-level I/O arbitration
(`roles/etcd_io_priority` — BFQ on the shared device, nr_requests 64 -> 32,
systemd IOWeight drop-ins). The 2011 C400 it was written for was replaced on
2026-09-07 and the same load class measured 11.36 -> 3.79 I/O PSI with the node
Ready throughout; the role was never run against any cluster, its author
recorded it as unproven with a node-3-first sequenced rollout, and its own
README concedes it does not separate the two tenants. Node-wide blast radius,
low ceiling, stale measurements (node 1 reinstalled, founder moved to node 2).
It stays an operator option if I/O PSI climbs back toward 5% — which, unlike in
September, we would now be told about. Nothing under infra/ansible/ is touched.
ALSO ALREADY DONE, so skipped: that branch's NodeIOPressureHigh/Critical and the
dashboard I/O row are on main already (efc6db6) with runbooks/node-io.md.
Verified live, not by reading the file: both rules loaded in
platform-health.nodes, health ok, state inactive, and
ALERTS{alertname=~"NodeIOPressure.*"} returns no samples.
Runbooks corrected in both directions: longhorn-degraded.md's prominent "DO NOT
RELY ON THAT: it was never applied" block becomes an accurate statement of what
is now declared, what to verify before trusting it, what it cannot do, and what
it costs; longhorn-capacity.md / node-disk.md stop citing a key that no longer
exists; node-io.md gains the one line an operator needs mid-incident.
No new ${...} substitution variables and not one dollar sign added anywhere in
the diff, so neither StrictPostBuildSubstitutions nor the L0 doubled-dollar guard
is affected.
Co-Authored-By: Claude Opus 5 <[email protected]>
Two things found while reviewing the Longhorn alert work.
Prometheus retentionSize was 45GiB of a 50Gi PVC, under a comment already
naming the right intent — 'headroom for WAL + head + compaction scratch'.
5 GiB is not that: retentionSize counts only persisted blocks, and a large
compaction can need roughly the size of the blocks being merged. ENOSPC on
/prometheus wedges every WAL commit, so scrapes keep succeeding while nothing
is stored and every absent()/==0 alert fires falsely at once. Lowered to
36GiB, which costs no history: the TSDB is 11.2 GiB at 30d, so 30d remains
the binding limit.
runbooks/longhorn-degraded.md asserted that concurrentVolumeBackupRestore-
PerNodeLimit and snapshotHeavyTaskConcurrentLimit 'was actually tightened
5 -> 1'. Both read 5 on prod and neither name appears anywhere in this repo —
the work is on the unmerged fix/etcd-io-priority branch. The page is read
during etcd I/O starvation on a node whose WAL shares a device with Longhorn,
so it was telling an operator mid-incident that the cap they depend on is in
place when it is not. Corrected, with the two mitigations that ARE real named
and a command to verify any of them.
Co-Authored-By: Claude Opus 5 <[email protected]>
(1) LonghornVolumeCapacityHigh alarmed on ALLOCATION while calling itself a
capacity alarm. Prometheus's TSDB volume read 0.973 while the kubelet saw the
same filesystem at 0.208, because 37.73 GiB of it is a Longhorn SYSTEM
snapshot — the kind taken for the full replica rebuild that prod-1's drive
replacement forced — growing +152 MB/h while the filesystem stayed flat. The
rule was therefore unclearable by design, i.e. permanently firing. Deleted as
redundant, shown rather than asserted: 23 of 24 volumes fall under the
kubelet-based filling rules, and the 24th is volumeMode Block, which the
deleted rule was not covering either.
(2) LonghornVolumeDegraded and LonghornVolumeFaulted could not fire even with
the scrape working. longhorn_volume_robustness on 1.12.1 is ONE-HOT over a
state label with values 0/1, not the 0-3 enum the rules and their comment
assumed, so a faulted volume would have alerted nobody. This survived the
scrape fix landing hours earlier.
(3) LonghornNodeStorageHigh was wrong twice: its 0.85 threshold sat above the
75% at which storage-minimal-available-percentage=25 already removes a disk
from the scheduling pool, and longhorn_node_storage_usage_bytes SUMS a node's
disks, so one full disk hid behind a half-empty sibling. Replaced with
per-disk LonghornDiskAllocationHigh (>0.70) and LonghornDiskUnschedulable,
the latter reading Longhorn's own verdict instead of re-deriving a formula
from three settings no metric exports.
Baseline green with 11/11 mutations red. retentionSize 45GiB of 50Gi and
discard/TRIM are both REPORTED with recommendations, not changed: TRIM would
not have helped here, since remove-snapshots-during-filesystem-trim is false
and the 37.7 GiB is a snapshot.
# Conflicts:
# DECISIONS-LOG.md
LonghornVolumeCapacityHigh went pending within minutes of the Longhorn scrape
landing today. Re-measured on prod: it read 0.9731 on the Prometheus TSDB volume
(actual_size 52243681280 / capacity 53687091200) while the kubelet reported the
SAME filesystem 20.8% used. actual_size is head + pinned snapshots, and one
user_created=false system snapshot from prod-1's replica rebuild holds 40510484480
bytes (37.7 GiB) of it; Longhorn returns no freed blocks without discard/TRIM, and
retentionSize 45GiB on a 50Gi volume targets 90% by design. The ratio is pinned
near 1.0 and cannot fall back under 0.85 — a permanently-firing alert, the same
shape as CNPGNoRecoverabilityPoint.
DELETED, not re-thresholded. Raising the threshold would tune away a true
statement about allocation. Redundancy proven, not asserted: of 24 Longhorn
volumes on prod, 23 appear in kubelet_volume_stats_inodes > 0 and so fall inside
MonitoringVolumeFillingUp (namespace="monitoring") or PersistentVolumeFillingUp /
PersistentVolumeAlmostFull. The 24th is unifi/unifi-rootdisk, volumeMode Block —
no filesystem for anyone to measure, and the deleted rule was not covering it
either.
AND A SECOND INERTNESS the scrape did not fix. longhorn_volume_robustness on
longhorn 1.12.1 is one-hot over a `state` label with values 0/1, not the 0-3 enum
the rules assumed:
count(longhorn_volume_robustness) => 96 (24 volumes x 4 states)
count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
count(longhorn_volume_robustness == 2) => (empty)
count(longhorn_volume_robustness == 3) => (empty)
So a FAULTED volume would still have alerted nobody. Re-keyed to the label; the
"44 series / 72 unknown(0, detached)" note in platform/longhorn/helmrelease.yaml
was the same misreading and is corrected in place.
The allocation question moved to where allocation is a budget. LonghornNodeStorageHigh
fired at 0.85, but storage-minimal-available-percentage is 25 on this cluster, so a
disk leaves the scheduling pool at 75% used — the old threshold could only fire
after scheduling had already been refused. Replaced by:
LonghornDiskAllocationHigh (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70, 30m
LonghornDiskUnschedulable longhorn_disk_status{condition="schedulable"} == 0, 15m
Per DISK, not per node: longhorn_node_storage_usage_bytes sums a node's disks, so
one full disk could hide behind a half-empty sibling. Measured prod usage
0.2254 / 0.2306 / 0.2369 on prod-1/2/3.
PROVEN NEITHER PERMANENTLY FIRING NOR INERT. platform/monitoring/tests/longhorn.test.yaml
adds 6 cases pinning every rule quiet on today's measured prod state and firing on
a constructed bad state, with every `for:` asserted, plus regression assertions that
the old `== 2`/`== 3` forms match nothing and that per-disk 0.72 is visible where the
node-summed 0.475 is not. Mutation-checked 11 ways; each turns the suite RED. Live on
prod: all four new expressions return empty, while the sum of their selector
cardinalities is 54. The L0 UNTESTED_ALERTS register shrinks by seven (7/56 -> 14/58
alert rules unit-tested), which also brings MonitoringVolumeFillingUp,
PersistentVolumeFillingUp and PersistentVolumeAlmostFull under test since the
redundancy claim rests on them.
REPORTED, NOT CHANGED (DECISIONS-LOG, owner decisions): retentionSize 45GiB on a
50Gi volume is too tight — recommend 36GiB, with reasoning; and discard/TRIM on the
longhorn StorageClass is feasible but not advisable now — remove-snapshots-during-
filesystem-trim reads false on prod, so trim would not have reclaimed the 37.7 GiB
anyway, and continuous discard on nodes where the etcd WAL shares the spindle points
at this cluster's known failure mode.
Co-Authored-By: Claude Opus 5 <[email protected]>
Each <app>-down rule read up{job=...} from a ServiceMonitor selecting a
Service port named 'metrics' that no app Service defined, so every one
discovered zero targets. PrometheusTargetsMissing could not see it either: a
target never discovered has no up series to be 0.
The port name was only the first of THREE layers. None of the twelve apps
serves Prometheus metrics natively at all — Plex, qBittorrent, the arrs,
Jackett, Pi-hole, Seerr, Semaphore and Nextcloud would each need an exporter,
and Gitea needs an app.ini flag. And every app namespace ships default-deny
ingress with no monitoring allow, so a correct port would still have been
dropped by policy — while renderer-contract.md claimed a baseline allow that
never existed.
Fixed in catalog/render.py (schema 1.6.0) rather than in twelve rendered
files: slis: [up] now renders <app>-down against kube-state-metrics
readiness, with an 'or absent(...)' half so the signal VANISHING fires
instead of silencing. The twelve dead ServiceMonitors are gone; a new
observability.metricsEndpoint renders the Service port, the monitor endpoint
and the scrape allow together, so the three cannot disagree again.
Deletion was rejected with evidence: kube-prometheus-stack's generic
kubernetes-apps rules are live, but they are namespace-keyed warnings, not
the per-service critical SLI R-OBS-01 asks for.
Two incidental finds: keycloak's rendered monitor was a duplicate pointing at
the wrong port, and platform/monitoring/tests/run.sh discovered test suites
only inside its own directory while documenting the convention as general —
so a suite added anywhere else was never run and still looked like coverage.
Twelve `<app>-down` SLI rules were STRUCTURALLY INERT. Each read `up{job="<app>"} == 0`,
and the ServiceMonitor rendered beside it named `endpoints[].port: metrics` — a port NO
rendered app Service defined. Every monitor discovered ZERO targets, so no `up{job="<app>"}`
series existed and the rule could not fire. Verified on prod before this change:
`count by (job) (up)` listed 24 jobs and not one of the twelve appeared; `up{job="plex"}`
and `up{job="gitea"}` both returned nothing. `PrometheusTargetsMissing` could not see it
either — a target that was never discovered has no `up` series to be 0.
THE PORT NAME WAS NEVER THE MISSING PIECE. Ground truth per app: Plex, Pi-hole, Jackett,
qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud export NO Prometheus metrics
without a separate exporter sidecar; Gitea can but only with app.ini metrics enabled;
Keycloak does, on the operator's management port, already scraped by
platform/keycloak/servicemonitor.yaml; vm-demo is a KubeVirt VM with no Service at all, so
its monitor selected nothing whatsoever. `metrics: true` was asserting a fact none of them
had. There was also a THIRD layer of inertness nobody had hit yet: every app namespace ships
default-deny with `ingress: []` and the renderer emitted no monitoring allow, so even a
correct port would have been dropped by policy — while renderer-contract.md claimed the
baseline carried such an allow.
FIXED IN THE GENERATOR, not in twelve rendered files:
* `slis: [up]` now renders a `<app>-down` alert against kube-state-metrics readiness of the
service's own workload object —
`kube_deployment_status_replicas_available{namespace,deployment} == 0 or absent(...)`,
`kube_statefulset_status_replicas_ready` for a StatefulSet, and
`absent(kubevirt_vmi_info{namespace,name,phase="running"})` for a VM. The same idiom
KeycloakDown has used since it was fixed. kube-state-metrics IS scraped, so this needs no
exporter and no new target. The `or absent(...)` half is load-bearing: the signal
DISAPPEARING now fires the alert instead of silencing it.
* schema 1.6.0 adds `observability.metricsEndpoint` — WHERE metrics are served. A
ServiceMonitor is rendered ONLY from it, and it renders all three halves of a working
scrape together: the named Service port, the monitor endpoint, and the
monitoring-namespace CiliumNetworkPolicy allow. No catalog entry declares one, so twelve
dead ServiceMonitors are gone. `metrics: true` keeps meaning logs-to-Loki plus
kube-state-metrics liveness.
* schema 1.6.0 also adds `observability.livenessWorkload`, and catalog/keycloak.yaml uses
it. Keycloak's server is a StatefulSet run by the operator; the default Deployment shape
would have made its `absent()` half fire FOREVER — the other failure mode this repo has
shipped.
* renderer selftest gains cases 9/9a/9b/9c: no metricsEndpoint => no monitor; the up SLI
must not read `up{job=...}`; endpoint port must exist on the Service; the metrics port
must NOT reach the Gateway allow; an unknown livenessWorkload.kind is REFUSED.
PROOF, both directions, for all twelve (apps/wave-{0,1,2}/tests/app-slis.test.yaml, run by
platform/monitoring/tests/run.sh over the RENDERED manifests):
* quiet on the healthy state; firing on no-ready-replica; firing when the series is ABSENT;
the VM asserted both gone and in a non-running phase; per-app isolation asserted, and
keycloak asserted un-maskable by its neighbours (keycloak-operator Deployment,
keycloak-db StatefulSet ready in the same namespace).
* mutation-checked: six mutations, each turning its suite red — reverting plex to
`up{job=...} == 0`; dropping seerr's `absent()` half; pointing keycloak at a Deployment;
capitalising the VMI phase to `Running`; widening keycloak's matcher to the namespace;
inverting plex's `== 0` to `!= 0`. Control run green.
* live on prod with `promtool query instant`: all ten deployed app Deployments export
`kube_deployment_status_replicas_available`,
`kube_statefulset_status_replicas_ready{namespace="keycloak",statefulset="keycloak"}` = 2,
`kubevirt_vmi_info` present with a lowercase `phase="running"`; and evaluating the twelve
SHIPPED expressions returns no series for eleven (quiet, not permanently firing). The
twelfth is vm-demo, which is not in apps/wave-0/kustomization.yaml and so is not deployed
— its rule is not applied either.
DELETION WAS NOT THE ANSWER, and the evidence is recorded rather than assumed:
kube-prometheus-stack's `kubernetes-apps` group IS live on prod (17 rules, KubePodNotReady /
KubeDeploymentReplicasMismatch among them), so app pods are not unmonitored — but that is a
generic warning keyed to a namespace, not the per-service CRITICAL SLI R-OBS-01 declares.
platform/monitoring/alerts.yaml carries no generic app-readiness rule at all.
GUARD SO IT CANNOT RECUR SILENTLY: new whole-repo L0 invariant
conformance/tests/l0-static/servicemonitor-ports-resolve. It walks the MONITORS, not the
alerts, so it needs no `up{job=...}` rule to reach one, and resolves each endpoint by SELECTOR
across the whole tree rather than by same-file adjacency. Fixing the alerts DELETED the only
thing that made the broken monitors visible to the sibling check, which is exactly why this
axis needed its own home. It declares `conformance.home.k3s/scope: whole-repo` and adds no
ci/l0.sh stage. Proven to FAIL on the pre-fix tree: 15 failures — eleven port disagreements,
vm-demo's monitor matching no Service at all, and the three generator/contract assertions —
and to pass after. Its resolver is additionally proven on in-check fixtures (agreeing pair
accepted, disagreeing pair rejected, target-less endpoint rejected), because after the fix
there are zero in-repo monitors left for it to judge and a guard whose only evidence is
"nothing failed" cannot be told apart from one that stopped working. It also asserts
r_servicemonitor does not HARDCODE a port name — the mechanism by which one edit gave twelve
services the same wrong one.
Also fixed, found on the way:
* platform/monitoring/tests/run.sh discovered the SUITE convention only inside
platform/monitoring/tests/ while documenting it as general — a suite added anywhere else
was never run and still looked like coverage. Now discovered tree-wide; a misplaced
*.test.yaml fails loudly instead of vanishing.
* KNOWN_UNWIRED_JOBS in alert-metrics-have-scrape-targets is now EMPTY, and the twelve
apps/* entries in alert-rules-unit-tested's UNTESTED_FILES are gone — both because the
gaps were CLOSED, not waived. Both registers are shrink-only and a stale entry fails.
* renderer-contract.md claimed the default-deny baseline carried a standing monitoring
scrape allow. It never did; the claim is corrected and the allow is now real, per declared
metrics endpoint and scoped to that one port.
Co-Authored-By: Claude Opus 5 <[email protected]>
The new flux-postbuild-substitution guard failed the merge on
apps/wave-0/qbittorrent.yaml:155 — a run of two dollars in a substituted
path that nothing justifies. It was inside the postStart hook's own comment
EXPLAINING the hazard: prose about what envsubst does to a doubled dollar,
written with a literal doubled dollar.
Functionally harmless — it is a bash comment, and collapsing it changes no
behaviour — but the guard cannot distinguish a comment from content that
reaches the workload, and it is right not to try: that is precisely how the
nextcloud-db break slipped through. Reworded in words instead.
Worth recording that this is the second time today the same trap caught an
explanation of itself: my own fix for the nextcloud-db incident (16b7fcd)
also shipped dollars inside the comment describing the danger, and I only
caught it because I re-read the file. This time the guard caught it, on work
merged minutes earlier — which is the guard earning its place.
Re-rendered so the golden matches the catalog.
Co-Authored-By: Claude Opus 5 <[email protected]>
R-C2-01 and R-C2-02 were the last two uncovered requirements (49/51). Now
100% with 0 gaps, via an offline L0 check (49 assertions) that compares two
repository facts: every home-zone hostname the repo routes over HTTP must be
linked from the apex dashboard, and every link the dashboard offers must
resolve to something the repo actually serves at L7 or L4.
'A service the root hid' was still a live defect: crafty.${BASE_DOMAIN} was
served and reconciled while nothing on the root linked to it. Committed RED
first so the history shows the check catching it, then fixed.
Also fixes a regex that could never have worked: ci/touched-requirements.sh
matched R-[A-Z]+-[0-9]+, where [A-Z]+ eats the C of R-C2-01 and then demands
a dash where the 2 is — so a manifest declaring either ID attributed nothing.
Every other consumer already used [A-Z0-9]+.
16 mutations proven caught. R-TLS-02's drafted amendment is recovered into
DECISIONS-LOG: revocation is not implementable with cert-manager, and the
choice between amending the requirement and a reviewed DOCUMENTED_GAPS entry
is the owner's.
# Conflicts:
# DECISIONS-LOG.md
qBittorrent 5.x minted a new temporary admin password on every process
start, so the Sonarr/Radarr download clients broke on the next restart — and
had already broken: both reported isValid: false, Authentication Failure.
A SOPS secret is now the single source of truth and a postStart hook derives
qBittorrent's stored PBKDF2 form on every container start, then logs in to
prove it. The hook rather than a CronJob because the failure IS a container
start: qbittorrent-nox reads the credential once at start from an RWO PVC the
app holds, so a 15-minute cron leaves a window after every restart where both
arrs are broken. An initContainer cannot do the ground-truth login.
Proven: testall flipped false -> true on both arrs, and survived a real pod
deletion and recreation with the conf still holding both keys.
Two bugs found in the inherited work, both the envsubst trap again: a
doubled-dollar port expansion that bash read as its own PID, which made the
hook report 'the WebUI rejected the credential' three times without ever
asking it anything; and one shared wait budget that let the first attempt
burn all 90s. Also caught that the SOPS file was never listed in
apps/wave-0/kustomization.yaml, so Flux would never have applied it — the
live Secret was not going to be adopted at all.
Working set measured at 111 MiB with a 6h peak of 123.3 MiB against a 128Mi
limit, so it died on every spike — exit 137, roughly every 11 minutes, 91
restarts in 16h.
The consequence is worse than the restarts. Thirteen alert rules read kube_*
series, so each kill made them intermittently unable to fire, which is the
quiet version of the inert-alert class found elsewhere today; KubePodNotReady
and KubeDeploymentReplicasMismatch were both sitting pending on it.
Self-inflicted: enabling customResourceState for the Velero Schedule metrics
earlier today added a CRD informer plus the mandatory CRD list/watch, and
that marginal cost crossed a limit already sitting about 4 MiB above the
working set. Raised to 384Mi, roughly 3x the observed peak, which costs
nothing on nodes reporting 18.7-22.3 GiB available.
Co-Authored-By: Claude Opus 5 <[email protected]>
Finishes the inherited WIP. qBittorrent 5.x had no WebUI\Username or
WebUI\Password_PBKDF2 in its conf, so it minted a NEW random admin password on
every process start. Both consequences had ALREADY fired: measured before this
change, `POST /api/v3/downloadclient/testall` returned isValid:false with
"Authentication Failure" on BOTH Sonarr and Radarr, so Completed Download
Handling was silently broken, and no human had a stable login either.
Mechanism: the SOPS/age qbittorrent-bootstrap Secret is the single source of
truth; a lifecycle.postStart hook derives qBittorrent's own stored form
(PBKDF2-HMAC-SHA512, 100000 iterations, 16-byte salt, 64-byte digest) and
asserts it into the conf on EVERY container start, then logs in to the WebUI to
prove it. Level-triggered like the repo's CronJob reconcilers, but triggered by
the container start because the failure it closes IS a container start: a cron
would leave a window after every restart in which the app serves a random
password, and would have to edit an RWO PVC from outside, under a running
process. AuthSubnetWhitelist was rejected — authentication is not weakened.
Two real bugs found in the WIP, both the postBuild-substitution trap:
* PORT came from a doubled-dollar WEBUI_PORT expansion. Correct on the cluster
(envsubst collapses it) and the renderer accepts it, but run any other way
bash reads the doubled dollar as its PID, so every curl got an unparseable
URL and the hook logged "the WebUI rejected the durable credential" three
times having never asked it anything — see the marker log at 05:35Z. Now a
literal PORT=8080; the script contains no braced expansion at all and
behaves identically however it is run.
* One shared wait budget covered both the s6 wait and the API wait, so the
first consumed all 90s and attempts 2-3 fired instantly. Separate counters
now; "the API never answered" is reported as a distinct fault from "the
password was rejected"; and a rejection restarts qbittorrent-nox so a retry
can actually change its outcome.
apps/wave-0/kustomization.yaml did not list the SOPS secret, so Flux would never
have applied it and the adoption of the live-only, kubectl-client-side-apply
qbittorrent-bootstrap Secret would silently not have happened. Listed now; the
decrypted values are SHA-256-identical to the live Secret's, so adopting it
cannot change the password.
Proven on prod: testall flipped isValid false -> true on both apps; after
deleting and recreating the pod the conf still held both keys, the WebUI answered
Ok. to a login with the credential decrypted from git, and both arrs still
reported isValid:true. The post-envsubst manifest was audited as `flux build
kustomization` emits it (no doubled dollar, no braced expansion, PORT=8080,
bash -n clean) and the hook's write path was exercised in-container against a
scratch conf: exit 10 when the keys are absent, exit 0 and a byte-identical file
on a second run, digest verified by an independent PBKDF2.
NOT proven, and recorded as such in DECISIONS-LOG: the hook has not yet run as a
real postStart on prod — the Deployment is Flux-owned from main and both routes
to the live object were refused by the session's permission classifier. Verify
on merge that the marker log gains a VERIFIED line on the next pod start.
Co-Authored-By: Claude Opus 5 <[email protected]>
The R-TLS-02 analysis and its drafted spec amendment were written on
feat/coverage-debt-paydown (b37defe, 2026-09-07) and lived ONLY in that branch's copy of
DECISIONS-LOG.md. That branch's tree is ~107 commits behind main and must never be merged,
so the amendment went out of view with it — the second time this decision has been raised
and lost. Recovered, re-checked against today's main, and written where it cannot vanish
with a branch.
THE SUBSTANCE IS UNCHANGED. R-TLS-02 says certificates MUST be automatically "issued,
renewed, and revoked without manual steps". Issuance and renewal hold. Revocation is not
implemented and is not implementable with the stack ADR-0006 selects: cert-manager has no
revoke-on-delete, no Certificate field requesting revocation, and no controller that calls
ACME revokeCert, and neither ADR-0006 nor docs/08 specifies a mechanism.
enableCertificateOwnerRef is NOT the remedy — it garbage-collects the Secret, which is
cleanup, not revocation — and it is a global controller flag that would drop the Gateway's
wildcard-home-tls Secret on any future Certificate rename/prune, so it needs its own risk
assessment rather than a side effect of a test task. Still not set.
WHAT CHANGED SINCE THE FIRST ESCALATION, and why it now matters more: R-TLS-02 is no
longer a COVERAGE gap (l0-static/tls-lifecycle-automated + l2-behavioral/
cert-lifecycle-automated both carry the label and main counts it covered), and the L0 check
is honest about the boundary in its own verdict line ("'revoked' is NOT proven at any
layer"). But the L2 test deletes a Certificate and requires its Secret to disappear, and
its failure message says "set enableCertificateOwnerRef: true ... (owner decision — WS-TEST
does not weaken the assertion instead)". That assertion is written to FAIL on a live run
until the owner decides. The decision is now arriving as a red test rather than as a
question, which is exactly how it gets weakened by whoever meets it next.
Two options are recorded verbatim for the architect, in the drafted preference order:
amend the requirement so revocation is an explicitly manual, documented break-glass action
(which also makes enableCertificateOwnerRef a CONSEQUENCE of the requirement), or keep the
text and record a reviewed DOCUMENTED_GAPS entry naming the clause, the reason and the
removal condition.
NOT APPLIED, and not appliable by an agent: the spec repo is read-only under R-TEST-06.
This is a request for a human decision, stated plainly as one.
Co-Authored-By: Claude Opus 5 <[email protected]>
Checked as part of closing the C2 coverage gaps, and it was live on main. The content
scan's pattern was `R-[A-Z]+-[0-9]+`, which CANNOT match an ID whose workstream segment
carries a digit: `[A-Z]+` consumes the `C` of `R-C2-01` and then demands a `-` where the
`2` is. `R-C2-01` and `R-C2-02` are the only two such IDs in
conformance/requirements/registry.yaml, and the scan was blind to BOTH.
REPRODUCED before the fix, on a throwaway repo whose only change is a manifest declaring
both IDs on real (non-comment) lines, in a path the map keys to nothing:
$ REPO_ROOT=$T BASE_REF=main ci/touched-requirements.sh
R-NET-06 # the all-letter control ID was the ONLY output
It failed by UNDER-attributing, which is the fail-OPEN direction: a change that retyped a
`conformance.home.k3s/R-C2-01` tag, or a manifest self-declaring R-C2-02, contributed
those IDs to NOBODY, so the C2 tests were filtered out of the merge gate. Same defect
class as the map's `*) : ;;` arm — a miss that reads as "nothing touched".
Every other consumer in the repo already allowed the digit, which is why this was the lone
dissenter: test_requirements_for_path() twenty lines above in this same script,
conformance/lib/coverage.py, conformance/lib/status.py and conformance/lib/
run_l1_offline.py all use `R-[A-Z0-9]+-[0-9]+`. Aligned to that form, with the reason
recorded at the line.
SCOPE OF THE HOLE. path_requirements() has no arm for platform/homepage/* or
platform/headlamp/* either, so before this the ONLY route to R-C2-01/02 was detection rule
3 (editing the C2 conformance test itself). No path-map arm is added here: c2-root-surface
declares `scope: whole-repo`, so ci/l0.sh Stage 3b's --include-scope union selects it on
EVERY gate invocation regardless of the touched set — which is the right mechanism for an
invariant whose input surface is the whole tree, and strictly stronger than a two-component
path arm would be.
REGRESSION-TESTED, in ci/tests/touched-requirements.test.sh (the suite `make -C ci test`
runs): a fixture declaring R-C2-01 / R-C2-02 / R-OBS-04 from `notes/` — a path outside
path_requirements(), so the IDs can only come from the content scan — asserts both digit
IDs ARE attributed, that the all-letter form still is (control), and that a purely numeric
workstream is not invented (the pattern must not widen the gate on noise).
PROVEN TO DETECT BREAKAGE: with the pattern reverted to `R-[A-Z]+-[0-9]+` the suite reports
`FAIL (2 failed, 14 passed)` naming both C2 IDs as missing; restored, all three ci/ suites
pass (16 assertions in this one).
ALSO: conformance/tests/l0-static/whole-repo-invariants-wired/check.sh cites this pattern
in the comment explaining why its BOGUS_REQ fixture is lowercase. Citation updated so the
reasoning still reads true; the fixture is unaffected (grep -E is case-sensitive, and the
check still passes 26/26).
NOTE: another branch may be editing this same script concurrently. This is a one-line
change to one statement plus its comment — expect a trivial conflict and keep the
`[A-Z0-9]` form.
Co-Authored-By: Claude Opus 5 <[email protected]>
The R-C2-01 check added in the previous commit is RED on main for a real reason.
platform/crafty/httproute.yaml serves crafty.${BASE_DOMAIN} on the shared Gateway,
clusters/base/crafty.yaml reconciles it (dependsOn platform-config, wait:false), the web
UI is Keycloak-gated by a per-pod oauth2-proxy sidecar — and nothing on the operator's
front door linked to it. The service was running and the root did not admit it existed.
R-C2-01: "The root `home.keithjackson.dev` MUST serve a UI that surfaces the services
running in the cluster and links to them." An unsurfaced route is a requirement gap, not
a missing nicety, and it is the exact failure mode nothing else in the gate notices: every
cluster test stays green, the apex answers 200, and the estate is one service larger than
the dashboard says.
This is the SAME defect the unmerged 49c160d found on 2026-09-07 (feat/coverage-debt-
paydown, ~107 commits behind main). That branch was never merged, so the gap survived —
re-fixed here against today's tree, in token form (`crafty.${BASE_DOMAIN}`, not the
literal hostname that branch predated).
Grouped under Media because Crafty manages game servers; Platform would be equally
defensible and purely cosmetic. The tile also does duty for the L4 game ports
(crafty-mc.${BASE_DOMAIN}, TCP 25565-25567 / UDP 19132-19134), which have no browsable URL
of their own — the check reports those as an advisory rather than requiring a link.
ALSO: clusters/base/homepage.yaml's postBuild comment claimed "Homepage references no
${VARS} today; the substitution is retained for parity". That is no longer true and the
claim is dangerous, because it invites someone to delete the substitution as dead weight.
This component now references ${BASE_DOMAIN} in httproute.yaml's apex hostname AND in every
tile href; without substitution the token is applied literally, the apex route binds a host
matching no listener, every link on the root is dead, and Flux still reports Ready. Comment
corrected, and the property is asserted by c2-root-surface (5) so it cannot rot back.
Verified: conformance/tests/l0-static/c2-root-surface -> check OK, 49 assertions.
Co-Authored-By: Claude Opus 5 <[email protected]>
The Test DECLARATION landed in the preceding WIP commit with no check.sh, so the test
could not run at all. This is the check: 49 offline assertions over the repository as
git tracks it, no cluster, no network, no spec repo (R-TEST-06).
THE RULE. Every home-zone hostname the repository routes over HTTP must be linked from
the apex dashboard's configuration; every home-zone link the dashboard offers must
resolve to something the repository actually serves (an HTTPRoute at L7 or a
hostname-publishing LoadBalancer Service at L4); and the two R-C2-02 surfaces must be
linked, reconciled and backed by real alert rules, with operational rights arriving
per-user through OIDC groups rather than through a shared cluster-admin identity.
IT IS RED ON MAIN, on the forward direction:
FAIL (8) every service the repo ROUTES over HTTP is surfaced on the C2 root.
Unsurfaced: crafty.${BASE_DOMAIN} (platform/crafty)
platform/crafty/httproute.yaml serves crafty.${BASE_DOMAIN} through the shared Gateway,
clusters/base/crafty.yaml reconciles it, and nothing on the C2 root links to it. R-C2-01
says the root MUST surface "the services running in the cluster and links to them", so
an unsurfaced route is a requirement gap, not a styling preference. Fixed in the next
commit, deliberately separate so the RED is in the history.
WHAT IS ASSERTED BEYOND THE PRIOR ATTEMPT (49c160d on feat/coverage-debt-paydown, whose
tree is ~107 commits behind and was NOT merged):
(0) THE INPUT SURFACE ITSELF. The forward direction is only as good as the set of
manifests it reads, so three assertions guard the guard: the scanned roots are
present, every manifest in them PARSES (an unparseable file is a service the check
cannot see), and EVERY Flux-reconciled path lies inside a scanned root — so a
future tier reconciling, say, ./services fails loudly instead of letting the
dashboard hide all of it. conformance/ is deliberately excluded from the scan: its
fixtures and goldens declare synthetic HTTPRoutes (paas-demo, gw-declared-
conformance) that are test data, not services the cluster serves. The prior
attempt excluded them only by accident, through a filename prefilter.
(5) postBuild.substituteFrom ON THE DASHBOARD'S TIER. Hostnames are ${BASE_DOMAIN}
tokens now, not the literals the prior attempt compared. Without substitution the
apex route binds the literal host `${BASE_DOMAIN}` (matching no listener) and
every tile links to a host that does not exist — while Flux still reports the
Kustomization Ready. It fails OPEN, which is this repo's most-repeated incident
shape.
(12) TOKENS, NEVER LITERALS. No href may name a literal `home.keithjackson.dev` host,
so prod's front door cannot link dev's services — the 2026-09-04 class (two
clusters authored to describe the same names, prod's external-dns then deleting
dev's records), applied to the C2.
(2) THE SECTIONNAME IS PROVEN LOAD-BEARING rather than asserted from prose: the
wildcard listeners are shown to exist AND to be distinct from the apex listener,
which is why `*.${BASE_DOMAIN}` cannot serve the bare apex.
(10) THE REVERSE DIRECTION NOW ACCEPTS L4. The UniFi tile points at :11443 on a
hostname published by apps/vms/unifi/50-service-l4.yaml (a KubeVirt VM behind a
pinned Cilium LoadBalancer VIP). That is something this repo serves, so it
resolves — no off-cluster declaration is needed for it any more. The declaration
hatch is kept and constrained (specific home-zone hostnames, may not shadow a
hostname the repo does serve) but is EMPTY today, which is the honest state.
WHOLE-REPO SCOPE, as the declaration says. The invariant quantifies over every tracked
manifest that can declare an HTTPRoute; the change that BREAKS it is a route added
somewhere else entirely (`apps/wave-1/newthing.yaml` attributes R-GITOPS-02 or nothing,
never R-C2-01), and ci/touched-requirements.sh has no path-map arm for platform/homepage
or platform/headlamp at all. Requirement-scoped selection would filter this check out of
exactly the change that introduces the bug — the R-TEST-03 blind spot. No ci/l0.sh stage
is added: Stage 3b's --include-scope union selects it, and whole-repo-invariants-wired
fails the gate if anyone hand-adds one.
HONEST GAPS, in the check's own header: it does not fetch the apex (L2, covered for the
Gateway/TLS half under R-DNS-01 / R-TLS-01); (19) reads a DECLARED auth posture because
no Gateway ext-authz attachment exists anywhere in this repo, and says so in the
assertion text; it does not judge whether a tile is useful; and an L4-ONLY endpoint
(crafty-mc, a Minecraft game port) is reported as an ADVISORY rather than required,
because a raw TCP/UDP endpoint has no URL a browser can follow.
Co-Authored-By: Claude Opus 5 <[email protected]>
INCOMPLETE — preserved by the PM after the authoring agent died to a server
error mid-task (its third attempt; the first two died to a session rate
limit). Do not merge.
R-C2-01 and R-C2-02 are the only 2 remaining coverage gaps on main (49/51).
This is the chainsaw Test declaration only: it quotes both requirements
verbatim from the spec, cites docs/12-command-center-ui.md for which surface
serves which half (Homepage at the root for R-C2-01, Headlamp for R-C2-02's
'controlled operational actions where safe', Grafana for dashboards/alerts),
and argues why the deciding half is L0 rather than L2 — reachability is
already covered by l2-behavioral/gateway-ingress-reachable, whereas whether
the root surfaces THE SERVICES THE CLUSTER RUNS is a comparison of two sets
that are both repository facts.
WHAT REMAINS: check.sh does not exist, so this test cannot run at all. Also
still open: whether 'a service the root hid' (from the prior art commit
49c160d on feat/coverage-debt-paydown) is still a live defect, whether
ci/touched-requirements.sh can attribute R-C2-01/02 at all (a digit in the
workstream segment reportedly broke the content scan), and the R-TLS-02
decision summary.
Co-Authored-By: Claude Opus 5 <[email protected]>
The existing check audited only the inverse — a ${...} undefined or
unescaped — so it passed the manifest that broke prod: Postgres
dollar-quoting under a postBuild.substituteFrom path, which envsubst halves,
delivering 'DO $' and a syntax error on every reconciled run (fixed 16b7fcd).
Measured rather than assumed: Flux 2.4.0 embeds fluxcd/pkg/envsubst (a fork
of drone/envsubst — the brief credited the wrong module), and --strict output
is BYTE-IDENTICAL to non-strict here, so strict mode is blind to this class
entirely. The decisive contrast is two commands on the same stream: kubectl
kustomize emits valid 'DO $$' — which is what the kubectl-apply
verification saw — while the same stream through flux envsubst emits 'DO $'.
357 assertions, 0 unjustified. Legitimate uses genuinely exist (18 ${} escapes
plus two reaching shell comments), so the allowlist is a reviewed opt-in
rather than a bypass: a stale entry fails, a wildcard is matched verbatim and
so fails as stale, a short context or reason is rejected, and a genuine entry
covers only the span it matches — a second occurrence in the same file still
fails. Strongest proof: restoring the real pre-16b7fcd manifest verbatim makes
the guard reject the exact commit that broke production.
Neither family was scraped at all, so every Longhorn alert (degraded,
faulted, capacity) and every cert-manager alert (expiry, issuance) was
structurally incapable of firing: storage health unmonitored on a cluster
whose node just had a failing drive, TLS expiry unmonitored with the live
wildcard cert 89 days out. Now proven on prod: longhorn_* 8717 series with
longhorn-backend up=3, certmanager_* 104 series across 3 targets.
Adds an L0 guard (118 assertions) that crosses from each alert rule to the
scrape config and re-proves each enabling declaration against the working
tree, including the negatives — cert-manager's monitor mutual-exclusion, the
Longhorn metrics NetworkPolicy, and kube_node_labels needing
metricLabelsAllowlist. Proven able to fail five ways.
Hubble is deliberately NOT enabled: it opens 9965 on every cilium-agent and
rolls the CNI DaemonSet across all three nodes, and platform/cilium carries an
explicit parity contract with the Ansible template that a HelmRelease-only
change would silently break. Registered as a declared gap instead.
Two findings this surfaced, neither fixed here and neither absorbed silently:
twelve apps/* <app>-down SLI rules discover zero targets because their
ServiceMonitors name port 'metrics' and no app Service defines that port; and
kube-state-metrics is OOMKilled roughly every ten minutes, making thirteen
kube_* rules intermittently inert.
The implementation landed in the preceding WIP commit had never been executed.
This finishes it: the arithmetic is re-measured, the guard and its allowlist are
attacked, and the header's one factual error is corrected.
WHAT ENVSUBST ACTUALLY IS. The header credited `github.com/drone/envsubst`. The
pinned flux 2.4.0 binary in the dev image embeds
`github.com/fluxcd/pkg/[email protected]` — Flux's own fork of drone/envsubst
("This package is a fork of drone/envsubst", its README) — read straight out of
the binary with `grep -ao`. Same behaviour, different module; the header now says
so and says how to check, because a claim about a dependency that nobody can
re-derive is how the last wrong assumption survived.
RE-MEASURED, NOT ASSERTED. Every row of the DOLLAR-RUN ARITHMETIC table was
re-run through `flux envsubst` (flux 2.4.0, DEFINED=VALUE) and every row holds:
`$$`->`$`, `$$$`->`$$`, `$${D}`->`${D}`, `$$${D}`->`$VALUE` (odd run: the brace
is LIVE), `$$$${D}`->`$${D}`, `DO $$ x $$;`->`DO $ x $;`, and `$$${ARR[0]}` dies
with "missing closing brace" while `$${ARR[0]}` is fine. `--strict` output was
byte-identical to non-strict, exit 0 both ways (`cmp`) — strict mode is blind to
this whole class, which is why a source assertion is the only guard available.
The header now carries the 30-second reproduction, flagged to be re-measured on
any FLUX_VERSION bump.
END TO END ON THE REAL PATH. With the pre-16b7fcd manifest restored,
`kubectl kustomize apps/wave-0` emits `DO $$` — valid SQL, and exactly what the
`kubectl apply` verification saw — while the same stream through
`flux envsubst --strict` emits `DO $` and `$;` and exits 0. That two-command
contrast IS the incident.
THE GUARD IS NOT TRIVIALLY SATISFIABLE — 16 adversarial cases, all as intended,
run against a throwaway copy of the tree so nothing could be left behind:
* an unmarked `$$` under a substituted path FAILS (DOLLAR-COLLAPSE); so do an
odd `$$${...}` (DOLLAR-HALF-ESCAPED) and an even `$$$${...}` (OVER-ESCAPED);
* a stale entry FAILS three ways — context no longer in the file, context that
only covers an already-accepted `$${` escape and so excuses nothing, and a
path that is not a file in the repo;
* the allowlist cannot blanket-exempt a path: a 2-char context is rejected
(<24), a `.*` wildcard is matched VERBATIM and therefore rejected as stale, a
24-char context carrying no dollar run is rejected, a 4-char reason is
rejected (<40) — and, the central case, a genuine entry covers only the span
its context matches, so a SECOND `$$` elsewhere in the same file still fails.
Two genuine entries for two real occurrences do pass, so the mechanism is
usable rather than merely unsatisfiable;
* the CLASS, not the instance: the pre-fix `DO $$ ... $$;` reconstructed in a
NEW ConfigMap under apps/wave-2 fails at both dollar runs, and restoring the
real pre-16b7fcd apps/wave-0/nextcloud-db.yaml verbatim fails the check at
lines 384 and 391. The guard rejects the exact commit that broke production.
The scratch fixtures lived only in the throwaway copy; `git status` on this
branch stayed clean throughout, which is the other half of the last attempt's
unfinished business.
LEGITIMATE USES EXIST AND STAY GREEN. 18 `$${...}` escapes (unifi cloud-init,
the gitea and nextcloud OIDC bootstrap scripts, two longhorn comments) pass
unremarked, and the 2 non-braced `$$` occurrences are the two allowlist entries.
Those two are shell comments inside block scalars, and `kubectl kustomize`
confirms the comment text really does survive into the stream envsubst reads —
so they are genuine, reaching content whose collapse is cosmetic, not a
hypothetical the allowlist was invented to excuse.
STILL NOT PROVEN, AND STILL NOT PROXIED: that a change was verified through a
substituting path. Which command a human ran is a fact about a shell session;
git stores the manifest, never the invocation, so it is not statically decidable
here and no honest formulation exists. Grepping commit prose for `kubectl apply`
would be satisfiable by rewording and would fire on the break-glass runbooks
whose whole content is that command. The answer is to move the proof off the
verification path and onto the source, which is what assertion (7) does.
No ci/l0.sh stage: the `conformance.home.k3s/scope: "whole-repo"` annotation is
what wires it, and whole-repo-invariants-wired fails the gate on a hand-written
stage.
Co-Authored-By: Claude Opus 5 <[email protected]>
ci/gate.sh set BLOCK=1 on a FAILED stage but only warned on a SKIPPED one, so
it printed the words 'not green' and then 'MERGE GATE: PASS', exit 0 — fail
open in the one component whose job is to fail closed. Observed live with
prom-rules, and independently with l0 when the pinned kyverno CLI was absent.
New contract: PASS/0 means every stage that was supposed to run ran and was
green; BLOCKED/1 a failure or a must-run stage that did not; UNDETERMINED/3
nothing failed but greenness was not established; NOT APPLICABLE/0 unchanged.
No knob inside gate.sh can turn UNDETERMINED into 0 — tolerating a non-answer
is the caller's decision to make and to document.
The distinction that makes it usable: a capability the environment DECLARES
absent may still pass (no cluster, named in the headline); a stage that COULD
NOT RUN may not. CI declares GATE_STRICT_SKIPS in the workflow rather than
sniffing for CI, so a missing pinned tool there is exit 1.
Carries over 47068ee's unusable-subset blocking from the unmerged
fix/touched-requirements-fail-closed, now with tests. Measured its companion
claim honestly: 159 of 613 paths (26%) are unmapped today, not 255 of 476
(54%) — 56 prose, 103 non-prose — which remains the larger fail-open and is
recommended as its own change rather than smuggled in here.
47 assertions, 32 of which fail against the old gate.sh.
An alert rule whose metric is never scraped is not a broken alert, it is an ABSENT
alert that reads as a present one. `longhorn_volume_robustness == 2` over a TSDB
with no longhorn_* series returns no vector: permanently quiet, permanently
"healthy", indistinguishable from a correct rule. It survives review (the PromQL is
right), it survives `promtool test rules` (the unit test supplies the series
itself), and it appears on no dashboard.
VERIFIED ABSENT ON PROD before this change (re-verified 2026-09-13 21:0xZ, ~15h
after the first audit, because a stale reading is not evidence):
count(longhorn_volume_robustness) => (empty) 4 rules inert
count({__name__=~"certmanager.*"}) => (empty) 3 rules inert
count(hubble_drop_total) / {__name__=~"cilium.*"} => (empty) 1 rule inert
count by (job) (up) => 20 jobs, none of them
Longhorn, cert-manager,
Cilium or Hubble
Consequence: storage health was unmonitored on the cluster whose prod-1 had just
had a failing SSD, and TLS expiry was unmonitored with the live `wildcard-home`
wildcard 89 days from expiry on an ACME DNS-01 renewal that can fail silently.
WHAT THIS CHANGES
* platform/longhorn: `metrics.serviceMonitor.enabled: true` (the supported chart
way — rendering longhorn 1.12.1 emits exactly 1 ServiceMonitor with the flag and
0 without), plus networkpolicy-metrics.yaml. The NetworkPolicy is LOAD-BEARING,
not belt-and-braces: measured with the monitor applied and no allowance, all 3
targets were DISCOVERED and every scrape failed `context deadline exceeded`
after 10.0s — a drop, not a refusal — while curling the endpoint from inside
the manager pod returned 200 in 0.031s. Longhorn 1.12.1 ships 6 NetworkPolicies
of its own, guarded on `networkPolicies.restrictInternalTraffic` (default TRUE)
and NOT on `networkPolicies.enabled`, so they are effectively unconditional and
the manager policy admits only same-namespace Longhorn components. The one-line
alternative (restrictInternalTraffic: false) was rejected: it deletes the
internal-traffic restriction across all six components to open one port to one
namespace, trading an R-SEC-02 control for an observability one when the two do
not conflict. This policy is Ingress-only, ns monitoring -> app=longhorn-manager,
TCP 9500 only; 9502/9503 (webhook, recovery backend) stay closed.
* platform/cert-manager: `prometheus.servicemonitor.enabled: true`. Chart default
already serves 9402; only the monitor was missing. No NetworkPolicy needed —
neither ns cert-manager nor ns monitoring carries one. NOT paired with
prometheus.podmonitor.enabled: the chart calls `fail` if both are set.
* platform/kubevirt: KubeVirtVMLaunchFailing queried
`kubevirt_vmi_phase_count{phase="Failed"}`, which DOES NOT EXIST on the deployed
KubeVirt — a different flavour of the same defect, with the scrape target up and
healthy. Prod has only `kubevirt_vmi_info` and `kubevirt_vmi_number_of_outdated`;
the only phase-bearing names are the phase_transition_time histograms. Corrected
to `kubevirt_vmi_info{phase="failed"}` — LOWERCASE, because KubeVirt lowercases
the value (observed `phase="running"` for the unifi VMI) and a capital-F matcher
would select nothing and be exactly as inert.
PROVEN, NOT ASSERTED. The rendered ServiceMonitors were applied to prod by hand and
the metrics observed to land, then REMOVED again so the merge's Helm upgrade creates
and owns them rather than colliding with an unowned object:
count(up{job="longhorn-backend"}) => 3 (all three managers, up==1)
count({__name__=~"longhorn.*"}) => 8717 (was empty)
count(longhorn_volume_robustness) => 44
up{namespace="cert-manager"} => 3 targets, all 1
count({__name__=~"certmanager.*"}) => 104 (was empty)
wildcard-home days to expiry => 89.2, ready_status{condition="True"}=1
All 58 alert rules in the repo were then evaluated against prod one by one: every
one returns a result without error, and exactly ONE fires.
TRIAGE OF THE ONE NEWLY-FIRING ALERT — A TRUE POSITIVE, NOT TUNED AWAY.
LonghornVolumeCapacityHigh on the Prometheus TSDB volume:
`longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes` = 0.971
(48.6 GiB allocated of 50 GiB) while the kubelet's view of the same filesystem is
only 20.4% used. Longhorn is not reclaiming freed blocks, so ~48.6 GiB x 3 replicas
is committed for ~10 GiB of real data. On a cluster whose failure mode is disk
pressure and whose etcd WAL shares a spindle with Longhorn, that is worth knowing
and was invisible until now. The threshold is left alone; the follow-up is
filesystem trim, not a bigger number.
THE GUARD (conformance/tests/l0-static/alert-metrics-have-scrape-targets)
Whole-repo L0 invariant, 118 assertions. It crosses from each alert to the scrape
config that makes its metric exist: every metric name is claimed by exactly one
declared owner, and each owner's enabling declaration is re-proven against the
working tree on every gate run, including the negative conditions that defeat an
enablement without deleting it. `up{job="X"}` is audited on its own axis, because
`up` always exists: where this repo authors a Service and a ServiceMonitor in the
same file, the monitor's endpoint port must be a port that Service defines.
It DECLARES conformance.home.k3s/scope: "whole-repo" and adds NO ci/l0.sh stage —
the whole-repo-invariants-wired meta-test fails the gate if one reappears.
PROVEN ABLE TO FAIL, five ways, each reverted:
1. an alert on a fabricated metric -> FAIL "claimed by NO entry"
2. longhorn metrics.serviceMonitor removed -> FAIL "no longer matches"
3. networkpolicy-metrics.yaml unwired -> FAIL "the monitor alone is NOT enough"
4. cert-manager podmonitor also enabled -> FAIL "FORBIDDEN pattern"
5. a register entry made stale / removed -> FAIL in both directions
TWO SHRINK-ONLY REGISTERS, because a blanket fail would make the gate red until
work that is not this change lands, and a disabled gate protects nothing:
* KNOWN_UNWIRED_METRICS (1): hubble_drop_total. See below.
* KNOWN_UNWIRED_JOBS (12): every apps/* `<app>-down` SLI rule is ALSO inert, and
this was found by the re-audit, not previously known. The rendered
ServiceMonitors name `port: metrics` and NOT ONE of the twelve app Services
defines a port of that name (zero occurrences of `name: metrics` across all
twelve manifests); prod confirms none of the twelve jobs appears in `up`. This
is the nastiest variant of the class: `up` exists in abundance so nothing looks
wrong, and PrometheusTargetsMissing cannot see it either, because a target that
was never discovered has no up series to be 0. Closing it needs an exporter per
app (Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr speak no
Prometheus) — except keycloak, where the live jobs are keycloak-service /
keycloak-discovery and the rule is a rename away from being live.
NOT DONE HERE, DELIBERATELY: Hubble. hubble.metrics.enabled RECONFIGURES THE CNI —
it opens 9965 on every cilium-agent and rolls the DaemonSet across all three nodes,
serialised by this repo's own maxUnavailable: 1 (set after the 2026-09-01
k8sServiceHost outage, where a CNI roll deadlocked 2 of 3 agents). It also needs a
matching edit to roles/cilium/templates/cilium-values.yaml.j2 in the same commit,
because platform/cilium/helmrelease.yaml carries a [parity] contract against it and
a rebuilt node would otherwise bootstrap a CNI config that disagrees with the
reconciled one. Recommended as its own change on the cilium Kustomization, with the
Ansible parity edit, a narrow labelsContext, and a one-node canary — not smuggled in
here.
ALSO FOUND, NOT FIXED HERE: kube-state-metrics is OOMKilled every ~10 minutes on
prod (90 restarts in 16h against a 128Mi limit) while its logs show the
custom-resource-state factory reloading every 3 seconds. Thirteen alert rules read
kube_* metrics, so those rules are INTERMITTENTLY inert — the same defect class,
arriving through a crashloop rather than a missing monitor. Needs its own change.
Co-Authored-By: Claude Opus 5 <[email protected]>
Completes the WIP preserved in the previous commit. `MERGE GATE: PASS` + exit 0 now means
what it says: every stage that was SUPPOSED to run ran, and was green.
THE DEFECT. In gate.sh's decision section a FAILED stage set BLOCK=1 but a SKIPPED stage only
warned, so the gate printed `l0: skipped (unwired seam / missing tool) — not green` and then
`MERGE GATE: PASS`, exit 0. Seen twice for real: `prom-rules` skipped because promtool was
absent, and `l0` skipped because the pinned kyverno CLI was. Exit 0 is what CI and every
`||`-chained script read as merge-approved, so the one component whose entire job is to fail
closed was failing OPEN — in a repo whose stated principle is that SKIPPED IS NOT GREEN, and
whose history already contains the 2026-08-31 "local false green" that UNDECIDED was added to
prevent. The offline stages simply never used that vocabulary.
A SKIP IS NOT ONE THING — the per-stage rule, with the justification in the file header:
* lint / l0 / render-golden / l1 — COULD NOT RUN (a pinned tool absent, a seam unwired).
BLOCKED under strict skips (a CI runner has the toolchain installed for it by
install-tools.sh, so a skip there is a defect); UNDETERMINED (exit 3) otherwise — a
workstation missing a tool is a real local condition, still non-zero, still never a pass.
* spec-immutability — the spec repo is not checked out, so R-TEST-06 COULD NOT BE VERIFIED:
UNDETERMINED. Deliberately NOT upgraded to a block by strictness, because the live runner
does not check the spec repo out at all and blocking would leave the only enforced status
check permanently red for a reason no PR can clear. It is the ONE skip a caller may
knowingly tolerate; REQUIRE_SPEC_VERIFICATION=true makes it hard once wired.
* conformance L2/L3 — CLUSTER_AVAILABLE!=true is the environment DECLARING the capability
absent, not a stage failing to run where it could have. A declared-inapplicable stage was
never "supposed to run", so it does not deny a PASS — but it is now NAMED in the PASS
headline and the step summary, so PASS never implies more than was verified. A skip
DESPITE CLUSTER_AVAILABLE=true still blocks.
declared-not-applicable may pass; could-not-run may not.
STRICTNESS IS DECLARED, NOT SNIFFED. .gitea/workflows/gate.yml sets GATE_STRICT_SKIPS=true on
the gate step — the workflow is the thing that knows it is CI — with GITHUB_ACTIONS/CI (set by
the runner itself) only as the `auto` fallback. No hostname/path/tool-availability heuristic.
A wrong answer in either direction cannot fail open: both branches are non-zero and neither is
a PASS. That step now calls ./ci/gate.sh directly and tolerates exit 3 in one visible line,
because under strict skips exit 3 can ONLY mean the unverifiable spec repo — a missing tool is
exit 1 and untolerated. Tolerating a non-answer is a reviewed choice in the CALLER; there is
deliberately no knob inside gate.sh that turns UNDETERMINED into exit 0.
ALSO, from the unlanded fix/touched-requirements-fail-closed (47068ee), whose reasoning is
carried over rather than its tree: gate.sh special-cased only exit 3 from
touched-requirements.sh, so ANY other non-zero exit left the touched set empty and printed PASS
having applied no coverage gate at all. An unusable subset is not an empty subset: it BLOCKS.
Latent today (the script exits only 0 or 3) and wired now so the hole is already closed when a
fail-closed path map makes exit 1 reachable — the measured state of that map is in the PM report.
TESTED — ci/tests/gate-decision.test.sh, 47 assertions over the whole decision table: the real
gate.sh and lib.sh run in a throwaway CI_DIR against STUB stage scripts whose exit codes the
suite dictates, so it asserts how the gate REASONS, with no tools, no cluster, no network. All
stages green -> PASS/0; each stage FAILED -> BLOCKED/1; each stage SKIPPED -> never PASS and
non-zero; strict skips -> BLOCKED/1; the spec exception; the declared-gap PASS; the
undetermined-touched-subset paths unchanged (on-base -> NOT APPLICABLE/0, no-base and
diff-failed -> BLOCKED/1); UNDETERMINED outranking NOT APPLICABLE; an unusable subset ->
BLOCKED/1; truthy()'s misspelling direction. 32 of the 47 FAIL against the pre-change gate,
including "l0 SKIPPED: printed 'MERGE GATE: PASS'" — the suite is falsifiable, not decorative.
lib.sh: truthy() is now the one spelling of "is this switch on" and cluster_available() uses it
(the comment claimed that; now it is true), and load_touched_requirements' rc!=0 branch says
what actually happened instead of "proceeding with the full layer".
Refs R-TEST-02, R-TEST-03, R-TEST-06.
Co-Authored-By: Claude Opus 5 <[email protected]>
INCOMPLETE — preserved by the PM after the authoring agent hit the session
rate limit mid-task. Do not merge without finishing and gating it.
Closes the inverse of what this check already covered. It asserted that every
${...} in a substituted path is defined or escaped; it did NOT catch content
containing a literal doubled dollar that the author needs delivered verbatim
and that envsubst silently halves. That shipped a broken CronJob (fixed in
16b7fcd): Postgres dollar-quoting became a single dollar and every
Flux-reconciled run died with a syntax error, while passing review because
verification used kubectl apply, which runs no envsubst.
Introduces a dollar-collapse.allow allowlist so each legitimate use is a
reviewed decision rather than an accident.
WHAT REMAINS: the agent's own next step was proving it catches the CLASS and
not just this instance, via a scratch manifest under a substituted path. Its
scratch fixture (apps/wave-2/zz-scratch-class.yaml and the wave-2
kustomization edit) was deliberately temporary and has been reverted here, so
the fail-proof is NOT yet demonstrated. Also unanswered: brief point 3, on
whether the check can say anything about a manifest verified only through a
non-substituting path.
Co-Authored-By: Claude Opus 5 <[email protected]>
INCOMPLETE — preserved by the PM after the authoring agent hit the session
rate limit mid-task. Do not merge without finishing and gating it.
Problem: in ci/gate.sh a FAILED stage sets BLOCK=1 but a SKIPPED stage only
warns, so the gate prints the words 'not green' and then concludes
'MERGE GATE: PASS' and exits 0. Observed live tonight with prom-rules, and
independently with l0 when the pinned kyverno CLI was absent. That is
fail-open in the one component whose whole job is to fail closed.
The agent was mid-way through adding truthy/strict_skips helpers to ci/lib.sh.
WHAT REMAINS:
- Per-stage justification for which skips must BLOCK and which may degrade
to a non-zero UNDECIDED (spec-immutability and L2/L3 skip for legitimate
environmental reasons and must stay usable locally).
- Whether CI and local can be distinguished reliably here.
- New ci/tests/ suite for the decision logic, and a before/after proof.
- Reconciling with the unmerged branch fix/touched-requirements-fail-closed,
whose 47068ee already makes an unusable touched subset block, and whose
56c242d claims 255 of 476 paths were unmapped in the path->requirement
map — a far larger fail-open that still needs measuring on current main.
Co-Authored-By: Claude Opus 5 <[email protected]>
INCOMPLETE — preserved by the PM after the authoring agent hit the session
rate limit mid-task. Do not merge without finishing and gating it.
Problem: qBittorrent 5.1.2 has no WebUI\Username or WebUI\Password_PBKDF2 in
its config, so it mints a NEW temporary admin password on every process start
and announces it only on stdout. Two consequences, both real: the Sonarr and
Radarr download clients configured earlier will silently stop working at the
next pod restart, and a human operator has no stable login either.
Adds a SOPS-encrypted qbittorrent-bootstrap Secret (verified encrypted: both
values are ENC[AES256_GCM]) plus the catalog/rendered wiring to consume it.
WHAT REMAINS:
- The agent's last note: the postStart hook works, but its output never
reaches kubectl logs, so it was switching to the marker-file logging
pattern gitea's hook already uses. That change is NOT here.
- The Secret exists on prod as kubectl-client-side-apply, i.e. live-only;
merging adopts it into git.
- Sonarr's and Radarr's stored download-client passwords were to be updated
to the durable value — NOT done, so the chain may break on restart until
it is.
- The required proof is unfinished: restart the pod, then show the WebUI
accepts the credential AND both arrs still report isValid: true.
Co-Authored-By: Claude Opus 5 <[email protected]>
kube-prometheus-stack does not scrape flux-system: the controllers are
bootstrapped, expose no Service, and no chart value covers them. So every
gotk_* alert rule in platform/monitoring/alerts.yaml could never fire —
verified on prod, count({__name__=~"gotk.*"}) returned nothing. A repo whose
entire delivery model is Flux had no alerting on reconciliation failing, a
Kustomization erroring, or a resource left suspended.
Proven live before landing: with this PodMonitor applied, gotk_suspend_status
returns 39 series and gotk_reconcile_condition 117, and FluxResourceSuspended
actually fired on a deliberate test suspension — evidence the rule is live,
not merely present.
Extracted from the wider inert-alert audit on
fix/scrape-targets-for-inert-alerts, which stalled unfinished. This is the
self-contained, verified half; Longhorn and cert-manager metrics are still
ABSENT and their alerts remain inert. It also adopts the object into git: it
existed on prod only as kubectl-client-side-apply, so a rebuild would have
lost it.
Co-Authored-By: Claude Opus 5 <[email protected]>
The SQL asserted convergence in a dollar-quoted PL/pgSQL DO block. That
cannot survive this delivery path: the ConfigMap is rendered through Flux's
postBuild substitution, and envsubst collapses a doubled dollar into one, so
Postgres received a half-quoted DO and failed with a syntax error. Three
scheduled runs had left nine Error pods in nextcloud-db.
It passed verification because it was hand-applied with kubectl, which runs
no envsubst — the defect only existed once Flux owned the object. The GRANT
on the first line still succeeded, so the privilege held and the blind spot
stayed closed; what was broken was the assertion meant to prove it, which is
the part that makes a silent regression visible.
Rewritten in plain SQL with no dollar sign anywhere in the file, comments
included. Escaping by doubling again would also work and would be the next
person's trap. has_database_privilege()::int is 1 or 0, so the division
raises division_by_zero and ON_ERROR_STOP=1 fails the Job; the divisor is
not constant so it cannot be folded away at plan time.
Co-Authored-By: Claude Opus 5 <[email protected]>
It had been permanently SKIPPED because it expected bootstrap/standup.sh,
which never landed on main — planned, never merged, still living only on the
unmerged prod-standup branches along with the 2026-09-05 incident it came
from. Dropping the real script in and running the old check produced FAIL
2/34: it would have false-blocked the very merge that landed its own input.
Now split into the half that needs no guard (no overlay ships a
placeholder-shaped effective cluster-vars value; an empty value must be
STATED by the overlay, not inherited in silence; a loud base sentinel must
be explicitly overridden) and the half that self-activates. The guard is
discovered by CONTENT — any tracked *.sh defining <name>_is_placeholder() —
so landing one anywhere activates the second half with no edit here, and two
definitions is a failure. The check never exits 3 again.
Unscoped L0 goes 13 PASS / 1 SKIP -> 14 PASS / 0 SKIP.
All 46 runbook_url annotations pointed at https://<apex>/runbooks/<name>
and every one returned 404 — there was no runbooks/ directory and no route.
Every alert that reached a human handed them a dead link at the moment they
needed it.
Runbooks are now served from their own non-gating tier at the apex prefix,
PUBLIC on purpose: gating them behind Keycloak would make keycloak-down
unreadable exactly when Keycloak is down, which is a loop with no exit. The
new L0 guard checks both directions — a runbook_url with no file, and a file
not wired into the generator — and it caught a live one on contact with
main: CNPGPrimariesCoLocated had shipped hours earlier pointing at a runbook
that did not exist.
324 lines of mutation-checked promtool tests had landed with nothing to
execute them: promtool was not in the pinned toolchain, so the coverage was
present and enforced by nothing — the failure mode this repo keeps hitting.
promtool 3.14.0 is now pinned in both versions.env files (matching prod's
Prometheus, verified against the running image) and baked into the
container. The two competing harnesses are consolidated into one, and the
two competing wirings into one: the bespoke ci/prom-rules.sh stage is
deleted in favour of a scope-declaring conformance test, because a
hand-written stage running one hardcoded script is exactly the pattern
b9f8807 just removed. Missing promtool is exit 3 — UNWIRED, never green.
51 of 58 alert rules still have no unit test; that gap is a shrink-only
ratchet that fails if it grows, rather than a blanket FAIL that would get
disabled or a report nobody reads.
`conformance/tests/l0-static/placeholder-guard-coupling` (R-ENV-02, R-STATE-03) exited 3
SKIPPED on every gate run since it landed, because its check keyed its whole existence on
`bootstrap/standup.sh`. Measured, not assumed: that file has NEVER been on `main` — there is
no deletion commit for it anywhere; it landed only on unmerged prod-standup branches
(`work/prod-standup-campaign`, `prod/*`, `feat/prod-hardening`). It was planned and never
merged, not removed. SKIPPED is not green, and a permanently-skipped incident-derived test is
the quiet version of the failure it was written to prevent.
The skip was also hiding a RED. Run against the real `standup.sh` from
`work/prod-standup-campaign`, the old check FAILED 2 of 34 assertions — false positives from
treating the empty string as placeholder-shaped for knobs cluster-vars authors empty on
purpose (`BACKUP_CREDENTIALS_SUFFIX`, `VELERO_BACKUP_PREFIX`, both `""` on dev and both
documented there as load-bearing). It would have false-blocked the merge landing its own input.
NOT deleted — the hazard is not gone. Split into two halves, one of which needs no guard:
HALF A (always runs): no environment overlay ships a placeholder-shaped EFFECTIVE
cluster-vars value; an empty effective value is STATED by the overlay rather than inherited
in silence; a LOUD base sentinel is explicitly overridden by every overlay. This is the
2026-09-05 outcome asserted at the source instead of at the guard — if prod's
S3_BACKUP_ENDPOINT is never a placeholder, velero cannot be pointed at a hostname that does
not exist. Input surface is `clusters/`, all of it on `main`.
HALF B (self-activating): the real predicate extracted and executed against those same
values, the consumer trace, and the mid-string anchoring property.
The check NEVER exits 3 again, and the guard is DISCOVERED by content rather than hardcoded:
any tracked `*.sh` defining `<name>_is_placeholder()` is found, so a guard landing at
`bootstrap/standup.sh` — or moved into a sourceable library, the remedy the check already
recommends — activates HALF B with no edit to the test. Two files defining it is a FAIL. When
no guard exists the gap is printed loudly every run instead of swallowed as a skip.
LOUD vs EMPTY is the correction that makes it honest in both modes: a loud placeholder is
always a defect in an overlay, while `""` is held to the explicitness rule instead — and still
counts as a placeholder for a knob a guard covers, because a `""` case-arm is the guard
author's declaration that empty is not a usable target.
Proof, all reverted: 84 assertions and exit 0 on `main`; prod S3_BACKUP_ENDPOINT set to the
literal `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` FAILS (A1); deleting dev's
explicit VELERO_BACKUP_PREFIX op so it inherits `""` FAILS (A2); a LOUD base BULK_NFS_SERVER
with prod's override deleted FAILS 2/86 (A1+A3); and with the real `standup.sh` present,
reverting `s3_is_placeholder` to the `TBD-*` PREFIX glob FAILS 12/110 (B3 anchoring). With the
pristine `standup.sh` the check PASSES at 110 assertions where the old one failed 2 of 34.
L0 layer: 14 checks, 14 PASS, 0 SKIP, 0 FAIL (was 13 PASS / 1 SKIP). Coverage unchanged —
R-ENV-02 and R-STATE-03 both still list this test, 49/51 with the same 2 gaps.
Also:
* `ci/touched-requirements.sh` — `bootstrap/*` -> R-STATE-03 R-GITOPS-01 is KEPT, its stale
comment corrected. It claimed standup.sh "carries the placeholder guards", describing a
file that is not here; R-STATE-03 there is ANTICIPATORY. Deleting the entry would leave the
change that lands a placeholder guard selected by nothing.
* `ci/tests/touched-requirements.test.sh` — new path-map section (12 assertions, was 8)
locking `bootstrap/*` and `clusters/*`, so removing either entry is a test failure rather
than a silent loss of selection. All three ci/tests pass (12 + 31 + 15).
* `docs/BULK-NFS-STORAGE.md` §7 — its "nothing in the gate currently refuses a
placeholder-shaped overlay value" is no longer true; corrected.
* Scope re-audited: still deliberately requirement-scoped, not whole-repo. Every path that
can break it attributes a requirement it declares.
Co-Authored-By: Claude Opus 5 <[email protected]>
The serving route's comment claimed that because no Gateway ext-authz
attachment exists, every forward-auth route is 'in practice reachable
unauthenticated today'. That is wrong, and wrong in a dangerous direction
for a document an operator reads during an incident.
Enforcement here is PER-POD, not gateway-level: each of the five services
declaring auth: forward-auth (pihole, jackett, qbittorrent, radarr, sonarr)
publishes both its app port and 4180 on its Service, and its HTTPRoute
targets :4180 — the oauth2-proxy sidecar. Verified on prod. The app port
remains reachable in-cluster on purpose, which is how sonarr/radarr drive
qBittorrent's API, and Cilium default-deny governs that path.
The public-by-design decision for this route is unchanged and still correct;
what changes is that it is genuinely the exception rather than one open route
among many. Headlamp's exclusion is likewise deliberate (native OIDC with
per-user id_token forwarding and no ServiceAccount cluster rights).
Co-Authored-By: Claude Opus 5 <[email protected]>
The CNPG alert unit tests (324 lines, mutation-checked) were run by NOTHING. promtool was
not in the baked toolchain, no gate stage evaluated any *.promtool-test.yaml, and the only
runner `curl | tar -xz`'d a promtool release from GitHub at TEST-RUN time if one was
missing. Hours later a second, independent harness landed for the Velero rules
(platform/monitoring/tests/) with its own fallback — a `podman run` of
quay.io/prometheus/prometheus. So the repo had two harnesses, two acquisition paths, and
coverage that looked present and was enforced nowhere: the exact failure mode that had just
produced a permanently-firing CNPG alert and a structurally inert Velero alert.
PROMTOOL IS NOW PINNED LIKE EVERY OTHER TOOL. PROMTOOL_VERSION=3.14.0 in
infra/devcontainer/versions.env and ci/versions.env (the two stay in lockstep, per the
repo's rule), baked by the Containerfile from the prometheus release tarball (promtool ships
inside it), recorded in /etc/dev-toolchain-versions, passed by dev.sh, and installed for the
l0 stage by .gitea/workflows/scripts/install-tools.sh so the CI runner is not permanently
SKIPPED. The pin is not free-floating: promtool IS the rule engine, so it tracks the
Prometheus prod actually runs — verified against the live cluster, not assumed:
kubectl --context k3s-prod get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'
-> v3.14.0-distroless (image quay.io/prometheus/prometheus:v3.14.0-distroless)
ONE HARNESS. platform/cnpg/run-alert-tests.sh is folded into
platform/monitoring/tests/run.sh, which now covers every PrometheusRule in the repo that has
tests, via two conventions it discovers automatically: `<manifest>.promtool-test.yaml`
beside its manifest (cnpg), and `<component>/tests/*.test.yaml` over that component's
manifests (monitoring). It writes the flattened rules under whatever name each test file's
own `rule_files:` declares, so no test file needs to know the harness exists, and it
`promtool check rules` first — a rules file that does not parse would make every unit test
pass vacuously by finding no rules at all. The cnpg script's reasoning is preserved: why a
CR must be flattened (`.spec.groups`, never re-typed), and why generated files go to a temp
dir. Its `${BASE_DOMAIN}` substitution is dropped in favour of extract_rules.py's verbatim
placeholders, with the trade-off recorded there — the hazard it guarded against is addressed
by the temp dir, and verbatim leaves one transformation fewer between the shipped rule and
the rule under test.
NO ENGINE IS ACQUIRED AT RUN TIME any more — not the GitHub release, not the container
image. Not because a pinned download is unusual here (the Containerfile downloads every tool
it bakes) but because those two decided the version, the network and the machine SILENTLY,
at test time, on whatever host ran the tests; and each quietly turned "the toolchain has no
promtool" into a green run, which is how the toolchain stayed without one. Missing promtool
is exit 3 (UNWIRED, never green); a PROMTOOL= that does not resolve is exit 2.
ONE WIRING: conformance/tests/l0-static/alert-rules-unit-tested, NOT a gate stage. The
`prom-rules` stage (ci/prom-rules.sh + its Makefile target + ci/gate.sh entry) is deleted:
it is the bespoke-per-check pattern b9f8807 had just removed from ci/l0.sh, and the
declarative replacement gives what a hand-written stage cannot — selection by
`conformance.home.k3s/scope: "whole-repo"` (asserted: the test is selected even under a
requirement filter matching nothing), requirement-attributed JUnit/coverage/ledger evidence,
and run_l1_offline.py's rule that a SKIP of a scope-declaring check is the runner's rc 3.
The scope declaration is load-bearing here: touched-requirements.sh keys apps/* to nothing
and platform/monitoring/* to R-OBS-01/02, never R-OBS-04, so a new untested PrometheusRule
would otherwise be gated by nothing.
The check asserts the pins and their lockstep, that the image bakes and records promtool,
that the promtool IN USE reports exactly the pin (a FAILURE, not a skip: the wrong engine
still prints SUCCESS), that exactly ONE harness exists and none fetches an engine, that
every rule-unit test file passes, and — on every run — that the harness CAN fail, by putting
a synthetic rule through it twice with one expectation mutated.
MISSING TESTS ARE A RATCHET, deliberately, and the reasoning is in the check's header. 51 of
58 alert rules have no unit test. Failing on all of them makes the gate red until ~51 tests
exist, so it gets disabled and protects nothing; merely reporting is exactly how the CNPG
tests came to be enforced by nothing. So the gap is DECLARED in the check as a shrink-only
register, printed with counts every run (13 alert files nothing tests at all; 35 named
alerts inside covered files that nothing asserts on), and it cannot grow: a new
PrometheusRule, or a new alert in an already-covered file, FAILS without a test, and a stale
entry FAILS too. A missing test is never an exit-3 skip — SKIPPED is reserved for the one
genuine environment gap, promtool being absent.
PROVEN, in the rebuilt image (promtool 3.14.0):
* both suites pass through the single harness: cnpg 6 rules, monitoring 36 rules, SUCCESS.
* the check passes: 95 assertions, 7/58 alert rules unit-tested, 13 declared gap files.
* NOT VACUOUS, twice over. Shortening CNPGPrimariesCoLocated's `for: 30m` to 5m turns it
red (pending expected at 25m/29m, got firing), reverted. Adding an untested
PrometheusRule (platform/kubevirt/probe-alerts.yaml) turns it red on the ratchet,
removed. The harness's own mutation self-test runs on every invocation.
* the scope mechanism selects it: `run.sh --layer l0 --requirements r-nosuchreq-99
--include-scope whole-repo --print-selection` lists alert-rules-unit-tested.
Co-Authored-By: Claude Opus 5 <[email protected]>
Investigating the path disagreement found the larger defect: sonarr and
radarr each returned [] from /api/v3/downloadclient — Completed Download
Handling had never been wired up, not merely broken. Both now have
qBittorrent configured plus a remote path mapping /downloads/ ->
/media/downloads/, proven with a real completed torrent and a negative
control: same torrent, same client, mapping absent gave an unresolvable
/downloads/... and mapping present resolved the real file at inode 36144.
This state lives in each arr's database, NOT in git — a rebuild from git
does not restore it. Recorded in DECISIONS-LOG with the recommendation to
close it properly via path agreement instead.
Three corrections after rebasing onto main.
1. THE GUARD CAUGHT A REAL ONE ON ITS FIRST CONTACT WITH main. main had gained
`CNPGPrimariesCoLocated` (platform/cnpg/alerts.yaml) pointing at
/runbooks/cnpg-primary-spread, with no runbook. The check failed the rebase exactly as
designed — 46 annotations, 35 names, 34 present — naming both the missing file AND the
missing configMapGenerator entry. runbooks/cnpg-primary-spread.md now exists: what
co-located primaries actually cost here (blast radius, not an outage, because
instances: 3 + required anti-affinity), why the drift is uncorrectable-by-declaration
(CNPG has no primary-placement field), the readiness gate that keeps the rule quiet
during a real node outage, and the ~9s-per-database switchover rebalance, one cluster
at a time. The 2026-09-07 outage in which all three primaries sat on one node is the
reason the alert exists at all.
2. WHOLE-REPO SCOPE, NOT A HAND-WRITTEN STAGE. main's refactor replaced ci/l0.sh's
per-check "UNCONDITIONAL" stages with a declaration:
`conformance.home.k3s/scope: "whole-repo"`, unioned into every gate run by Stage 3b's
`--include-scope`. My +38-line Stage 3f is removed (ci/l0.sh is now byte-identical to
main) and the annotation is declared instead. Assertion (7) of the check was rewritten
to match: it no longer asserts its own path appears in ci/l0.sh — it asserts the test
DECLARES the scope and that ci/l0.sh still passes --include-scope, since neither half
alone makes the check load-bearing. That matters here because
ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02", not
R-OBS-04, and maps platform/kubevirt/* and runbooks/* to nothing at all.
3. A FACTUAL CORRECTION IN ALL 35 RUNBOOKS. Every page claimed `dev.sh` defaults its
current-context to k3s-dev. main's `9b4744d fix(devcontainer): dev.sh defaults to
prod` made that false, and a runbook that misstates which cluster a command talks to is
worse than one that says nothing. The pages now say what is durably true: the default
has already changed once, so naming `--context k3s-prod` is what makes a command mean
the same thing next month, and the per-invocation banner is there to be read. The
check's own failure message was corrected the same way.
Re-verified after all three: check green at 374 assertions (46 annotations, 35/35 runbooks
present, 35/35 projected), re-applied to prod, and re-probed in-cluster through the
Gateway — node-down, keycloak-down, keycloak-client-secrets, cnpg-archiving,
longhorn-faulted, vpn-down, node-io and cnpg-primary-spread all HTTP 200 with
text/plain; charset=utf-8 and 8.9-12.1 KB of real body; /runbooks/ index 200; an unknown
name 404s with a pointer rather than a blank page; and homepage still answers the apex `/`,
confirming Gateway API longest-path precedence.
Full merge gate: PASS (spec-immutability, lint, l0, render-golden, l1 all green;
prom-rules SKIPPED for want of promtool in the image; L2/L3 SKIPPED, no cluster).
Co-Authored-By: Claude Opus 5 <[email protected]>
qBittorrent reports finished releases at /downloads/<name>; sonarr and radarr
see the same inodes at /media/downloads/<name>, so Completed Download Handling
could never resolve what the client handed it. Configured, on prod, via each
app's own HTTP API:
* download client qBittorrent -> qbittorrent.qbittorrent.svc.cluster.local:8080
(it did not exist in EITHER arr: both /api/v3/downloadclient returned [])
* Remote Path Mapping /downloads/ -> /media/downloads/ for that host
Both built from each app's /downloadclient/schema, matched on identity before
writing, and re-running creates nothing. downloadclient/testall reports
isValid=true in both. Proven with a negative control: Sonarr's queue showed
outputPath=/downloads/... with the mapping deleted and /media/downloads/...
with it present, for the same completed torrent.
This state lives in sonarr.db / radarr.db on the config PVCs, NOT in git — a
rebuild from git alone does not restore it. Recorded as a wart, with path
agreement (repointing SavePath to /media/downloads) recommended as the durable
fix and put to the operator rather than taken unilaterally.
Co-Authored-By: Claude Opus 5 <[email protected]>
Every platform-health alert rule carries
`runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations over 34
distinct names, across platform/monitoring/alerts.yaml,
platform/monitoring/pvc-capacity-alerts.yaml, platform/cnpg/alerts.yaml and
platform/kubevirt/alerts.yaml. There was no runbooks/ directory and no HTTPRoute
serving that path: an in-cluster probe of
https://home.keithjackson.dev/runbooks/node-down returned HTTP 404. Every alert that
reached a human handed them a dead link at the exact moment the documentation is read,
and the annotation made it look solved — the same green-surface-over-absent-mechanism
class as the import Job that reported success while storing placeholder secrets.
No alerts.yaml is touched and no runbook_url value changes. The existing URL scheme is
made to resolve instead.
WHAT LANDS
- runbooks/<34>.md — one per name, each grounded in this cluster's real topology and,
where one exists, its real incident history: the 2026-09-07 k3s-prod-1 etcd I/O
starvation (measured PSI 11.36 vs 0.75/1.29, the exact journal lines, the four
mechanisms proven inert), the Longhorn reboot tax and the freeze/thaw of
concurrent-replica-rebuild-per-node-limit, Cilium's once-resolved egress interface
cache (Ifindex 3 against a live 1780) and the absence of OSS failover, the CNPG
38-hour ContinuousArchiving=False with its measured 6.8 MiB/h disk-full clock and the
2026-09-12 aggregation-scope fix, the 2026-09-06 Keycloak lockout in which all 12
clients held literal placeholder secrets behind an entirely green cluster, the Velero
R2 x-amz-tagging NotImplemented incompatibility and its v1.14.3-rc.2 pin, and swap
re-enabling itself across reboots via systemd-gpt-auto-generator.
Each page states what the alert means HERE, the first diagnostic commands, likely
causes ordered by probability with a remediation each, and when to escalate or stop.
- Every page uses FULLY-QUALIFIED resource names and says why: three CRDs register the
kind `Backup` here, the bare short name resolves to Longhorn's (zero objects), so
`kubectl get backup -A` prints "No resources found" on a cluster holding 6 Velero and
27 CNPG backups. Acting on that false negative is how a CLI quirk becomes data loss.
- Every page states that kubectl only works via
`infra/devcontainer/dev.sh run bash -c '...'` with an explicit `--context k3s-prod`,
because dev.sh defaults its current-context to k3s-dev.
- Pages also record which alert groups are INERT today (measured 2026-09-13: no
Longhorn, cert-manager, Flux-controller or Hubble scrape job exists, so 13 rules
evaluate against nothing) so nobody reads their silence as health.
SERVING — runbooks/serving/ plus clusters/base/runbooks.yaml
A two-replica nginx-unprivileged serving a ConfigMap generated from runbooks/*.md, at
the apex `/runbooks` prefix, in its own non-gating post-platform Flux tier
(dependsOn platform-config, wait:false) — the platform/homepage pattern, file for file.
- The service catalog cannot express this route: ingressRoutes[].host is pattern-bound
to a single DNS label plus the apex, so the bare apex is unrepresentable, and there is
no path field.
- Rejected: redirecting to the Gitea web UI. A runbook is read when the platform is
broken, and Gitea is an app-tier workload whose own dependencies several of these
runbooks document; Gateway API's RequestRedirect also has no "append .md" modifier,
so it would need 34 hand-maintained ReplaceFullPath rules.
- configMapGenerator maps each file to an EXTENSIONLESS key (node-down=node-down.md),
so /runbooks/node-down is a plain static lookup with no rewrite rule. Served as
text/plain: the namespace has DNS-only egress, so a CDN markdown renderer could not
load anyway.
- The tier declares NO postBuild.substituteFrom, deliberately: envsubst would run over
the prose and silently empty every ${...} shell example (it fails open — the 2026-09-01
emptied Cilium k8sServiceHost). The HTTPRoute omits `hostnames:` and inherits the
https-apex listener's, so nothing needs substituting.
AUTH — PUBLIC, deliberately
auth-mode=none. Gating these pages behind oauth2-proxy/Keycloak would make
runbooks/keycloak-down unreadable when Keycloak is down and
runbooks/keycloak-client-secrets unreadable during the exact 2026-09-06 failure mode.
An operator who cannot log in is this route's primary reader. Content is operational
procedure only, no secrets. The rationale is recorded in an annotation on the HTTPRoute
so the route is left OUT of the Gateway ext-authz attachment when forward-auth
enforcement lands.
PROVEN — in-cluster probe through the Gateway on prod
/runbooks/node-down HTTP 200 8613 bytes text/plain; charset=utf-8
/runbooks/keycloak-down HTTP 200 9378 bytes text/plain; charset=utf-8
/runbooks/cnpg-archiving HTTP 200 11729 bytes text/plain; charset=utf-8
/runbooks/longhorn-faulted HTTP 200 9135 bytes text/plain; charset=utf-8
/runbooks/vpn-down HTTP 200 11092 bytes text/plain; charset=utf-8
/runbooks/backup-failure HTTP 200 9604 bytes text/plain; charset=utf-8
HTTPRoute Accepted by io.cilium/gateway-controller on sectionName https-apex; the
`/runbooks` prefix wins over homepage's apex `/` by Gateway API longest-path precedence.
GUARD — conformance/tests/l0-static/alert-runbooks-resolvable (R-OBS-04)
363 offline assertions. It checks BOTH halves, because the regression is silent in both
directions: an alert with a new runbook_url and no runbook links to a 404, AND a runbook
committed without its configMapGenerator entry is in git, reviewed, merged, and still
404s in the cluster. Also asserts a quality floor so a stub cannot make the link look
fixed, and that route/listener/nginx prefix agree with the annotations.
Proven to fail, then reverted: a bogus runbook_url (2 FAIL), a runbook removed from the
generator (1 FAIL), and a 31-byte stub (7 FAIL).
Co-Authored-By: Claude Opus 5 <[email protected]>
VeleroNoRecentBackup could not see the failure that actually happened:
velero-daily sat paused having never produced a successful backup, so
velero_backup_last_successful_timestamp{schedule=...} did not exist and
time() - <absent series> yields no series. Prod had no volume backups and
nothing alerted, further masked by a resources-only schedule succeeding
daily. No velero_* metric describes a Schedule OBJECT, proven by scraping a
throwaway paused Schedule and finding zero lines, so this exposes the CRs
via kube-state-metrics customresource_state and keys on
defaultVolumesToFsBackup — the property, not a name list. Adds the repo's
first prometheus rule-unit-test harness, mutation-checked both ways.
The gate had been accreting one bespoke UNCONDITIONAL stage per whole-repo
L0 test (3c/3d/3e), each repeating the same ~12-line rationale and each one
something the next author had to remember to write. Forget it and the test
runs only when some unrelated requirement happens to be touched.
Tests now declare conformance.home.k3s/scope: whole-repo and run.sh unions
them in via --include-scope; the three stages are deleted. 10 of 13 tests
are whole-repo, 3 stay path-scoped with the path map widened instead
(infra/* -> R-ENV-01, ci/* -> R-TEST-06). A new meta-test makes the bespoke
pattern fail the gate if it creeps back, and a change under
conformance/tests/<test>/ now attributes that test's own requirements, so
weakening a test is gated by what it certifies.
rc-3 (SKIPPED is UNWIRED, not green) is preserved and demonstrated; the
incident narratives moved into the tests they describe rather than being
deleted. Gate cost for a scoped change rises 20.7s -> 41.8s, a ceiling any
empty-touched-set change already paid.
VeleroNoRecentBackup could not fire for the failure mode that actually happened.
`(time() - velero_backup_last_successful_timestamp{schedule!=""}) > 24*3600` is the
right SHAPE — per-schedule, un-aggregated — but Velero only creates that series once
a backup for the schedule SUCCEEDS, and `time() - <absent series>` yields no series.
So it detects "backed up once, then stopped" and is structurally INERT on "never
backed up". Prod ran with `velero-daily` (the schedule carrying VOLUME DATA) paused
and never-successful: no volume backups at all, nothing alerted, and it was masked
because `velero-daily-resources-only` completed cleanly every 03:00, so every other
backup signal read green while only Kubernetes objects were protected.
A PromQL-only fix is impossible, verified rather than assumed: velero v1.18.1 exports
35 velero_* metric names and every one carrying a `schedule` label is created by a
backup ATTEMPT, not by the Schedule object. A throwaway `paused: true` Schedule on
prod produced ZERO velero_* series, and nothing exposes `spec.paused`. The fact is
not in the TSDB, so the Schedule CRs are exported via kube-state-metrics'
custom-resource-state collector — the path this repo already uses for
kube_node_labels — as kube_velero_schedule_{created,paused,protects_volume_data,info}.
Proven by running KSM v2.20.0 with this exact config against prod's API: the CRD
list/watch rule the chart adds is MANDATORY, not cosmetic — without it KSM emits the
CRS metrics as an empty set, silently.
Three rules, all keyed on PROPERTIES rather than schedule names:
* VeleroScheduleNeverBackedUp (critical) — a Schedule older than 36h with no
successful backup ever. The 36h grace comes from creationTimestamp so a newly
added schedule is not paged before its first run is even due.
* VeleroSchedulePaused (warning) — the direct root-cause signal; a paused backup
schedule must never be silently indefinite.
* VeleroNoVolumeDataBackup (critical) — THE ANTI-MASKING RULE. Fires unless some
schedule with spec.template.defaultVolumesToFsBackup=true succeeded in 36h, so a
healthy resources-only schedule can never make the platform look protected.
Reading the property that DEFINES volume protection (not a name list) matches
platform/velero/volume-policies.yaml and survives renames. `or vector(0)` is
load-bearing: `count(<empty>) == 0` returns no series and would be inert —
absence of the exporter is deliberately the alarm.
Adds the repo's first Prometheus rule-unit-test harness (platform/monitoring/tests/,
wired as the `prom-rules` gate stage). It flattens the COMMITTED PrometheusRule CRs —
never a copy — and pins every rule in the group both ways: quiet on today's healthy
prod state, firing on the reconstructed bad state, with `for:` durations asserted.
Two mutation checks confirm the tests are not vacuous: dropping `or vector(0)` and
breaking the `unless on (schedule)` join each turn the suite red. This is the tool
that makes both of this repo's recurring alert bugs — permanently-firing and
structurally inert — provable offline, which neither yamllint, Kyverno, nor a healthy
cluster can do.
Proof of the fix is by construction; the live metrics cannot show it any more, because
`velero-daily` was unpaused on 2026-09-12 and has since completed 2492/2492 items.
The throwaway probe Schedule and the probe KSM deployment were deleted and the
deletion confirmed; `velero-daily` remains unpaused.
Co-Authored-By: Claude Opus 5 <[email protected]>
R-TEST-03 runs the conformance subset a change TOUCHES. That has one blind spot, and it
is the worst-shaped kind: an invariant that quantifies over the whole tracked tree — or
over subtrees ci/touched-requirements.sh's coarse path map keys to NOTHING (apps/, infra/,
most of platform/) — cannot be selected BY the requirements a change touches, because the
change most likely to break it attributes no requirement at all. Requirement-scoped
selection then filters the check out of exactly that change.
The repo's answer had been a hand-written "UNCONDITIONAL" stage per such check in
ci/l0.sh: 3c flux-postbuild-substitution, 3d keycloak-client-secret-reconciler, 3e
bulk-tier-distinct-from-replicated. Three near-identical 12-line rationales, three
hardcoded check.sh paths, three copies of the same rc 0/3/* case block — and worst, a
mechanism that failed SILENTLY, because whoever added the NEXT whole-repo invariant had
to REMEMBER to write a stage, and if they forgot the check ran only when some unrelated
requirement happened to be touched.
A test now DECLARES the scope of its invariant and the runner enforces it:
metadata:
annotations:
conformance.home.k3s/scope: "whole-repo"
* conformance/lib/common.sh conf_dirs_for_scope(s) resolves the annotation. Carries the
general rationale, once, instead of per stage.
* conformance/run.sh --include-scope SCOPE unions declaring tests into select_dirs,
still bounded by --layer. New --print-selection resolves the
selection and exits, running nothing.
* lib/run_l1_offline.py --unwired-scope: a SKIP of a declaring test exits 3, not 0.
--summary-out: one step-summary line per executed check.
* ci/l0.sh Stage 3b gains --include-scope whole-repo and an rc-3 arm.
Stages 3c/3d/3e DELETED (-150 lines).
* ci/l1.sh, ci/conformance.sh pass --include-scope too, so the next whole-repo invariant
at ANY layer needs only the annotation.
rc 3 SEMANTICS PRESERVED EXACTLY. A whole-repo check reporting "nothing to audit" is
reporting that the tree it audits has VANISHED. The deleted stages called that "not a
pass" and set the gate's rc=3; the runner now returns 3 and ci/l0.sh maps it to rc=3.
A missing check file still degrades to not-green, never to silence.
THE REASONING WAS RELOCATED, NOT DELETED. Each stage's incident narrative moved into the
header of the test it describes, beside its scope annotation: the Cilium k8sServiceHost
empty-substitution CNI outage (2026-09-01), 12 unusable Keycloak clients behind a green
cluster (2026-09-06), and a multi-TB bulk volume landing on the 3-replica tier.
AUDITED ALL 13 EXISTING l0-static TESTS, individually. 10 declare whole-repo scope; 3 are
DELIBERATELY left requirement-scoped and say so in their own headers:
* placeholder-guard-coupling — inputs are bootstrap/ + clusters/, both already mapped to
a requirement it declares. Scoping cannot hide a breaking change.
* dev-nodes-provisioned-as-code — inputs are infra/ and nothing else.
* spec-immutability-enforced — inputs are ci/ and the read-only spec repo.
The latter two were nonetheless under-gated because the PATH MAP was incomplete, which is
the right place to fix a path-scoped invariant: infra/* -> R-ENV-01 added, and R-TEST-06
added to .gitea/*|ci/* (it is no longer an uncovered ID — spec-immutability-enforced
carries the label, so attributing it no longer false-blocks the coverage gate).
NEW conformance/tests/l0-static/whole-repo-invariants-wired (R-TEST-03) keeps the
declarative mechanism honest for ALL declaring tests at once, replacing the three tests'
individual "ci/l0.sh greps my own check.sh path" self-assertions. It proves the union
EMPIRICALLY: with a requirement filter matching no test the L0 selection is empty, and
with --include-scope it is every declaring test; a scope-declared synthetic SKIP exits 3
while an undeclared one exits 0; and a ci/*.sh that hardcodes a check.sh path must belong
to a test that also declares the scope, so the bespoke pattern cannot creep back.
TASK 2 — a changed test attributes the requirements it CERTIFIES.
ci/touched-requirements.sh gains detection rule 3: a change anywhere under
conformance/tests/<test>/ attributes that test's own conformance.home.k3s/R-* labels, read
from its chainsaw-test.yaml. The content scan could not close this — it reads only
added/removed lines, so it sees a change to the LABEL BLOCK but not to the test BODY,
which is exactly the case that matters. Labels are read from HEAD or, failing that, from
the BASE, so DELETING a test still attributes what it used to certify; otherwise removing
the only verifier of a requirement is the cheapest way to make the gate stop checking it.
Whole-line `#` comments are dropped first and *.md is untouched, honouring the existing
"reference, not declaration" rules.
Both changes are fail-closed: every one of them can only ADD tests or requirements to a
run, never remove one.
TESTS: new ci/tests/touched-requirements-test-attribution.test.sh (15 assertions) covers
rule 3 — body-only edit, nested-fixture walk-up, test deletion, header-comment R-IDs not
attributed, non-test paths unaffected, and both new path-map entries. The two existing
suites still pass (39 assertions). `make -C ci test` runs all three.
Co-Authored-By: Claude Opus 5 <[email protected]>
qBittorrent's SavePath was /downloads and no such directory existed in the
container, so the download client had no working destination and nothing
downstream could see its output. It now mounts the same NFS export the arrs
use, narrowed to the downloads subtree via subPath — one mount, so the
EXDEV hardlink trap the bulk-tier test guards stays closed. PUID/PGID are
load-bearing: the export is owned 1000:1000 and the image's default 911 is
denied.
All three primaries had re-concentrated onto prod-1 (the node with known
etcd/Longhorn I/O contention), so losing it meant three simultaneous
failovers. Two switchovers rebalanced them; semaphore-db deliberately stays
on prod-1 as the lightest writer, saving a third write interruption.
CNPG offers no role-aware scheduling — targetPrimary exists only under
status — so the placement is CURRENT, NOT DURABLE: the next failover or
rolling update can re-concentrate it. What is durable is CNPGPrimariesCoLocated,
which reports the condition instead of leaving it to be found by hand.
Every unprefixed './dev.sh run kubectl ...' pointed at the dev cluster,
which was decommissioned on 2026-09-12. The banner cheerfully named dev and
the command then hung or errored against a context that no longer answers,
with nothing saying the cluster is gone. I hit this repeatedly tonight.
CLUSTER=dev is still accepted rather than removed, because the dev VM disks
were deliberately kept so the environment is reversible — but it now warns.
Co-Authored-By: Claude Opus 5 <[email protected]>
qBittorrent was the one member of the download chain with NO media volume.
Its live config said
Session\DefaultSavePath=/downloads/
Session\TempPath=/downloads/incomplete/
Downloads\SavePath=/downloads/
while `ls /downloads` in the container returned "No such file or directory".
The catalog entry declared exactly one volume — config at /config on longhorn —
and carried a comment deferring the downloads volume "until OQ-2 is resolved".
OQ-2 WAS resolved (schema 1.5.0 static bulk NFS PVs); plex, sonarr, radarr and
nextcloud all took their mounts and qBittorrent's was simply never added. So the
download client was writing, or failing to write, a path that did not exist —
and anything it did create would have been invisible to sonarr/radarr, lost on
every pod recreation, and consuming the node's ephemeral disk until then.
WHAT THIS ADDS to catalog/qbittorrent.yaml (rendered into apps/wave-0):
* storage[downloads]: class bulk, export /mnt/Storage/plex, subPath downloads,
mountPath /downloads, 9Ti advisory. ONE mount of the SAME export sonarr,
radarr (rw) and plex (ro) already consume, narrowed to the handoff subtree.
* env PUID=1000 / PGID=3000. LOAD-BEARING, not cosmetic: the container ran as
uid 911 and /mnt/Storage/plex/downloads is 0755 owned 1000:1000, so the
export would have mounted cleanly and then failed every write with EACCES.
Measured both ways on prod: a write as uid 1000 succeeded, the same write as
uid 911 was denied.
* podSecurityContext.fsGroup 3000, for the LONGHORN config PVC only, exactly as
sonarr/radarr set it — fsGroup is a no-op on an in-tree NFS volume.
WHY subPath AND NOT THE EXPORT ROOT AT /media (sonarr/radarr's shape): a download
client has no business writing the library. The arrs need the root mount because
THEY hardlink downloads/ -> tv/|movies/ and link() is a same-MOUNT operation;
qBittorrent performs no such link, so narrowing to downloads/ removes movies/,
tv/ and photos/ from its mount namespace without removing any operation it
performs. It is still ONE mount of the export, so the L0 assertion that no
workload mounts one export twice (the EXDEV trap) holds — and the arrs' own
link() stays intra-mount. Proven live: nlink 1 -> 2, same inode, no EXDEV, every
step executed as uid 1000.
NETWORK: nothing to grant and nothing to steer. An NFS PersistentVolume is
mounted by the NODE KERNEL in the host netns — the established socket to
192.168.2.102:2049 is on the node's own LAN address, the pod netns has no socket
to 2049 at all, and no CiliumNetworkPolicy on this cluster names that host or
port. `ip route get 192.168.2.102` picks eno1 even on the tunnel-owning node, and
Cilium's own egress map lists 192.168.0.0/16 as an Excluded CIDR for every
vpn-class pod while 0.0.0.0/0 goes to the gateway. Public egress still exits the
PIA address; the VPN guarantee is untouched.
NOT CLOSED BY THIS COMMIT, deliberately: absolute paths still differ between this
pod (/downloads/x) and the arrs (/media/downloads/x), so Completed Download
Handling needs one Remote Path Mapping per arr. That lives in the arrs' own
databases, not in git.
Co-Authored-By: Claude Opus 5 <[email protected]>
`instances: 3` + `podAntiAffinityType: required` fixed the INSTANCE spread but moved
the PRIMARY concentration from k3s-prod-2 to k3s-prod-1: all three primaries
(keycloak-db-2, nextcloud-db-2, semaphore-db-2) ended up on one node, so losing it
meant three simultaneous failovers.
There is NO declarative fix, established from the CRD installed on prod (CNPG 1.30.0,
clusters.postgresql.cnpg.io/v1) rather than from memory. Every spec.affinity field and
spec.topologySpreadConstraints applies to "all the cluster's pods" and is not role-aware;
spec.replica.primary names a Cluster object in a distributed topology, not a node;
primaryUpdateMethod/switchoverDelay/primaryLease govern how a promotion happens, never
where. `targetPrimary` exists only under STATUS, never spec — so `cnpg promote` is an
imperative status patch by construction and there is nothing to put in git. The obvious
workaround is worse than nothing: pod anti-affinity is only ever
...IgnoredDuringExecution while CNPG promotes by relabelling a RUNNING pod, and with
3 instances on 3 nodes it would leave the third instance unschedulable.
So this commits the part that IS durable — visibility — and rebalances live.
CNPGPrimariesCoLocated joins CNPG's own answer to "who is primary" (the <cluster>-rw
Service selects only the primary) against the node: kube_endpointslice_endpoints carries
endpoint_nodename + targetref_name, which two label_replaces lift into node/pod. Neither
family works alone — cnpg_* metrics have no node label, and kube_pod_labels is EMPTY here
(no kube-state-metrics label allowlist), so cnpg.io/instanceRole is not in Prometheus at
all. `* on (namespace, pod) group_left () cnpg_collector_up` keeps an unrelated `*-rw`
Service from triggering it, and uses the one metric present for EVERY instance rather than
the cnpg_pg_stat_* family that is missing for nextcloud-db.
A Ready-node gate stops this becoming another permanently-firing alert: with three
clusters on three nodes a node loss NECESSARILY co-locates two primaries, so the rule
only speaks when the spread is actually achievable, and re-arms when the node returns.
Proven both ways. Live: the expr returned {node="k3s-prod-1"} => 3 before, empty after,
with the inner join still resolving all three primaries (quiet because false, not blind);
loaded into prod's Prometheus as a throwaway `for: 1m` twin it went pending at +70s and
firing at +130s. alerts.promtool-test.yaml pins four cases under promtool 3.14.0 (the
exact version prod runs) and is mutation-checked — dropping the join, dropping the gate
or shortening `for:` each makes a test FAIL.
Rebalanced live, one cluster at a time: keycloak-db -> k3s-prod-2, nextcloud-db ->
k3s-prod-3, semaphore-db deliberately LEFT on k3s-prod-1 (lightest writer, and that node
has the known etcd/Longhorn I/O contention) — two switchovers, not three. Write outage
measured with a `select txid_current()` probe, not estimated: 9.4s and 9.43s, one
contiguous burst each. All three back to readyInstances: 3 with
synchronous_standby_names = ANY 1 (...) and both standbys streaming/quorum.
The placement is CURRENT, NOT DURABLE, and the alert is the point: the next failover or
rolling update can re-concentrate the primaries, and now that gets reported instead of
discovered by hand.
Co-Authored-By: Claude Opus 5 <[email protected]>
cnpg_metrics_exporter lacked CONNECT on the nextcloud database: an
out-of-band REVOKE materialized its datacl and dropped PUBLIC's implicit
CONNECT, which keycloak and semaphore still inherit. CNPG's API cannot
express a database-level grant, so this follows the repo's existing
level-triggered CronJob reconciler pattern rather than a one-shot Job that
a rebuild would silently skip. CONNECT only; pg_monitor already carries the
read access.
CNPGNoRecoverabilityPoint fired permanently on all six CNPG replicas because
only the primary owns the backup catalogue; CNPGNoRecentBaseBackup had the
same latent defect, held back only by a > 0 guard. Both now aggregate pod
away and key on the cluster. The remediation text also told the on-call to
run 'kubectl get backup', which reports no resources here.
CNPGNoRecentBaseBackup's own remediation hint said `kubectl get backup`.
On this cluster longhorn.io, postgresql.cnpg.io and velero.io each register
kind Backup, so the bare short name resolves to Longhorn's empty set and
prints 'No resources found' while CNPG backups exist. An on-call following
the alert that fires BECAUSE backups look stale would have been told, with
authority, that no backups exist at all — and might re-bootstrap a database
over good ones. Fully qualify it.
Co-Authored-By: Claude Opus 5 <[email protected]>
The CNPG metrics exporter could not CONNECT to the `nextcloud` application
database, so every user-defined collector query against that target failed on
all three instances:
FATAL: permission denied for database "nextcloud" (SQLSTATE 42501)
DETAIL: User does not have CONNECT privilege.
The whole cnpg_pg_stat_* / pg_settings / pg_postmaster family was therefore
MISSING for nextcloud-db while the scrape target still reported up=1 — the blind
spot platform/cnpg/alerts.yaml's CNPGMetricsCollectorErroring was written to
catch. It had been firing on all three pods; measured before this change,
cnpg_pg_settings_setting had 765 series for keycloak, 765 for semaphore-db and
ZERO for nextcloud-db. Nextcloud's database had never been monitored.
Root cause is the live ACL, not the spec — all three Cluster specs bootstrap
identically:
nextcloud : datacl = {nextcloud=CTc/nextcloud} <- explicit, no PUBLIC entry
keycloak : datacl = NULL
semaphore : datacl = NULL
A NULL datacl means PostgreSQL's defaults, which include CONNECT for PUBLIC, so
keycloak and semaphore admit the exporter only by inheriting PUBLIC's implicit
CONNECT. On `nextcloud` something ran the equivalent of REVOKE ALL ON DATABASE
nextcloud FROM PUBLIC, which materialises datacl and drops that implicit entry.
That REVOKE is in neither this repo nor DECISIONS-LOG.md — it was applied out of
band, which is why the remedy has to be declarative rather than another hand-run
psql.
CNPG 1.30.0 cannot express a database-level GRANT (verified against the live
CRDs): `Database` has no ACL field (`allowConnections` is the datallowconn flag,
not a per-role grant), and managed.roles/DatabaseRole cover role attributes plus
`inRoles` membership only — and membership would hand the exporter the owner's
CREATE/TEMP and all table access to buy one CONNECT. postInitApplicationSQL runs
once at bootstrap, so it cannot repair the already-bootstrapped instances that
are actually broken. So this follows the pattern the repo already uses for state
Flux cannot reach: the level-triggered CronJob reconciler alongside
platform/keycloak/client-secret-reconciler.yaml. A one-shot Job is deliberately
avoided — a completed Job never re-runs, so a rebuild or PITR restore would
silently reopen the blind spot.
Exactly one privilege, and nothing else:
GRANT CONNECT ON DATABASE nextcloud TO cnpg_metrics_exporter;
giving `cnpg_metrics_exporter=c/nextcloud` — CONNECT only, no CREATE, no
TEMPORARY, no ownership, no superuser, no table or schema grants. Nothing more is
needed because CNPG already creates the role as a member of the built-in
pg_monitor role (verified on all three clusters), which carries the read access
the collector's pg_catalog queries want. It runs as the database OWNER, not a
superuser: enableSuperuserAccess stays off, and a database owner holds the grant
option on its own database.
Verified live on prod:
* the grant issued under SET ROLE nextcloud, proving no superuser is needed;
* permission-denied and "Error collecting user query" both went to 0 on
nextcloud-db-1/-2/-3, i.e. every collector query now succeeds;
* cnpg_pg_settings_setting now 765 series for nextcloud-db, matching the other
two, and cnpg_pg_postmaster_start_time present on all three instances;
* cnpg_last_error = 0 on all NINE instances, so keycloak-db and semaphore-db
are unaffected and their datacl is still NULL;
* convergence proven by revoking the grant again and letting the committed
CronJob restore it, then re-running it against a converged database to show
idempotence;
* CNPGMetricsCollectorErroring, firing on all three pods beforehand, has
cleared (`for: 6h` delays firing, not resolution).
Co-Authored-By: Claude Opus 5 <[email protected]>
Three API groups register kind Backup (longhorn, cnpg, velero). The bare
short name resolves to Longhorn's empty set, so kubectl reports absence
while 27 CNPG backups and live Velero backups exist. A false negative on
'do I have backups' is the worst possible time to be wrong, so record the
rule: fully-qualified resource names only.
Co-Authored-By: Claude Opus 5 <[email protected]>
CNPGNoRecoverabilityPoint has been firing continuously on prod against six
pods — keycloak-db-1/-3, nextcloud-db-1/-3, semaphore-db-1/-3 — i.e. exactly
the REPLICAS of all three Clusters, while the databases themselves are fine:
every Cluster reports firstRecoverabilityPoint=2026-09-07T07:0x, a
lastSuccessfulBackup within the last few hours, ContinuousArchiving=True and
LastBackupSucceeded=True.
Root cause is aggregation scope, not archiving. `cnpg_collector_first_recoverability_point`
reads the backup CATALOGUE, which only the primary owns; a replica reports a
hard 0 for as long as it is a replica. The metric carries no cnpg.io/cluster
label (its label set on prod is container/endpoint/instance/job/namespace/pod),
so `max by (namespace, pod)` evaluated each replica as its own subject and an
`== 0` comparison then held forever. The rule went from correct to permanently
wrong the moment the Clusters went to `instances: 3`; nothing about the
expression changed.
WHY THIS IS WORSE THAN HAVING NO ALERT. An alert that is always firing has
zero information content: its value is identical in the healthy state and in
the failure state, so it cannot distinguish them. It is worse than silence in
two specific ways. First, it trains the operator to delete the notification on
sight, so the day a base backup genuinely stops the seventh identical email is
discarded with the other six — the alert actively MASKS the failure it was
written to catch, which for a DR alarm means discovering the loss at restore
time. Second, it is indistinguishable from a broken alert, so nobody can tell
whether the absence of new alerts means health or means the rule rotted. This
file exists because a real 38-hour archiving failure told nobody; a rule that
cries wolf about six healthy replicas rebuilds exactly the blindness it was
supposed to remove. This repo has been here before — see the permanently-firing
critical VPNTunnelDown post-mortem in platform/monitoring/alerts.yaml.
Fix: group by (namespace, job) and aggregate `pod` away. The CNPG operator
creates one PodMonitor per Cluster, named after the Cluster, in the Cluster's
namespace, so prometheus-operator derives job="<namespace>/<cluster>" —
keycloak/keycloak-db, nextcloud-db/nextcloud-db, semaphore-db/semaphore-db —
which identifies the CNPG Cluster uniquely. `job` rather than bare `namespace`
deliberately: namespace alone works today (one Cluster per namespace, verified
live) but would let two Clusters in one namespace mask each other, since the
healthy one's non-zero value would satisfy `max` for both. The alert now
states what its summary always claimed — this CLUSTER has no recoverability
point — and fires only when no instance of that Cluster reports one.
Proven live against prod Prometheus, read-only (promtool query instant):
OLD max by (namespace, pod) (...) == 0 -> 6 series, and 6 ALERTS firing
NEW max by (namespace, job) (...) == 0 -> empty
The new expression is not vacuous: forcing the metric to 0 for one cluster's
primary only yields exactly one series, {job="keycloak/keycloak-db"}, so a
genuine single-cluster loss of DR still alerts, per cluster. `promtool check
rules` passes on all 5 rules.
CNPGNoRecentBaseBackup carried the same defect and is fixed the same way.
`cnpg_collector_last_available_backup_timestamp` is also primary-only: on prod
the primaries read 3.1h/2.1h/0.1h old while all six replicas read 0, i.e.
time()-0 = ~56.7 YEARS. It was not firing only because the `> 0` guard was
load-bearing — a single clause was all that stood between this rule and six
more permanent false positives, too thin a margin for a DR alarm. Aggregating
`pod` away removes the dependence: `max` takes the primary's timestamp as the
cluster's answer, and the guard now means "this cluster has had at least one
backup". Verdict unchanged live (old empty, new empty).
The other three rules were audited and deliberately left on `by (namespace, pod)`:
* CNPGWALArchiveBacklog — every instance reports archive_status and the shape
is `> 0`, so a replica at 0 never fires and no pod can mask another; `pod`
names the instance whose archiver is stuck.
* CNPGWALVolumeNotRecycling — per-pod is REQUIRED, not merely safe:
aggregating it away would compare one instance's WAL count against
another's max_wal_size.
* CNPGMetricsCollectorErroring — unchanged in meaning AND grouping. Its
grouping is already correct: cnpg_last_error is genuinely per-instance, and
prod proves it discriminates rather than tracking role — all three
nextcloud-db pods report 1 while all six keycloak/semaphore pods report 0.
It is a true positive (the nextcloud 42501 CONNECT grant) owned by separate
work; nothing here silences or rescopes it.
The general rule, recorded in the file: `== 0`/staleness shapes must aggregate
`pod` away or every structurally-silent pod fires forever; `> 0` shapes are
safe per-pod and keep the instance name.
Co-Authored-By: Claude Opus 5 <[email protected]>
The CNP that lets flux-system's source-controller clone from the in-cluster Gitea
Service was APPLIED LIVE on prod on 2026-09-13 and existed nowhere in this repo. It
carried a provenance annotation saying so. Until now it could not be authored: the
gitea CNPs are GENERATED from the catalog entry, and the contract had no field for
"also admit this in-cluster client" (fixed in the previous commit).
Why it matters more than a missing allow normally would. prod's GitRepository points
at the Service, not a public name (clusters/prod/kustomization.yaml:
`url: http://gitea.gitea.svc.cluster.local:3000/Home-Network/home-network-k3s-impl.git`),
deliberately, so prod's GitOps depends on neither public DNS, Cloudflare, prod's own
Gateway nor ${BASE_DOMAIN}. But gitea-allow-gateway-ingress admits only
`fromEntities: [ingress]`, so source-controller's clone was dropped — and the
HTTPRoute that would have served the post-apex-flip public name instead lives in a
DOWNSTREAM tier that cannot apply while the source is down. Source down -> routes
never rename -> source stays down. Every `kubectl delete cnp` and every cluster
rebuild re-armed that deadlock, and it would have been extremely hard to diagnose
cold.
The rendered object is EQUIVALENT to the live one — same apiVersion/kind, same
name/namespace/labels, and a byte-identical spec (endpointSelector
app.kubernetes.io/name=gitea; fromEndpoints matchLabels
k8s:io.kubernetes.pod.namespace=flux-system; toPorts 3000/TCP), verified against
`kubectl --context k3s-prod -n gitea get cnp gitea-allow-flux-ingress -o json`. The
name matches, so Flux ADOPTS the live object in place: no spec change, so no policy
recompute and no flap on prod's own source. The only difference is that the rendered
object carries no `homelab.keithjackson.dev/provenance` annotation — the reasoning now
lives where the object is authored, in catalog/gitea.yaml, next to the field that
generates it.
Golden re-rendered with the canonical command; the whole apps/ diff is the one new
document.
Co-Authored-By: Claude Opus 5 <[email protected]>
A platform controller that dials a service's ClusterIP directly had NO authorable
form in the ServiceDefinition contract:
* ingressRoutes renders the north-south allow only — gateway-ingress admits
`fromEntities: [ingress]`, the Gateway's Envoy identity, and nothing else;
* dependencies compiles edges only BETWEEN CATALOG SERVICES, from the source
entry's own declaration.
Under the default-deny baseline (R-NET-01) such a client is therefore DROPPED, and
the only way to unblock it was a live `kubectl apply` of a hand-written CNP — a
cluster object outside git, one `kubectl delete` or one rebuild from coming back.
`inClusterClients: [{namespace, port, protocol?, name?}]` closes that. Each entry
renders ONE least-privilege ingress CiliumNetworkPolicy: this service's
endpointSelector, `fromEndpoints` matching ONLY `k8s:io.kubernetes.pod.namespace`,
`toPorts` carrying ONLY the declared port/protocol — never `fromEntities: all`.
Additive: the namespace default-deny keeps denying everything undeclared (R-NET-02),
and the entry is NOT a dependency edge, so it creates no wave ordering.
Shape notes:
* selected by NAMESPACE, not pod labels: an upstream controller's labels belong to
its chart, not to this platform's contract. The cost is explicit — every pod in
that namespace is admitted on that port, so the field is for platform-controller
namespaces (flux-system, monitoring), never a shared one. The schema description
says so.
* optional `name` (default: the namespace) only names the rendered policy,
`<service>-allow-<name>-ingress`. That name is operator-facing (kubectl get cnp,
a Hubble drop), and pinning it lets an allowance that already had to be applied
live be authored under its EXISTING name, so Flux adopts the live object in place
instead of creating a near-duplicate and leaving the original unmanaged.
* one field, not a policy DSL: no arbitrary selectors, entities or CIDRs.
Covered by render.py --selftest (4c): name/namespace/selector/port shape, no
fromEntities widening, default-deny still `ingress: []`, the gateway allow not
displaced, `name` defaulting to the namespace, and nothing rendered when the field is
absent. No golden changes here — no catalog entry uses the field yet.
Co-Authored-By: Claude Opus 5 <[email protected]>
The apex flip deadlocked prod's GitOps and this is the durable fix.
The GitRepository pointed at a public hostname. The flip renamed prod's HTTPRoute
hostnames, so that name stopped serving — but the HTTPRoute that would serve the new
name lives in a DOWNSTREAM tier which cannot reconcile while the source is broken.
Source down -> routes never rename -> source stays down. Observed live: srcReady=False,
"unable to list remote for https://gitea.home.keithjackson.dev".
Pointing at gitea.gitea.svc.cluster.local:3000 removes the cycle permanently: prod's
GitOps no longer depends on public DNS, Cloudflare, its own Gateway, or ${BASE_DOMAIN}.
No future domain change can sever prod from its own source.
Plain http is deliberate: cluster-internal traffic to a ClusterIP, TLS terminates at
the Gateway for external clients, the GitRepository has no secretRef (anonymous clone),
and all secrets are SOPS-encrypted at rest regardless of transport.
Co-Authored-By: Claude Opus 5 <[email protected]>
The GitRepository url is a hardcoded literal, not templated on ${BASE_DOMAIN}, so
the apex flip did not carry it. But the flip renames prod's HTTPRoute hostnames, so
external-dns (policy=sync) deletes gitea.prod.home.keithjackson.dev and CoreDNS's
split-horizon zone moves from prod.home. to home. — which would have severed prod
from its OWN GitOps source moments after the flip reconciled.
Landed in the same push as the flip deliberately: prod has not yet fetched 9161cab
(still on b9c0970, all hostnames still prod.home.), so both commits arrive in one
fetch and the url changes in the same apply that renames the routes. A live patch
would not survive — the root Kustomization renders this very file and reverts it.
Co-Authored-By: Claude Opus 5 <[email protected]>
Dev is decommissioned and has released the name. Prerequisites completed and
verified BEFORE this flip:
1. dev's 18 A records released by its own external-dns (policy=sync; sources
deleted after suspending dev's Flux so they could not be recreated).
2. dev's 18 LEGACY UNPREFIXED TXT registry entries deleted from Cloudflare by
hand. external-dns v0.21.0 deleted only the `a-` prefixed form and then logged
"All records are already up to date" — it would NEVER have cleared the older
unprefixed format it no longer writes but still READS for compatibility. They
carried `owner=default`; prod runs `owner=prod`, so prod could not have
claimed these names while they existed. This was the real blocker, and it was
invisible until measured.
3. dev's `*.home.keithjackson.dev` + apex Certificate deleted so it stops
renewing. Issued ~18d ago, so prod's request is #1 this week against LE's
duplicate limit (5 per identical name set per week).
Verified at the AUTHORITY (betty.ns.cloudflare.com), not via a resolver: 1.1.1.1
served stale TXT for minutes after deletion and would have shown a false failure.
UNCHANGED, deliberately: EXTERNALDNS_TXT_OWNER_ID stays `prod` — giving prod
`default` would let it adopt and sync-delete records it does not own, which is the
2026-09-04 incident. EXTERNALDNS_DOMAIN_FILTER stays the ZONE `keithjackson.dev`:
finding F-DNS1 proved a strict subdomain filter drops the hosted zone and external-dns
then publishes NOTHING.
The zone has wildcards `*.keithjackson.dev` and `*.home.keithjackson.dev` pointing at
the public IP (legacy non-k3s prod). Dev's specific records were shadowing the latter;
prod's new specific records shadow it again. The legacy stack lives on
`*.keithjackson.dev` and is untouched.
k3s_base_domain moves with it to hold the [parity] contract, but that value renders
the apiserver's install-time oidc-issuer-url: it does NOT take effect until the
k3s_server role re-renders config.yaml and k3s is restarted serial: 1. Until then the
apiserver trusts the OLD issuer while Keycloak serves the NEW one, so kubectl OIDC
fails in that window. Cert-based kubeconfigs are unaffected.
Co-Authored-By: Claude Opus 5 <[email protected]>
Records K1-K9: extend-vs-sibling and why, the generated+gated desired state, verbatim
token pass-through, per-pass fail-closed, order-insensitivity, unmanaged-means-untouched,
public clients' URLs managed, one alert pair not two, and the mutation harness.
Also records a process note: my first pass reported three false findings (reconciler
absent from main, import not templated, BASE_DOMAIN dead) because I diffed against
origin/main -- origin is dev's Gitea, stale at 4809199, while the live remote is
prod-gitea at fa4deac. All three were wrong. The rule for this repo is written down so
the next agent does not repeat it: origin is not the source of truth.
Co-Authored-By: Claude Opus 5 <[email protected]>
L0 (212 assertions, was 175):
* runs ci/render-keycloak-client-urls.py --check, so client-urls.yaml must be exactly
what the realm import renders. A hand edit that makes the two disagree is
unmergeable -- this is what keeps the realm import the single source of truth for
hostnames rather than just the place they were first typed.
* asserts client-urls.yaml and the realm import are rendered by the SAME Flux
Kustomization. postBuild.substituteFrom is per-Kustomization, so a split would let
one ${BASE_DOMAIN} resolve to two different values and the realm would converge on
a hostname the HTTPRoutes do not serve.
* asserts the desired state uses no substitution token the realm definition lacks,
names only clients the import declares, and declares no field outside the
five-field write surface; that the CronJob mounts it readOnly at the path the
script defaults to; and that the script, renderer and check agree on which fields
are writable.
* NEW mutations.py: injects 12 plausible regressions into the EXTRACTED script and
requires --selftest to catch every one. A passing suite proves nothing if it cannot
fail, and "reported success while the realm was broken" is precisely what this
component exists to end -- so the same standard applies to its own test. ~20s.
L2: the audit Job now mounts keycloak-client-urls and asserts
KEYCLOAK_CLIENT_URLS_AUDIT_OK. Without that it would pass on a realm still holding
pre-migration redirect URIs -- the exact state prod was measured in.
Alerts: the existing Failing/Stale pair is EXTENDED to name redirect-URI drift, not
duplicated. One CronJob gets one pair; a second pair on the same job_name regex would
fire twice for one cause. No new critical, no new permanently-firing rule.
Co-Authored-By: Claude Opus 5 <[email protected]>
The realm import is a one-shot that never re-runs, so a redirect URI changed in git
reaches the realm never. Prod's stored URIs were measured still on the pre-migration
hostname while git declared the new one -- after a BASE_DOMAIN flip every OIDC login
fails with a redirect_uri mismatch, with the CR still reading Done=True and nothing
alerting. Same failure class as the literal-placeholder secrets, same cure.
EXTENDS the existing reconciler rather than adding a sibling. Decisive reason: a URL
write needs NO new endpoint. It is the same minimal client PUT, and the current URLs
already come back in the allow-listed client lookup -- so the allow-list is widened by
nothing and the users/groups/roles/authentication guarantee is preserved unchanged
rather than re-declared in a second copy. The ConfigMap also carries exactly one
script key by L0 contract, so a sibling would have had to fork ~300 lines of
safety-critical code that the header explicitly forbids forking.
Reconciles redirectUris, webOrigins, rootUrl, baseUrl, adminUrl -- only the fields a
client actually declares, and on a write only the fields that actually drifted. A
field the desired state omits is UNMANAGED and left untouched; absence never means
force-to-empty.
THE TWO PASSES ARE INDEPENDENT. Each reads its own desired state and fails closed for
its own fields only. A malformed URL ConfigMap cannot stop client-SECRET repair, which
would otherwise make credential healing depend on a hostname file.
List fields compare order-insensitively (sorted, not set -- duplicates are still
drift), because Keycloak does not preserve the authored order of redirectUris and an
order-sensitive compare would rewrite every client forever while reporting success.
--selftest grows to 84 assertions covering the domain flip, a client gaining a URI, a
client losing a stale one, a public client's URLs being managed while its secret is
not, per-pass fail-closed, order-insensitivity, and a server that accepts a PUT
without applying it.
Co-Authored-By: Claude Opus 5 <[email protected]>
The realm import is a one-shot: once Done=True it never re-runs, so a redirect URI
changed in git never reaches the realm. Healing that needs a DESIRED STATE the
reconciler can read -- and the reconciler pod has no Kubernetes API access at all
(automountServiceAccountToken: false), so it cannot read the KeycloakRealmImport CR.
Its desired state must be mounted, hence a ConfigMap.
A hand-authored ConfigMap would be a second source of truth for every public
hostname in the realm, free to disagree with the import -- the same shape as the bug
being closed. So it is GENERATED from the import and the render is gated, the same
render==golden idiom catalog/render.py already uses for apps/.
URL strings are copied VERBATIM: a ${BASE_DOMAIN} token in the source appears
unchanged here, and both files are rendered by the SAME Flux Kustomization
(./platform-config), so both are substituted in one pass with one value. The
renderer never parses a hostname and never reads cluster-vars, so there is no
second place the domain can be decided.
Co-Authored-By: Claude Opus 5 <[email protected]>
One historical `-dirty` row reddened this gate permanently, even after a clean
re-run of the same test. status.py already resolves duplicates latest-wins
(latest_by_test), so a superseded row is not evidence of anything — but this check
iterated every row, so the poisoned row from the git_sha bug (fixed in the parent
commit) would have failed the audit forever.
The dirty-tree assertion is now scoped to rows status.py would actually honour, by
importing latest_by_test rather than reimplementing the rule, so the two cannot
disagree about which rows are live claims. Structural assertions — provenance keys,
sentinel, test-still-exists — still apply to EVERY row.
Nothing is erased: the row stays in the ledger and runs/20260913T014437Z-19fc8bc/
run.log stays committed. It simply stops being read as a live claim. Verified the
guard is intact afterwards — the check still REFUSES both forgeries it is built to
catch (a self-skip presented as a pass, and a proven row from the root commit).
A dirty-then-clean sequence is also the normal shape of noticing a dirty tree,
fixing it, and re-running; that should not permanently fail an audit.
Co-Authored-By: Claude Opus 5 <[email protected]>
1. auditability-proofs-producible rejected any proven ledger row without a
sentinel, with no layer exemption — while record_run.py DELIBERATELY records
L0/L1 rows sentinel-less via its own SENTINEL_EXEMPT_LAYERS, because
run_l1_offline.py renders skips as JUnit <skipped/> and those layers have no
pass token to attribute. So ANY `--record` run touching L0 or L1 wrote a row
this gate then rejected. Consequence: a requirement whose verifier set includes
an L1 check could never be both PROVEN and L0-green — R-NET-06 is exactly such
a requirement, and hit it the moment it was re-proven on prod. The check now
honours the same constant instead of restating the policy, so the two halves
cannot drift apart again.
2. git_sha() marked the tree dirty from tracked files, but conformance/runs/
ledger.jsonl IS tracked — so the FIRST `--record` run of a session dirtied the
tree and every LATER run was stamped `-dirty`, which check.sh then refuses to
honour. Recording one test poisoned the next, and that cost a second live
fault-injection run against prod to work around. Its docstring already stated
the intent for the untracked runs/<id>/ half of the same artifact; the ledger is
the tracked half. Now excluded via pathspec, mirroring status.py's
IMPL_EXCLUDE_PREFIXES which already lists this path for the same reason.
Neither change weakens an assertion: (1) applies an exemption that already existed
and was already documented, (2) stops the recorder invalidating its own output.
Co-Authored-By: Claude Opus 5 <[email protected]>
Supersedes the -dirty row from the previous commit by the ledger's own
append-only / latest-wins convention (lib/status.py latest_by_test): nothing
is rewritten, a newer row simply outranks the older one.
runs/20260913T014923Z-9d0814a vpn-killswitch (L3) proven env=prod
git_sha 9d0814ae... (CLEAN — resolvable, so the freshness check passes)
Fault injection against LIVE prod, gateway k3s-prod-3. Timeline (UTC):
01:49:27 tunnel-stop Job applied
01:49:31 Job succeeded — host vpn-egress-tunnel STOPPED, tun0 gone
01:49:31..01:51:01 vpn-class blackholed for 8 consecutive polls, NO response
of any kind; internet-class unaffected on the WAN
01:51:02 self-healing finally: tunnel-restart Job
01:51:05 kubectl wait confirms the host brought the unit back
01:51:04 egress-uplink label AND the BPF egress map both move 134 -> 139
(measured independently, out of band, during the run)
01:51:08 vpn-class egress back on PIA 158.173.241.11
RECOVERY WAS UNAIDED and took ~3s from tunnel-up to a re-steered datapath —
far inside the ~2-minute convergence timer, so the EVENT-DRIVEN up-script
path did the work, not the timer backstop. This is the half that was
previously unprovable: before the egress-uplink label, Cilium held the stale
ifindex and vpn-class egress never came back.
WHAT THE NO-LEAK ASSERTION DOES AND DOES NOT PROVE: it proves vpn-class
fails CLOSED (zero packets out, never the WAN IP). It does NOT credit the
host nftables backstop, which provably cannot fire for pod traffic —
bpf_redirect() skips netfilter. Stopping the tunnel removes tun0 and Cilium
drops on its own because the cached egress ifindex is unusable. Fail-closed
is real; the mechanism is Cilium's drop, as the test header now says.
Co-Authored-By: Claude Opus 5 <[email protected]>
Two recorded runs against PROD (CONF_ENV=prod, context k3s-prod):
20260913T014321Z-19fc8bc egress-class-render (L1) proven
20260913T014437Z-19fc8bc vpn-killswitch (L3) proven, but git_sha -dirty
The L3 run PASSED for real on prod — every declared sentinel present
(NS_PRIVILEGED_OK, TUNNEL_HEALTHY_OK, NO_LEAK_OK, INTERNET_UNAFFECTED_OK,
TUNNEL_SELF_HEAL_OK, TUNNEL_RECOVERED_OK), tunnel stopped 01:44:48Z and
recovered unaided by 01:46:25Z.
Its row is nevertheless NOT usable as proof: git_sha carries `-dirty`, so
status.py's ancestry check cannot resolve it and the row degrades to
stale-proof. Cause is a harness flaw, not the cluster:
lib/record_run.py's git_sha() ignores UNTRACKED files (so a run's own
runs/<id>/ does not mark it dirty) but conformance/runs/ledger.jsonl is
TRACKED — so the FIRST --record run of a session appends a row, dirties the
tree, and every SUBSEQUENT --record run in that same session is stamped
-dirty. Recording one test poisons the proof of the next.
Committed as-is rather than rewritten: the ledger is append-only and this is
honest evidence of a real run. A clean re-run follows and supersedes it by
latest-wins (lib/status.py latest_by_test).
Co-Authored-By: Claude Opus 5 <[email protected]>
`VPNTunnelDown` goes green the instant a tun0 exists, which is not the same as
vpn-class pods having egress: Cilium caches (egress IP, egress ifindex) at first
resolution, so after a reconnect the BPF egress map points at a destroyed
interface and the packets are dropped while tun0 looks perfectly healthy. Dev sat
in exactly that state for ~80 minutes with nothing alerting.
Two rules close it:
* VPNEgressSteeringStale (critical, for: 15m) — fires when the live tun0 was
re-created more times in the last hour than the gateway node's `egress-uplink`
label changed, i.e. Cilium was never told about a reconnect.
* VPNEgressUplinkLabelMissing (warning, for: 30m) — the watchdog for the
watchdog: a gateway node whose uplink label never reaches kube-state-metrics is
a cluster where the rule above is inert.
Both gated on a gateway node existing, reusing `VPNTunnelDown`'s
`kube_node_labels` guard — a cluster with no gateway cannot be made to fire them.
The direct label-vs-map comparison is NOT available and nothing was invented to
pretend otherwise: no `cilium_*`/`hubble_*` series exist on either cluster (the
cilium-agent exposes no metrics port, there is no ServiceMonitor), and Cilium OSS
publishes no series carrying egress-map contents at all — the map is readable only
via `cilium-dbg bpf egress list`. PromQL also cannot compare a metric value (the
live ifindex) against a number inside a label value, so the level-triggered
equality test is not expressible; the same invariant is counted over a window
instead. Both limits, and the textfile-collector/black-box-probe routes that would
lift them, are written down in the rule comments and the DECISIONS-LOG.
`egress-uplink` added to kube-state-metrics.metricLabelsAllowlist in the same
kustomization, because without it the first rule has nothing to compare.
Verified read-only against LIVE prod (port-forward + /api/v1/query), rendered expr
extracted from `kubectl kustomize platform-config`: empty when healthy; empty when
evaluated at a timestamp before prod had a gateway node; and FIRING when pointed at
a label that genuinely failed to follow prod's real 128 -> 129 tun0 flap earlier
today. Nothing on prod was disturbed to produce that.
Co-Authored-By: Claude Opus 5 <[email protected]>
Closes the loop on the collision entry with the recovery evidence rather than
leaving it as an open question for the operator: dev's last Inactivity timeout is
23:05:06 UTC — the minute prod switched to .135 — and at 23:12:03 that same
session was still the live one, seven minutes unbroken against a two-minute
cycle, 4/4 ping replies at 146 ms, 0% loss, and vpn-class pod egress back to
158.173.241.36 on three consecutive probes.
Co-Authored-By: Claude Opus 5 <[email protected]>
The stand-up broke dev, so the "one gateway per PIA server" rule and the
measurement behind it are now written down where the next person will look:
platform/vpn-egress/README.md and the session's DECISIONS-LOG entry. Both carry
the verbatim journal lines — same server, same pushed `ifconfig 10.72.192.7` on
both gateways — and the flap timestamps, because "same account, inside the
connection limit" is true and is exactly what makes this non-obvious.
Also corrected: the dev/prod table now shows the two environments on different
servers with different exit IPs (dev 158.173.241.36, prod 158.173.241.11), and the
open item about them sharing one public IP is replaced with the real remaining
decision — whether prod gets its own PIA account, which would remove the
hand-maintained per-environment server value entirely.
The unplanned bonus, recorded in both places: moving prod to .135 gave it a
genuinely NEW tunnel address on a real reconnect (10.72.192.7 ifindex 128 ->
10.150.192.86 ifindex 129) — the precise case Cilium does not follow and the one
defect 1 was about. The egress-uplink label moved on its own, Cilium re-resolved,
and all four vpn-class workloads kept egressing with no manual step. Better
evidence than the synthetic address swap used to characterise it, and it cost
nothing to capture.
Co-Authored-By: Claude Opus 5 <[email protected]>
Standing up prod broke dev, and the mechanism is worth spelling out because
"same PIA account, well inside its connection limit" is true and still not enough.
The vaulted .ovpn pins ONE server. With both gateways using it, dev and prod
landed on the SAME server AND were handed the SAME tunnel address — both journals
read `[Server-10779-2a] Peer Connection Initiated with [AF_INET]158.173.241.112:1198`
and `ifconfig 10.72.192.7 255.255.192.0`. Each reconnect steals that address on
that server; the loser then receives nothing, trips `--ping-restart` after 60s,
reconnects and steals it back. Measured cycle: dev restarting at 22:57:05,
22:59:03, 23:01:05 UTC with prod interleaved about a minute off, each ~2 minutes
apart, indefinitely. Neither environment had reliable vpn-class egress — prod's
probes looked green only because they happened to land inside an up-window.
Fix: `vpn_egress_gateway_pia_remote` ("<host> <port>", empty = honour the .ovpn)
rewrites just the `remote` line of the deployed .ovpn, and prod's group_vars sets
it to 158.173.241.135. That is another member of the same PIA Stockholm LB pool —
`sweden.privacy.network` resolved from k3s-prod-3 to
158.173.241.{34,55,112,135,239} — and every server in the pool takes the same
credentials and the same inline CA, so only the address differs. Dev keeps the
.ovpn's .112 untouched, which is why the override is per-group rather than in the
role or a second vaulted .ovpn.
Two details that matter for whoever touches this next:
* The regex is anchored to a line STARTING with `remote` plus whitespace, so
`remote-cert-tls` and `remote-random` are not collateral damage. The task keeps
`no_log: true` — the transformation never widens what is logged.
* A PIA server address is NOT a credential, so it lives in
group_vars/vpn_egress_gateway_prod/vars.yml in the clear, next to the vault
rather than inside it. Keeping it out of the vault is what makes it reviewable.
If prod ever gets its own PIA account, this override can be deleted again.
Co-Authored-By: Claude Opus 5 <[email protected]>
Four places in the tree told the reader that the host nftables table is the
authoritative fail-closed backstop for egress-class=vpn pods: platform/vpn-egress
README, egress-steering.yaml, the role README, and the vpn-killswitch test's own
GIVEN/THEN prose. It is not, and cannot be — Cilium's eBPF egress-gateway
datapath bpf_redirect()s those skbs to the egress device without invoking
netfilter POSTROUTING. Corrected with the measurement in each place rather than
softened, because a guard the repo trusts and that does not fire is worse than no
guard.
What replaces the claim, everywhere: vpn-class fails closed because CILIUM drops
the traffic when it cannot use its cached egress interface, verified by tcpdump on
both the WAN and tun0 during a tunnel stop and a tunnel restart (0 packets out
either). The nft table keeps a narrower, now-proven job: node-originated traffic
sourced from a tunnel address (killswitch_drop moved 0 -> 3 packets on
k3s-prod-3 when such a packet was routed at the WAN).
Also recorded:
* egress-steering.yaml — `interface: tun0` is resolved ONCE and cached, and the
gateway node is what tells Cilium it changed (the egress-uplink node label). A
future reader's instinct will be to have the node write to that object; the
comment says not to, because it is Flux-managed, and points at the label and
`cilium-dbg bpf egress list` as the debugging entry points instead.
* egress-steering.yaml — why the new NFS media mounts are not affected by
pod-scoped steering: an NFS PV is mounted by the node kernel in the host netns,
so podSelector never matches it. The 192.168.0.0/16 exclusion covers the server
too, but that is the second reason, not the first.
* vpn-killswitch test — its recovery step depends on the re-steer path, and could
not have passed before it existed. Its no-leak assertion was always right; only
the explanation was wrong. Also states the prerequisite plainly: the test cannot
run at all without a labelled gateway node.
* platform README — prod is live on k3s-prod-3 with the before/after egress IPs,
and the note that prod's vpn-class pods had NO internet beforehand (CEGP with no
matching node -> `Gateway IP: Not Found` -> dropped), so arming the gateway
restored egress rather than moving it off the WAN.
* role README — the standing "add the shared fwmark to the interface contract"
escalation is closed as not implementable on Cilium OSS, rather than left open
for someone to attempt.
DECISIONS-LOG carries the full session entry, including the drafted ADR-0015 /
docs-03 amendment for the architect (specs are immutable to agents) and the five
things left for the operator — chief among them that a silent gateway outage still
has no alert, which is what let the dev gateway sit dead for 80 minutes.
Co-Authored-By: Claude Opus 5 <[email protected]>
`[vpn_egress_gateway_prod]` is a SEPARATE inventory group with its own vaulted
credentials, not a second member of `[vpn_egress_gateway]`. That group is the
default target of playbooks/vpn.egress.gateway.yml, so adding a prod host to it
would mean a flagless `ansible-playbook playbooks/vpn.egress.gateway.yml`
silently reconfigures a prod node's kill-switch datapath. Prod is reached only by
asking for it, with `--list-hosts` first:
-e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3
Node 3 because node 1's SSD was replaced yesterday (least operational history)
and node 2 carries the §0b memattr history. Exactly one node, forever: Cilium OSS
picks "the first node in lexical ordering" among nodeSelector matches, so a
labelled k3s-prod-1 would silently steal the gateway role — and the public egress
IP — from k3s-prod-3, with no event and no alert.
vault.yml is a verbatim copy of the dev group's CIPHERTEXT: same PIA account per
the operator, and copying the encrypted blob means the plaintext was never
decrypted to disk, through a shell, or into a transcript. The two files are now
independent — rotating means re-encrypting both, which is the right behaviour for
two environments.
Also corrects a stale inventory note that claimed k3s-prod-3 has wlp1s0 up on
192.168.2.195. Re-verified before choosing it as the gateway, because a second
live uplink in a kill-switch datapath is not a footnote: there is no wl* netdev on
the node at all, no phy80211 device, the /etc/network/interfaces wlp1s0 stanza is
commented out, 192.168.2.195 is assigned nowhere, and the only IPv4 addresses are
lo and eno1 192.168.2.12. Nothing needed bringing down.
Co-Authored-By: Claude Opus 5 <[email protected]>
Measured on k3s-dev-3, not reasoned about. Three findings, one of which had the
dev gateway silently dead for ~80 minutes before this change:
1. Cilium caches (egress IP, egress ifindex) when it first resolves the CEGP's
`interface: tun0` and does NOT re-resolve when tun0 changes. Every tunnel
restart re-creates tun0 with a NEW ifindex, so vpn-class egress is blackholed
from then on. Observed: node had tun0 at ifindex 1987 while `cilium-dbg bpf
egress list` still said 1986; pod curl timed out. Forcing a re-apply moved the
map to 1987 and egress recovered instantly. `egress-gateway-reconciliation-
trigger-interval: 1s` does NOT cover netlink link events — dev has that same
1s setting and still did not self-heal, 20s+ after recovery.
Fix: vpn-egress-resteer.sh publishes "<tun0 addr>-<ifindex>" as a node label.
A node update makes the gateway re-process the policy (verified: cached
ifindex 1986 -> 1987 within 10s). A node label, not an annotation on the
policy, because the policy is Flux-managed GitOps and the node's labels are
already this role's territory. Fired by the OpenVPN up-script on every
connect, plus a 2-minute convergence timer for the boot case where the
kube-API is not up yet. Proof: `systemctl stop` then `start` of the tunnel now
recovers on its own (map 1989 -> 1990, PIA IP returns); before this commit the
identical sequence stayed broken indefinitely.
2. The nft kill-switch CANNOT see the traffic it claims to guard. A counters-only
diagnostic table on the gateway node recorded, while a vpn-class pod egressed:
0 packets in forward and 0 in postrouting for both the pod CIDR and the PIA
range, against 45166 packets on the same postrouting hook for WAN traffic and
16 packets for a node-originated curl out tun0. Cilium's eBPF datapath
bpf_redirect()s these skbs to the egress device without invoking netfilter.
So `ip saddr @set oifname != tun0 drop` can never fire for vpn-class pods.
What actually fails closed is Cilium's own drop when the cached egress
interface is unusable — tcpdump during a tunnel stop AND during a restart
showed 0 packets leaving either the WAN or tun0. Every rule now carries a
`counter` so the claim stays falsifiable, and every comment that called this
table the authoritative vpn-class backstop is corrected rather than left to
mislead the next reader.
Also removed the `meta mark 0x1c8 oifname != tun0 drop` rule and its variable:
nothing stamps that mark (the only Cilium mark on the node is 0x200/0xf00, the
L7-proxy rule to table 2004), the traffic never reaches netfilter anyway, and
a live rule matching an arbitrary mark is a hazard to whatever picks it next.
3. The up-script flushed the nft source set on every connect, so a packet still
sourced from the previous tunnel address matched neither the ip-rule nor the
drop rule. The set is now additive (`add element` is idempotent; the
down-script already retained it): an element can only ever tighten the guard.
Finding 2 means that window could not have leaked POD traffic, but
node-originated tunnel traffic does traverse netfilter, and additive is free.
Kill-switch reloads no longer disarm the guard either: the .nft file uses
`table` + `flush table`, which clears rules but leaves the named set's elements
intact, so an Ansible run cannot empty the set mid-flight.
Co-Authored-By: Claude Opus 5 <[email protected]>
platform/monitoring/alerts.yaml has shipped VeleroBackupFailure (critical),
VeleroBackupPartialFailure and VeleroNoRecentBackup since the observability
wave. None of them could ever have fired. The velero chart defaults
`metrics.serviceMonitor.enabled: false`, and prod confirms it live:
`kubectl -n velero get servicemonitor` returns no resources, so
velero_backup_failure_total and velero_backup_last_successful_timestamp were
never scraped. A nightly backup could fail — or silently stop running — for
weeks with nothing to say so. That is exactly what R-STATE-03 is for, which
makes the scrape part of the backup path rather than an observability extra.
Both flags verified against chart 12.1.0: `helm template` emits neither object
without the monitoring.coreos.com/v1 capability, and emits both PodMonitor and
ServiceMonitor with it — so the chart gates on capability detection AND
`enabled`, and helm-controller supplies real capabilities from the API server.
No additionalLabels are needed: platform/monitoring already sets
serviceMonitorSelectorNilUsesHelmValues and podMonitorSelectorNilUsesHelmValues
to false, so Prometheus selects every monitor regardless of labels.
Independent of the R2/bulk-tier fix in the previous commit; droppable on its own
if the PM wants to sequence it separately.
Co-Authored-By: Claude Opus 5 <[email protected]>
Velero was resumed on prod (live `flux resume`; suspension was never in git) and
its BackupStorageLocation validated Available against Cloudflare R2 on the first
try — bucket k3s-prod-backup, prefix `velero`, region auto, checksumAlgorithm "".
Backups still failed, and a second, unrelated hazard was armed. Both are fixed
here; both were measured on prod, not reasoned about.
1. velero-plugin-for-aws v1.14.2 cannot write to R2 at all.
It sets PutObject's `Tagging` field unconditionally, so the SDK emits an empty
`x-amz-tagging` header on every PUT. R2 does not implement object tagging in
any form and answers HTTP 501 NotImplemented. Two separate Backups reproduced
it on the write of velero/backups/<name>/velero-backup.json: kopia uploaded
the volume data fine (it uses its own S3 client), then the backup went
phase=Failed with no metadata object — i.e. unrestorable. No config avoids it;
`config.tagging` with a real value sends a tag set R2 also rejects. Upstream
PR #309 makes the field conditional and landed on release-1.14 after v1.14.2
was cut, so v1.14.3-rc.2 is the earliest build with the fix. Pinned to it,
with the exit condition (v1.14.3 GA) stated at the pin.
2. The opt-out FSB schedule would have uploaded the NFS bulk tier.
Schedule (1) runs defaultVolumesToFsBackup:true and excludes only monitoring
and unifi. Since 2026-09-11 prod also carries four static NFS PVs —
plex-media, sonarr-media and radarr-media all bound to the SAME export
/mnt/Storage/plex (8.32 TiB measured via zfs list), plus
nextcloud-legacy-archive at 270 GiB. The 03:00 run would have read that
library three times over NFS and pushed ~8.6 TiB to R2 nightly, retained
720h. Namespace exclusion cannot express the fix: those four namespaces also
hold real Longhorn PVCs (plex/sonarr/radarr config, nextcloud-data) whose
loss is unrecoverable.
So the opt-out is now per VOLUME, by SOURCE TYPE: a velero volume policy
(ConfigMap velero-volume-policies) skips any volume whose source is NFS, and
schedule (1) references it. That tests the property that makes a volume
bulk-tier (R-STATE-04: the bulk tier IS the NFS export) instead of a name list
the next PV would slip past. The per-pod backup-volumes-excludes annotation
was rejected for failing OPEN — a new NFS mount would be backed up in full
until someone remembered.
PROVEN on prod (velero 1.18.1 / chart 12.1.0): a Backup of ns nextcloud with
defaultVolumesToFsBackup:true plus this policy produced exactly ONE
PodVolumeBackup — `data` (nextcloud-data, Longhorn, 875,417,102 bytes,
Completed). The NFS `legacy-archive` volume produced none.
Residual gap, stated rather than hidden: the rule matches NFS only. A future
bulk volume on another out-of-cluster source needs its own condition.
Co-Authored-By: Claude Opus 5 <[email protected]>
The four forward-auth media UIs (jackett, qbittorrent, radarr, sonarr) gate on
`--allowed-group=media-users` alone. On prod the `homelab` realm has exactly one
user — the operator's `jakman85` — and it is a member of `platform-admins` ONLY.
`media-users` has zero members, so those four hosts were reachable by nobody:
oauth2-proxy completed OIDC discovery, redirected to Keycloak, got a token whose
`groups` claim is `["platform-admins"]`, found no allowed group in it, and
answered 403. No login form appeared because the browser already held a Keycloak
SSO session from a platform-admins login — hence "403 immediately".
pihole and crafty were unaffected: they already gate on platform-admins.
`--allowed-group` is repeatable and OR-ed, so listing both groups means "media
consumers OR platform operators". This grants no real privilege — platform-admins
already maps to cluster-admin via k8s-oidc-rbac.yaml (oidc:platform-admins), so a
member can already exec into these pods — and it keeps media-users meaningful for
non-admin household accounts.
Ruled out while diagnosing: client-secret drift (all 7 sidecar secrets hash-match
keycloak-realm-clients), stale apex URLs (every live sidecar carries
keycloak.prod.home.keithjackson.dev), missing group mappers (all 7 clients emit
claim=groups, full.path=false), and the central oauth2-proxy pod restarts — those
were Keycloak returning 503 "no healthy upstream" to OIDC discovery at startup, a
consequence of the Keycloak roll, not a cause.
Goldens re-rendered with the canonical command:
python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation
R-IDP-06
Co-Authored-By: Claude Opus 5 <[email protected]>
sonarr, radarr, jackett and qbittorrent all mounted their Longhorn config PVC at
the renderer's default /data/config, but every one of these images runs with
`-data=/config`. So the PVC held nothing but lost+found while the app wrote its
entire configuration to the container's EPHEMERAL writable layer — destroyed on
every pod recreation: restart, reschedule, image bump.
Measured in-cluster before fixing, per app, rather than inferred:
/data/config dev=2080 [lost+found] <- the Longhorn PVC
/config dev=2053 [Sentry asp config.xml logs ...] <- the node's own disk
and a marker file written to /config did not appear in /data/config, so they are
demonstrably different storage. qbittorrent's /config sat on dev=66309
(/dev/nvme0n1p5), i.e. whichever node it happened to land on.
This is the SAME failure catalog/plex.yaml:54 already records having fixed for
Plex; plex was the only one of the five that was right.
The live /config contents were copied onto each PVC BEFORE this landed, so
jackett's API key and qbittorrent's credentials — which other apps reference —
survive the move rather than being regenerated.
catalog/sonarr.yaml:61 documented this as a latent bug and deliberately left it
because it changes live on-disk state. It is being fixed now because all four pods
had just been recreated, so their configs were freshly generated and nothing had
accumulated to lose.
Co-Authored-By: Claude Opus 5 <[email protected]>
The operator added the pfSense port-forward, so BULK_NFS_SERVER is no longer a
placeholder. Prod gets 192.168.2.102 (pfSense WAN, DNAT tcp/2049 -> 192.168.1.9);
dev stays on the base 192.168.1.9 because it is already on pfSense's LAN.
This value could not be left as a placeholder safely: spec.nfs.server is
unvalidated by Kubernetes, so the PV would be created, the PVC would BIND, and the
failure would only surface as four pods stuck in ContainerCreating.
Verified from k3s-prod-1 before committing, deliberately as uid 1000 rather than
root (the export carries no_root_squash, so a root write proves nothing, and LSIO
images default to PUID 911 which would go Ready and then fail every import EACCES):
mount negotiated nfs4 vers=4.1 proto=tcp, write succeeded, and ln() across
downloads/ -> tv/ gave link count 2 — real hardlinks, so no silent copy-then-delete.
The Nextcloud export was additionally tightened to server-side ro= for prod's three
nodes; a write as root is refused. Its content is config/, config_old/ and
config_backup/ with no data/ directory, so it is an archive, not a live data store.
Co-Authored-By: Claude Opus 5 <[email protected]>
The gate BLOCKED this branch, correctly, and the block was the interesting part.
Attaching the media library selects R-STATE-04 as a touched requirement ("bulk media storage has
a capacity strategy distinct from the replicated tier"), and R-STATE-04 had NO scaffolded test, so
the coverage REQUIRE stage failed both l0 and l1. There were two ways out. One was to keep
avoiding the token — the existing discipline in catalog/plex.yaml and catalog/qbittorrent.yaml,
which say in as many words that naming the requirement would block the merge. That made sense while
the tier was a stub. It does not make sense in the change that DELIVERS the tier: it would mean
shipping the requirement and claiming nothing. So: the check.
WHAT IT ASSERTS, and why each one is a real failure mode rather than a shape
The `bulk` StorageClass sat as `server: REPLACE-ME` for six roadmap phases, and the only thing
keeping a multi-terabyte library off the 3-replica `longhorn` class was four hand-written comments.
That is the class of failure being closed: a bulk volume on the replicated tier does not fail
loudly — Longhorn accepts the PVC and starts replicating 3x of data three mini-PCs cannot hold.
(1) DISTINCTNESS — no bulk PV/PVC on any class the repo declares with the Longhorn provisioner
(derived from platform/longhorn/, so adding a Longhorn class cannot create a blind spot),
and the class is explicitly EMPTY.
(2) CAPACITY STRATEGY as arithmetic — a bulk volume at least an order of magnitude above the
same service's replicated volume. The requirement is about two magnitudes of data; a "bulk"
volume sized like a config PVC means the tier was named, not applied. The 10x floor is
CALIBRATED against the real corpus, not guessed: the media library is ~460x its config
volume and the Nextcloud archive ~13x its 20Gi, so a higher floor would fire on a legitimate
volume and teach the next reader to weaken the check. My first draft used 100x and did
exactly that — the corpus caught it.
(3) PRE-BOUND FROM BOTH ENDS (claimRef + volumeName, agreeing). Guards the most deceptive
failure available in this tier: a dynamic bulk class hands out a FRESH EMPTY directory, the
pod mounts, the PVC Binds, everything is green, and the 8 TiB was never attached.
(4) Retain, always. (5) mountOptions carry `hard` and exactly one nfsvers=4.x, and never `soft`
— a soft mount hands a server hiccup to the app as a short read, which for a media library
is silent corruption; and v4 is what keeps the firewall surface at one port.
(6) ONE EXPORT, AT MOST ONE MOUNT PER WORKLOAD. This is the assertion I most wanted. Splitting
one export into two subPath mounts looks TIGHTER and breaks link() with EXDEV; Sonarr and
Radarr then degrade to copy-then-delete, rewriting every import over NFS and destroying the
seeding copy, surfacing weeks later as "imports are slow and my ratio is gone". No reviewer
sees that in a diff.
(7) READ-ONLY COUPLING — nfs.readOnly, the volumeMount flag and ReadOnlyMany must agree, so
dropping one still leaves the others; and every bulk mode is a `...Many`.
(8) The server is a BARE cluster-vars token, that knob has a base default, and EVERY overlay
supplies it — the seam that lets one authored definition serve two clusters that reach this
host by different addresses.
(9) LOAD-BEARING and NON-VACUOUS. Wired unconditionally into ci/l0.sh as Stage 3e for the same
reason as 3c/3d: Stage 3b scopes to touched requirements, and the path map keys NEITHER
apps/* NOR platform/longhorn/* to this requirement — so the change most likely to break
this could attribute nothing and be gated by nothing. An EMPTY bulk corpus FAILS rather
than passing trivially, and a floor on the assertion count keeps a pass from degenerating.
It reads the COMMITTED apps/ goldens rather than calling the renderer. Asking the renderer would
only prove it agrees with itself; the goldens are what Flux reconciles, and render==golden is
already its own gate stage.
PERTURBED, because a check that has never failed is not evidence. Each of these was applied to the
real tree and each fired with the intended message: bulk PVC moved onto `longhorn`; Retain -> Delete;
`hard` -> `soft`; the token replaced by the literal 192.168.1.9; sonarr's single mount split into
two subPath mounts (assertion 6); and the prod overlay's knob deleted.
Also hardens render.py: `size` is structurally required of a bulk entry (a PV with no capacity is
rejected by the API server) but the JSON Schema only enforces `export`, so omitting it raised a bare
KeyError three frames down. It now raises a sentence naming the service, the volume and the remedy.
Found by the same perturbation pass.
Coverage: 48/51 -> 49/51 (96.1%), 2 gaps, 0 safety-critical.
BASE_REF=main make -C ci gate -> MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 <[email protected]>
Three instances replicating ASYNCHRONOUSLY still lose data on failover: COMMIT
returns as soon as the primary has flushed locally, so a primary that dies
before its standbys catch up takes the last few transactions with it and the
promoted standby comes up quietly missing them. On keycloak-db that is a
just-created user, a rotated credential or a consent vanishing at exactly the
moment of a failover.
`method: any` + `number: 1` makes COMMIT wait for ONE of the two standbys to
confirm — the point of having them.
dataDurability: preferred, NEVER required. `required` is a strict quorum:
fewer than `number` healthy standbys and writes BLOCK. Postgres here shares
three nodes with etcd and the CNI, so one degraded standby — a node reboot, a
Longhorn replica rebuild, a Cilium restart — would turn a partial degradation
into a total write outage, and on the SSO database that takes kubectl OIDC,
Grafana, Headlamp, Gitea and forward-auth with it. `preferred` keeps
synchronous commit while a healthy standby is there and relaxes the required
count dynamically when one is not. dataDurability is only honoured while
standbyNamesPre/standbyNamesPost are unset — both are.
NON-DISRUPTIVE: this compiles to `synchronous_standby_names`, a reload-only
PostgreSQL parameter. No pod restart, no switchover, no client disconnect.
It is the last of the three commits because it wants standbys that already
exist and are already on their final nodes.
Field shape verified against the live clusters.postgresql.cnpg.io CRD before
relying on it (CNPG 1.30.0, byte-identical schema on k3s-prod and k3s-dev):
spec.postgresql.synchronous requires `method` (enum any|first) and `number`
(x-kubernetes-validation self > 0); `dataDurability` is enum required|
preferred. None of the three Clusters previously had a `postgresql:` block,
so this adds one rather than merging into existing parameters.
DEV IMPACT: identical and non-disruptive, same reload-only path.
Verified: yamllint clean; both tiers render the block.
Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit f1ce19461415e77eb2f1ca5109ebab49edf0860b)
plex, sonarr, radarr and nextcloud each gain one bulk volume. Authored in the catalog and
re-rendered, never hand-edited into apps/ (render-golden: 10/10). Layout, the export change the
operator must apply, the uid analysis and the validation probes are in docs/BULK-NFS-STORAGE.md;
everything there was measured on the host over read-only SSH, not assumed.
WHAT THE LAYOUT TURNED OUT TO BE, AND WHY IT DECIDED THE DESIGN
/mnt/Storage/plex holds movies/ tv/ photos/ AND downloads/ as SIBLINGS in one dataset — so the
arrs' import source and their libraries are one filesystem on the server, and the download client's
completed folders are already named radarr/ and tv-sonarr/. The k3s arrs are joining a tree an arr
stack has been driving for a long time.
That sibling fact is the whole argument for the arrs mounting the export ROOT rather than a
subPath per directory. Sonarr and Radarr import by hardlinking or atomically renaming, both
same-filesystem operations, and to the kernel "same filesystem" means SAME MOUNT. Two subPath
mounts of one export are two mounts: link() across them returns EXDEV regardless of what the
server knows, and the arrs do not fail loudly — they degrade to copy-then-delete, rewriting every
file over NFS and destroying the seeding copy. The smallest subtree containing both tv/ and
downloads/ IS the export root, so the root mount is the MINIMAL mount that preserves hardlinks.
Least privilege bounded below by a hard requirement. The cost is stated, not hidden: sonarr can
also write /media/movies.
plex /mnt/Storage/plex -> /media ro ReadOnlyMany
sonarr /mnt/Storage/plex -> /media rw ReadWriteMany
radarr /mnt/Storage/plex -> /media rw ReadWriteMany
nextcloud /mnt/Storage/nextcloud -> /mnt/legacy-nextcloud ro ReadOnlyMany (subPath: nextcloud)
Plex is read-only because it indexes and streams; every write on that tree belongs to the arrs, and
saying so removes Plex from the set of things that can damage 8.33 T. All three media consumers use
the SAME mountPath deliberately: when qbittorrent is eventually given this export, the arrs and the
download client will already agree on absolute paths, which is the other precondition for hardlinked
imports.
*** /media AND NOT /data — A COLLISION CAUGHT WHILE RENDERING. *** The arrs' Longhorn config PVC
renders at the renderer default /data/config. A media mount at /data is mounted OVER it and MASKS
the config volume: the pod comes up with its app database shadowed by an 8 T media tree. That
discovery also exposed a SEPARATE PRE-EXISTING BUG, reported and deliberately NOT fixed here:
LinuxServer arr images run with -data=/config, so /data/config is very likely the wrong home for
that PVC and the app has been writing its EPHEMERAL layer — the exact failure catalog/plex.yaml's
mountPath comment records already having been bitten by. Plex was fixed; sonarr, radarr, qbittorrent
and jackett were not. Fixing it starts those apps from an empty config volume, so it needs its own
change with the operator present. This change does not make it worse.
UID/GID — THE FAILURE THIS TIER ALWAYS HAS
The export is sec=sys + no_root_squash, so the client's numeric uid passes through verbatim and the
server's ownership is the entire access story. The tree is 0755 owned 1000:3000, so ONLY uid 1000
can write it. LinuxServer images default to 911. Without intervention sonarr and radarr would have
mounted cleanly, gone Ready, and failed EVERY import with EACCES.
And fsGroup cannot fix that — the in-tree NFS plugin does NO ownership management, so
podSecurityContext.fsGroup is a no-op on an NFS volume. Reaching for it is how this gets
misdiagnosed. So PUID=1000/PGID=3000 are added to both arrs (PGID 3000 keeps what they write
group-readable to the live non-k3s consumers of the same export), and fsGroup: 3000 is added for
the LONGHORN config PVC only, so changing PUID does not depend on the image re-chowning /config.
plex already runs as 1000 and reads a 0755 tree; nextcloud reads as www-data. Neither needs work.
NEXTCLOUD IS MOUNTED READ-ONLY, WHICH IS A DEVIATION FROM THE BRIEF — ON EVIDENCE
The task said nextcloud writes. The dataset says otherwise: of 270 G, 267 G is config_backup/, the
rest is config/ (103 M) and config_old/ (3.4 G). There is NO data/ directory. This is the OLD
non-k3s instance's configuration plus a backup blob — an archive — and that instance is STILL LIVE
on it (exported rw to 192.168.1.6 and .5, which the operator has said must stay active). Two
Nextclouds writing one instance's config directory, each with its own database and therefore its own
oc_filecache, is unrecoverable corruption. "Do not grant more than needed" lands on ro, and ro is
the trivially reversible choice. If the intent is in fact a MIGRATION, that is a larger and more
deliberate change than a mount; the one-line flip is documented.
Read-only for nextcloud can additionally be enforced SERVER-side (only nextcloud consumes that
dataset from k3s), and the sharenfs value in the doc uses ro= for the k3s clients. The media tree
cannot: plex and the arrs may land on any node, so the server sees the same addresses for reader
and writers. Plex's ro is therefore a client-side guarantee — stated as a limitation, not left
implied.
Co-Authored-By: Claude Opus 5 <[email protected]>
`instances: 3` alone does not keep the three instances apart. All three
Clusters used CNPG's default `podAntiAffinityType: preferred`, which is a
scheduler HINT — two instances may share a node whenever that scores better,
and they WILL on the path that matters: when CNPG recreates an instance after
a node loss, the surviving nodes are the only candidates and one of them
already runs an instance. `preferred` is how a 3-instance cluster decays back
into the single-node failure domain the previous commit exists to remove.
`required` makes it a scheduling constraint. On a 3-node cluster with no
spare, the cost is that an instance whose node is gone stays Pending until a
node returns. That is correct: a replacement squeezed onto an already-occupied
node dies with that node while making the cluster LOOK three-way replicated.
2 of 3 instances still serve and CNPG still promotes.
ALSO SETS primaryUpdateMethod: switchover. This is not cosmetic. Changing the
affinity changes the instance pod spec, so CNPG performs a rolling update, and
the CRD default for primaryUpdateMethod is `restart` (verified against the
live CNPG 1.30.0 CRD on both clusters) — a delete-and-recreate of the PRIMARY
pod, i.e. tens of seconds of no Postgres. On keycloak-db that is an SSO
outage, which also means kubectl OIDC, Grafana, Headlamp, Gitea and
forward-auth. With standbys now present, `switchover` promotes an
already-updated streaming replica instead, reducing the roll to a connection
reset. Known constraint documented in the manifest: under `switchover` the
operator rejects an update that changes imageName and a postgresql parameter
in the same reconcile.
Verified satisfiable before merge (read-only): k3s-prod and k3s-dev each have
3 Ready, schedulable, untainted nodes, and every pgdata PVC already has a
Longhorn replica on all three (replica-soft-anti-affinity=false), so any
instance can attach from any node.
DEV IMPACT: identical, and dev can absorb it. Longhorn dev nodes are 124.9 GiB
each with 152.0 GiB already scheduled under a 200% over-provisioning ceiling
(249.8 GiB); the six new 5Gi volumes add 30 GiB scheduled per node -> 182.0,
still inside the ceiling, and only ~600 MiB per volume is really consumed.
Prod is far looser: 239 -> 269 GiB scheduled against a 829 GiB ceiling.
THIS IS THE DISRUPTIVE COMMIT of the three. Land it on its own reconcile,
after the scale-out has produced two Ready standbys per Cluster, so the blast
radius is one clean switchover per database rather than a scale-out and a roll
at once. Expect a brief SSO interruption for keycloak-db.
Verified: yamllint clean; both tiers render the new fields; field names,
types and enum values checked against the live clusters.postgresql.cnpg.io
CRD (CNPG 1.30.0, identical on prod and dev).
Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit e1d364238d73e6ee99c9a50108fe8b4e09481a7e)
The bulk tier's PersistentVolumes need a server address, and there is no single correct one: the
two clusters sit on OPPOSITE SIDES of pfSense from the storage host, which has exactly one
interface (vmbr0 192.168.1.9/24, on pfSense's LAN).
dev 192.168.1.40/.41/.42 — the SAME segment. Reaches it directly, and already holds one NFS
export from it (/mnt/Storage/k8s-volumes, in the host's /etc/exports). Proven path, so
base's default is the dev value per this file's existing convention, and dev restates it.
prod 192.168.2.10/.11/.12 — pfSense's WAN. tcp/2049 does not traverse that today; 22 and 8006
answer only because pfSense port-forwards them via 192.168.2.102. Opening the storage path
is a PENDING operator decision whose two candidate outcomes give DIFFERENT values here:
192.168.2.102 (another port-forward) or 192.168.1.9 (a route). So prod holds
TBD-PROD-BULK-NFS-SERVER, this repo's established loud-placeholder convention.
Two findings recorded in the prod overlay because they change what the operator has to do:
* THE EXPORT'S CLIENT LIST IS THE SAME UNDER BOTH OUTCOMES. Measured today from this
workstation (192.168.2.132) through the existing port-forward: the storage host logged the
connection as coming from 192.168.2.132 — a WAN-side address — not from a pfSense LAN address
(`ss -tn` peer 192.168.2.132; the host's only route off-LAN is its default via 192.168.1.1).
pfSense DNATs WITHOUT source NAT, so the NFS server sees prod's REAL node addresses either
way. The `zfs set sharenfs=` work therefore does not have to wait for the routing decision.
* ONE PORT, NOT FOUR. The PVs mount nfsvers=4.1, and NFSv4 carries mount and lock in-protocol —
no rpcbind/111, mountd or statd. Whichever outcome is chosen, it opens tcp/2049 and nothing
else. NFSv3 would have needed four ports, three of them on ephemeral numbers.
*** AND ONE HONEST WEAKNESS, stated in the overlay where an operator will hit it. ***
Unlike the LAN knobs above it, this placeholder does NOT fail at apply time. A PV's
spec.nfs.server is an unvalidated string: the API server accepts it, the PV is created, and the
PVC BINDS (binding is pure control-plane). The failure appears only when a pod mounts —
MountVolume.SetUp "Failed to resolve server TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr
and nextcloud would sit in ContainerCreating on prod. There is no CRD validation to lean on here,
so the guard has to be procedural: set the value before prod reconciles a revision carrying the
bulk volumes. Dev renders a working value and is where these manifests are meant to be proven.
Related finding, NOT fixed here because the file does not exist on main: the static guard this
convention is supposed to lean on, `bootstrap/standup.sh`'s `s3_is_placeholder` family, is absent
from main (it lives on unmerged prod branches), so
conformance/tests/l0-static/placeholder-guard-coupling exits 3 SKIPPED and nothing in the gate
currently refuses a placeholder-shaped overlay value. When standup.sh lands, its guard needs an
arm for this knob and a consumer that calls it — assertions (2) and (3) of that check then cover
this value automatically, since the detector's SEED vocabulary already matches "TBD".
Evidence: `kubectl kustomize clusters/dev` -> BULK_NFS_SERVER: 192.168.1.9;
`kubectl kustomize clusters/prod` -> BULK_NFS_SERVER: TBD-PROD-BULK-NFS-SERVER. Both exit 0.
Co-Authored-By: Claude Opus 5 <[email protected]>
The `bulk` class has existed since Phase 0 as a deliberate stub: the StorageClass named an NFS
CSI driver that is not installed and a server/share of REPLACE-ME, so plex, sonarr, radarr and
qbittorrent all carry a comment saying "attach the library once OQ-2 is resolved". This is that
resolution — for the case that actually exists here, which is NOT the case the stub anticipated.
The stub assumed DYNAMIC provisioning: ask a CSI driver for a bulk volume and get a fresh empty
subdirectory. That is the wrong shape for this platform's bulk data. The media library is 8.33 TiB
that already exists, on ZFS datasets on the Proxmox host, with live non-k3s consumers. Nothing
needs provisioning; existing datasets need ATTACHING. So a bulk storage entry now renders a
STATIC, pre-bound NFS PersistentVolume plus its PVC, and needs no CSI driver at all (the in-tree
NFS volume source is not deprecated and has no CSI migration).
Schema 1.4.0 adds, on a storage entry:
export absolute NFS export path; REQUIRED when class is bulk, FORBIDDEN otherwise (an `if`
in the schema, so a bulk entry cannot be authored without saying what it attaches).
subPath mount a subtree instead of the volume root — any class.
readOnly any class; on bulk it also sets the PV's nfs.readOnly, so the mount really is `ro`.
The SERVER is deliberately NOT a schema field. It is the `${BULK_NFS_SERVER}` cluster-vars knob,
because dev and prod reach the SAME storage host by different addresses and R-ENV-02 says the
service definition is authored once. A ServiceDefinition says WHICH dataset; the overlay says
where the host is.
Three decisions worth the reader's time, argued in full in the r_bulk_pvs docstring:
* PV/PVC, not an inline `nfs:` pod volume. Inline is fewer objects and would work. It cannot
express `persistentVolumeReclaimPolicy: Retain`, which is the one property most worth having
in git for 8.33 TiB of data with consumers outside this cluster; it puts read-only enforcement
one Deployment edit away from silent removal; and it is invisible to `kubectl get pv,pvc`,
which is where an operator looks for attached external storage.
* storageClassName: "" on both ends, NOT the `bulk` StorageClass. Explicitly empty is the
canonical marker for a statically provisioned volume. It also sidesteps a trap: a
StorageClass's `provisioner` is IMMUTABLE, so re-pointing the deployed `bulk` stub would make
Flux fail on that object forever.
* Access mode is always `…Many` and follows the read/write posture — ReadOnlyMany for a reader,
ReadWriteMany for a writer. One export is consumed by several namespaces AND several nodes at
once, which is exactly what ReadWriteOnce cannot express; which `Many` it is then makes the
intent legible in `kubectl get pv`.
mountOptions are a platform decision held in the renderer rather than a per-service field:
nfsvers=4.1,hard,noatime,timeo=600,retrans=2. NFSv4 carries mount and lock in-protocol, so the
only port that must be reachable is tcp/2049 — no rpcbind/111, mountd or statd. That is a
firewall decision as much as a storage one. `hard` over `soft` because a soft mount converts a
server hiccup into a short read handed to the application.
The contract doc records the trap that will otherwise be discovered in production: fsGroup is a
NO-OP on an NFS volume (the in-tree plugin does no ownership management), so a writable bulk mount
requires the container's own uid/gid to suit the export's ownership. A mount that succeeds and
then cannot write is this tier's characteristic failure.
Every pre-existing golden renders byte-identical: the new keys are emitted only when declared, and
`readOnly` defaults to False. render-golden: 10/10. selftest: PASS.
Co-Authored-By: Claude Opus 5 <[email protected]>
All three CNPG Clusters ran `instances: 1`. Longhorn's 3-way replication
saves the DATA when a node dies; it does not save the SERVICE — a single
instance has to be rescheduled, re-attach its volume and replay WAL before
Postgres accepts a connection again.
With nothing in the spec keeping them apart, all three primaries had drifted
onto ONE node (k3s-prod-2 on prod, k3s-dev-3 on dev). On 2026-09-07 that node
took SSO, Nextcloud and Semaphore down together, and with SSO went kubectl
OIDC, Grafana, Headlamp, Gitea and forward-auth.
Three instances give each Cluster a streaming standby CNPG can promote in
seconds, repointing the -rw Service with no volume re-attach and no restore.
UNBLOCKED BY WORKING BACKUPS, NOT A SUBSTITUTE FOR THEM. Standbys replicate
corruption and deletions faithfully; only the Barman archive is a backup, and
adding standbys while archive_mode=on with a failing archive_command would
have multiplied the disk-full clock, not hedged it. All three Clusters on
both k3s-prod and k3s-dev now report ContinuousArchiving=True with a real
firstRecoverabilityPoint (verified read-only, 2026-09-07).
Scaling out is non-disruptive on its own: CNPG creates -2 then -3
sequentially, each seeded by pg_basebackup from the RUNNING primary. The
primary is not restarted and clients are not disconnected. The anti-affinity
change that DOES cause a switchover is deliberately the next commit.
DEV IMPACT: identical — dev is on the same shape (three single-instance DBs,
all on k3s-dev-3) and gets the same scale-out. Dev has three untainted nodes
and Longhorn headroom for it (see the next commit's message for the numbers).
Cost: 2 extra 5Gi PVCs per Cluster; ~30 GiB provisioned cluster-wide against
live pgdata of ~600 MiB each, on Longhorn with 200% over-provisioning.
Verified: yamllint clean; `kubectl kustomize --load-restrictor
LoadRestrictionsNone platform-config` and `kubectl kustomize apps/wave-0`
both render `instances: 3`.
Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit ed6fe6cb019b96f677a92448a0347f979d1a6d02)
The requirement's literal reading ("no prod-specific artifact in a shared path")
forbids the environment-selected PAIR that R-ENV-02 prescribes for a value that
must differ per environment while the manifests stay identical — and a Secret's
difference cannot live in cluster-vars, because it is ciphertext in a SOPS file.
Drafted wording that keeps the guard's teeth (an artifact only prod APPLIES
still fails) and removes the false positive, for the System Architect: the spec
repo is read-only here and CI-gated by R-TEST-06.
Also recorded, checked rather than assumed: the merge that turned R-ENV-03 red
was gated with a touched-requirement subset that never selected R-ENV-03, so
ci/l0.sh Stage 3b filtered the already-red check out of the run that printed
PASS. Replaying that delivery's twelve paths through
fix/touched-requirements-fail-closed's requirement-paths.map still yields no
R-ENV-03, so landing that branch would NOT have caught this. The two changes
that would (unconditional whole-repo L0 stages; a changed conformance test
attributing its own requirement labels) are ci/ work and are left to that
branch, not smuggled in here.
The blanket regex around RENDERED_DIRS failed any tracked file whose name
carries a delimited `prod` token. That caught the four per-environment backup
credential halves (BACKUP_CREDENTIALS_SUFFIX) even though BOTH halves are
applied to BOTH clusters as base resources and the overlay selects only which
one the shared consumer REFERENCES — i.e. exactly the R-ENV-02 contract the
regex exists to protect. Renaming the files to dodge the regex was rejected as
gaming the guard, so the rule now encodes the property instead.
The fix is not an allowlist of those four paths. A prod-token file in a shared
path is excused only when the tree proves all of:
(A) it declares nothing but Secrets whose names end in the environment suffix
(1) its own directory's kustomization applies it, and EVERY environment value
of the selector resolves to a Secret that some resource of that same
kustomization declares, in the same namespace
(2) nothing names the suffixed Secret — or the suffixed file — literally;
every reference is `<base-name>${VAR}`
(3) that VAR is defined for the base and for every overlay, and prod's value
is the one that selects this file
References are read from parsed YAML values, never raw text: the mechanism is
spelled out by name in comments beside the resources, and a comment references
nothing.
No other assertion is weakened; the check still runs 96 assertions, and the
pass output now notes each excused file and why.
Records the 2026-09-07 I/O-starvation incident and the drive swap: the C400 was
replaced with a Samsung 850 EVO, and under the same load class (an active
18-replica Longhorn rebuild, with node 1 as the write target) I/O pressure fell
from full avg300 11.36 to 3.79 — from ~4x worse than its peers into node 3's band.
Updates the three §0 cells that had become false (disk model, drive health, kernel
— all three nodes now run 7.1.8, not 6.12) and adds §0c rather than rewriting the
'NOT hardware-symmetric' analysis, which is still the correct reasoning about the
state it described.
Two things the rebuild taught, recorded so the next one is cheaper: Longhorn
refuses a replaced disk with DiskFilesystemChanged and needs disable -> remove ->
re-add in that order, and the inventory's [k3s_prod_primary] decides BOTH
--cluster-init and k3s_api_host, so rebuilding the founder without moving that
designation would have founded a second cluster.
The replacement does NOT close §2: etcd and Longhorn still share one SATA spindle.
Co-Authored-By: Claude Opus 5 <[email protected]>
The unit name in the swap-active detail read as an unlabelled aside. Naming it as
the unit that activated the swap is what makes the line actionable.
Co-Authored-By: Claude Opus 5 <[email protected]>
The claim guard builds node_baseline_lh_protected_devices from ansible_mounts plus
`swapon --show`. With swap.target masked, a baselined node has no active swap, so
that half of the list contributes nothing — which reads like a guard that quietly
stopped guarding. It is not: the claim candidate comes from whatever is mounted at
node_baseline_longhorn_claim_mountpoint, and a swap partition is neither mounted nor
ext4, so it could never be the candidate. Say so where the next reader will ask.
Co-Authored-By: Claude Opus 5 <[email protected]>
The WARN read "roles/node_baseline runs swapoff and neutralises the fstab entry;
this node has not been baselined yet". That was a MIS-DIAGNOSIS. All three prod
nodes HAD been baselined, had their fstab swap line commented out by the role, and
were still swapping 977 MiB each after a reboot. Anyone reading that WARN would
have re-run the role that had already run, and watched swap come back next boot.
The detail now names the actual mechanism — systemd-gpt-auto-generator re-creating
a .swap unit from the partition TYPE GUID 0657fd6d-a4ab-43c4-84e5-0933c84b4f4f at
every boot, without reading /etc/fstab — says in as many words that this is not
"the node has not been baselined", and points at `node-baseline.yml --tags system`.
The check was also blind in the other direction: SwapTotal == 0 is not evidence
that swap is disabled, only that nothing has been swapped on yet this boot, so a
node freshly swapoff-ed PASSed while still set to swap again on reboot — exactly
the state the old role left behind. The collector now reports memory.swap_persistence
(swap.target masked?, .swap units the generators emitted this boot, which of them
are masked, leftover active fstab swap lines — all world-readable, no root, no
systemctl parsing), and the check WARNs on "clean now, swapping after the next
reboot".
Rendered against every fact shape (swap active + unmasked, swap off + unmasked,
after the fix, a dev VM with no swap partition at all, per-unit mask only,
persistence unreadable, an older collector's output with the key absent, and
/proc/meminfo unreadable) and verified live on k3s-prod-1 post-reboot:
[PASS] memory.swap — no swap active and none can be activated: swap.target is masked
Co-Authored-By: Claude Opus 5 <[email protected]>
The role's `swapoff -a` + commented fstab line held only until the next boot,
on every prod node. Measured 2026-09-07: all three came back from a reboot with
977 MiB of swap ACTIVE (k3s-prod-1 /dev/sda4, k3s-prod-3 /dev/sda4, k3s-prod-2
/dev/nvme0n1p4) while their fstab swap entry was commented out by this role. The
role reported success and the setting silently expired.
CAUSE. systemd-gpt-auto-generator claims any GPT partition typed
0657fd6d-a4ab-43c4-84e5-0933c84b4f4f ("Linux swap") from the PARTITION TABLE,
synthesises a .swap unit into /run/systemd/generator.late/ and links it into
swap.target.wants/. It never reads /etc/fstab, so neutralising fstab cannot stop
it. Re-running the generator by hand on k3s-prod-1 says so verbatim:
Probed fstype 'swap' on partition /dev/disk/by-diskseq/1-part4.
Adding swap: /dev/disk/by-diskseq/1-part4
MECHANISM. Mask swap.target. `systemctl show` on the generated unit says its only
inbound dependency is `WantedBy=swap.target` — Requires/RequiredBy/BoundBy empty,
and the matching .device unit has an empty Wants=, so a device appearing does not
pull its swap unit in by itself. swap.target is the single choke point for every
activation path: gpt-auto, systemd-fstab-generator, systemd-cryptsetup-generator.
It is device-shape independent (no device name appears in the change, so one task
covers sda4, nvme0n1p4 and dev's vda VMs), boot-safe (swap.target is Wants=/After=
from sysinit.target and After= from tmp.mount; RequiredBy= is EMPTY on all three
prod and all three dev nodes, and a masked unit reached only via Wants= is skipped,
not failed), and inert if the device is absent or renamed. The generated units are
masked too, as a second independent barrier.
Rejected: `systemd.gpt_auto=0` (disables GPT auto-discovery wholesale, needs a GRUB
edit + reboot to become true, and on DEV would switch off root auto-discovery — dev's
vda1 carries the discoverable GUID 4f68bce3-e8cd-4db1-96e7-fbcaf984b709); clearing
the partition's GPT type GUID (a partition-table write to the live boot disk of a node
serving etcd and Longhorn replicas, and an irreversible decision about that 977 MiB
that belongs to the operator); `systemd.swap=0` (systemd-fstab-generator only — this
swap does not come from fstab).
`swapoff -a` is now conditional on `swapon --show` returning something, so the task
reports honestly instead of hiding a command that ran anyway behind changed_when: false.
PROVEN on k3s-prod-1 (all 18 Longhorn volumes healthy before the reboot):
run 1 changed=2, run 2 changed=0
after `reboot`: /proc/swaps empty, `free` Swap 0B, swap.target masked, the
generated unit still emitted by the generator and still masked, `systemctl
--failed` empty, `systemctl is-system-running` = running.
Co-Authored-By: Claude Opus 5 <[email protected]>
Node 1's SSD failed and was replaced; Debian was reinstalled, so node 1 is now a
JOINING node with no cluster state. It was still listed in [k3s_prod_primary],
and that group decides two things at once:
1. which play gets k3s_server_mode: primary -> --cluster-init, which FOUNDS A
NEW CLUSTER, and
2. what k3s_api_host resolves to -- the --server join target.
So re-provisioning node 1 unchanged would have pointed a rebuilt, empty node at
ITSELF and founded a second cluster beside the live one. The role's
`creates: /etc/rancher/k3s/k3s.yaml` guard does not help: on a fresh install that
file is absent, so cluster-init WOULD have run.
Node 2 is the senior surviving server (up since 2026-09-01) and takes the founder
slot. Its own cluster-init task is inert precisely because k3s.yaml already exists
there. Verified after the change: k3s_api_host renders 192.168.2.11, and
groups[k3s_primary_group][0] is k3s-prod-2.
This is a latent gap the role documented but did not handle -- its comment assumes
the founder is always alive ('On a WARM single-node rejoin the founder is up and
answering too'). Rebuilding the founder itself was not covered.
Co-Authored-By: Claude Opus 5 <[email protected]>
infra/PROD-PROVISIONING.md §2 recorded the etcd/Longhorn shared-partition risk as
"accepted to be MEASURED, not assumed", with an exit condition. The measurement
arrived on 2026-09-07 in production, 2 d 7 h after bring-up, as an outage.
§2 now carries "THE MEASUREMENT CAME BACK": the journal evidence, the three-node
IO PSI table (11.36 / 0.75 / 1.29), and the honest verdict that the section's own
premise — "every prod node is SSD or NVMe, where rebuild I/O costs latency rather
than collapsing it" — was right about node 2 and wrong about node 1. A 2011 C400
with 61,440 reallocated sectors and no power-loss protection is not the "SSD" that
reasoning assumed, and §0 had already named the three-way drive asymmetry as the
fault line. The exit condition is OPEN and the second bullet (dedicated device)
applies; that remains the operator's.
Also records what the three software mitigations in this branch do and do not buy,
including the point that no cgroup or scheduler control arbitrates the ext4 journal
both tenants share — and the gap that etcd's own fsync histograms, which the exit
condition literally asks for, are not collected because kubeEtcd is disabled for
k3s. PSI is the signal that exists.
DECISIONS-LOG gains the full entry: the four I/O mechanisms found INERT on these
nodes (IOSchedulingClass, IOSchedulingPriority, io.latency, bare IOWeight), the
finding that concurrentReplicaRebuildPerNodeLimit was already set and already held
during the outage, the third instance of config that looked applied and was not
(orphanAutoDeletion), and the three things still requiring the operator.
infra/ansible/README.md: CORRECTED a stale claim that k3s-prod-2/-3 "stay commented
out" of the inventory. They have been live in [k3s_prod_secondary] since
2026-09-03. That sentence is exactly the kind used to reason about blast radius —
`-e k3s_hosts=k3s_prod` targets THREE hosts, not one — so it is replaced with the
--list-hosts command that settles it. The new etcd-io-priority play is added to the
run order, the layer table and the prod command list.
Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit a00a27a2567ea6a805c8c196b75df2c92a7c63a4)
Self-review of the role I had just written. On k3s-prod-2 the scheduler path is
deliberately skipped, so the `stat` on /sys/fs/cgroup/system.slice/k3s.service is
skipped too — and a SKIPPED registration has no `.stat` key at all. Ansible
templates `loop:` BEFORE it evaluates `when:`, so
loop: "{{ ['system.slice'] + (['system.slice/k3s.service']
if _eio_k3s_cgroup.stat.exists else []) }}"
raises on that host even though the task would never have run. The role would have
failed on the one node it is designed to leave alone — and it would have failed at
the very end, after doing nothing, which reads like a broken mitigation rather than
an intentional skip.
`| default(false)` on both uses. Exercised offline against a simulated skipped
registration; `--syntax-check` re-run.
Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit 5ccf5111a54857abb934a64c41332b7aa5046323)
roles/etcd_io_priority + playbooks/etcd-io-priority.yml, enabled for [k3s_prod]
only. Mitigation for the 2026-09-07 k3s-prod-1 outage: etcd's WAL fdatasync took
3.4-6.7 s (expects 1 s), the node went NotReady while the host stayed up, and
/proc/pressure/io full avg300 read 11.36 there against 0.75 (node 2, NVMe) and
1.29 (node 3).
WHAT IT DOES, on the SATA nodes only:
1. k3s-etcd-io-priority.service (oneshot, Before=k3s.service) switches the
shared device to BFQ and drops nr_requests 64 -> 32.
2. IOWeight=10000 drop-ins on system.slice AND k3s.service, which BFQ turns
into io.bfq.weight=1000 — a 10:1 share against the default-weighted
kubepods.slice where Longhorn's instance-managers live.
3. Reads all of it back out of the kernel and ASSERTS. If the scheduler did
not switch or the weight did not land, the play fails.
THE THREE OBVIOUS ANSWERS ARE INERT ON THESE NODES, ALL CHECKED LIVE, ALL
DELIBERATELY ABSENT:
* IOSchedulingClass=realtime on k3s.service. mq-deadline DOES honour the
priority class, so this looks right — but ioprio is inherited across fork
and kept across exec (block/blk-ioc.c __copy_io copies ioc->ioprio) and
CONTAINERD IS A CHILD OF THE K3S SERVER PROCESS. Verified on k3s-prod-1:
containerd pid 84553 has ppid 84333 = k3s-server, and
ionice -c2 -n0 -- sh -c 'ionice -p $$; sh -c "ionice -p \$\$"'
prints best-effort prio 0 at both levels. Realtime for k3s is realtime for
every container on the node, Longhorn included. No relative gain.
* IOSchedulingPriority (the level inside a class): mq-deadline buckets by
CLASS only, so BE-0 vs BE-4 is a no-op.
* io.latency, which would be the ideal scheduler-independent protection:
CONFIG_BLK_CGROUP_IOLATENCY is not built in Debian trixie's 7.1.8 kernel
(only IOCOST is) and no io.latency file exists under /sys/fs/cgroup.
AND ONE NON-OBVIOUS TRAP: bare IOWeight= is DECORATION here. systemd writes it
to io.weight (blk-iocost — compiled in but disabled, io.cost.qos is empty) and
io.bfq.weight (BFQ — not loaded). Neither reader exists on the stock nodes, so
the scheduler change and the weights have to be one role with one verification.
The value is 10000 rather than a "reasonable" 1000 for the same reason: systemd
rescales 1..10000 onto io.bfq.weight's 1..1000, so 1000 maps to ~101 against
BFQ's unweighted default of 100. The role asserts the result is >= 500.
k3s-prod-2 IS SKIPPED ON PURPOSE. The role discovers the device under
k3s_data_dir per node (verified: prod-1/3 -> /dev/sda5 -> sda, prod-2 ->
/dev/nvme0n1p5 -> nvme0n1) and writes NOTHING on NVMe rather than leaving inert
files behind — BFQ is a poor fit for NVMe and node 2 measured 0.75 IO PSI.
DEV IS UNTOUCHED. Master switch defaults false and is set true only in
group_vars/k3s_prod; dev has a dedicated etcd device (docs/plans/ssd-etcd-
isolation.md) and nothing to arbitrate. Verified with --list-hosts: no extra
vars -> the three dev hosts (where the role is a no-op); -e k3s_hosts=k3s_prod
-> the three prod hosts; --limit k3s-prod-3 -> one.
NOTHING IS RESTARTED. The oneshot writes two sysfs files; the weights go in
through daemon-reload. If daemon-reload turns out not to re-realize a running
k3s.service's cgroup, the verification FAILS and names the rolling restart as
the fix rather than quietly leaving the weight unapplied.
WHAT IT CANNOT DO, stated in the role README and worth repeating: it does not
separate the two tenants. Both still write one ext4 filesystem, and no cgroup
control arbitrates the shared jbd2 journal that etcd's fdatasync waits on. It is
also UNPROVEN on this hardware — it fails loudly if the settings do not land, it
cannot tell you they helped. Apply to k3s-prod-3 first and watch
NodeIOPressureHigh. Replacing the C400 / giving etcd its own device remains the
real fix and the operator's (PROD-PROVISIONING.md §2 exit condition).
Syntax-checked, and the fact-derivation and assert expressions were exercised
offline against all three node shapes plus negative controls (a node still on
mq-deadline fails the gate; IOWeight=1000 -> 101 fails the floor).
Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit de8c1f832724c011b03466842e18296de07a6776)
On 2026-09-07 06:05:21Z k3s-prod-1 went NotReady ("Kubelet stopped posting node
status") with the host up, pingable and SSH-able. The cause was disk I/O
starvation of the embedded etcd, from its own journal:
slow fdatasync took=3.56s / 6.71s / 4.95s / 3.40s (expects 1s)
apply request took too long took=8.30s (expects 100ms)
etcdserver: too many requests
/proc/pressure/io FULL avg300 read 11.36 on k3s-prod-1 against 0.75 on
k3s-prod-2 (NVMe) and 1.29 on k3s-prod-3 — all three nodes put /data/k3s
(etcd's WAL) and /data/longhorn (every replica) on ONE partition, and node 1's
is a 2011 consumer SATA SSD with 61,440 reallocated sectors.
The stack alerted on CPU-PSI and memory and had NO I/O alert at all, so the
first signal the operator got was a node already out of the cluster. Adds:
NodeIOPressureHigh full IO PSI > 5% for 10m warning
NodeIOPressureCritical full IO PSI > 10% for 5m critical
Thresholds are measured, not guessed: rate() is a 0-1 fraction, so the outage's
11.36% is 0.1136 and the healthy steady state on the same node is ~0.01.
Verified live that node-exporter already exports
node_pressure_io_{stalled,waiting}_seconds_total for all three prod nodes, so
these rules fire on arrival — no new exporter, no scrape change. (etcd's own
fsync histograms are NOT available: kube-prometheus-stack's kubeEtcd scrape job
is deliberately disabled for k3s, so PSI is the signal we actually have.)
PSI *full* (every runnable task blocked on disk), not *some*: `some` is noisy on
a storage node and did not separate the failing node from its peers.
Dashboard gains the matching I/O row and is retitled accordingly. Not scoped to
prod — dev shares this file and its k3s VMs are HDD-backed, where the same alert
is if anything more likely to earn its keep.
Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit cb28fc80e7e2085d2539530d55f90fc92e3b1bba)
Non-secret, environment-specific values must not live inside an encrypted
Secret: Flux's ${BASE_DOMAIN} substitution rewrites rendered manifest text and
cannot reach inside ciphertext, so such a value silently freezes at whatever
one environment happened to need. Dev was correct only by coincidence (dev IS
the apex); prod was wrong from its first reconcile, for both Headlamp and
Semaphore. StrictPostBuildSubstitutions cannot catch this — there is no
unresolved ${...} to complain about.
Co-Authored-By: Claude Opus 5 <[email protected]>
Shipped in the previous delivery and failing on every run since:
FATAL: cannot read /etc/keycloak-realm-clients (PermissionError)
BackoffLimitExceeded
The client-secrets volume used defaultMode 0400 — owner-only — but the owner of a
projected Secret volume is root, while the pod runs runAsUser/runAsGroup 65532.
The sibling script volume worked only because it is 0444. So the healer that
exists to correct client-secret drift was itself inert.
It failed CLOSED (no writes attempted, no drift introduced), and the two alerts
shipped alongside it are exactly what surfaces this: KeycloakClientSecretReconciler
Failing fires on the failed Jobs. The observability caught its own subject.
Fixed with fsGroup 65532 + defaultMode 0440 rather than relaxing to world-readable
0444: fsGroup makes the projected volume group-owned by the running user, so the
secret stays unreadable to any other uid in the container.
Co-Authored-By: Claude Opus 5 <[email protected]>
Semaphore's whole SEMAPHORE_OIDC_PROVIDERS JSON blob lived inside the SOPS-encrypted
semaphore-bootstrap Secret (key `oidc-providers-json`) because it CONTAINS the confidential
client secret. That buried two environment-specific, NON-SECRET URLs in ciphertext —
provider_url (the realm issuer) and redirect_url — where Flux's ${BASE_DOMAIN} postBuild
substitution cannot reach. Both were frozen at `home.keithjackson.dev`, so prod's Semaphore
was pointed at dev's Keycloak and dev's own redirect URI: identical root cause to the
Headlamp login timeout, not yet reported only because nobody had tried prod SSO.
FIX: split the blob. The URLs are public and now live as a plaintext literal in
catalog/semaphore.yaml (rendered to the apps/wave-1 golden), ${BASE_DOMAIN}-substituted like
every other hostname. Only the client secret stays encrypted, in its own key
`oidc-client-secret`, and the KUBELET splices it back via dependent-env-var expansion:
$(SEMAPHORE_OIDC_CLIENT_SECRET) resolves against the secretKeyRef env var declared
immediately above it (order is load-bearing — the kubelet resolves env in list order and
leaves a forward reference as literal text).
Verified: the new key holds the same value as keycloak-realm-clients/semaphore-client-secret
(the mirror invariant), Semaphore v2.19.10's util.OidcProvider consumes client_secret as
before, and `flux envsubst --strict` leaves $(...) untouched while resolving ${BASE_DOMAIN} —
dev renders keycloak.home.keithjackson.dev, prod keycloak.prod.home.keithjackson.dev.
apps/wave-1/semaphore.yaml is a render golden; regenerated with the canonical
`catalog/render.py --all catalog -o apps --wave-layout --no-image-automation`.
Co-Authored-By: Claude Opus 5 <[email protected]>
The operator could not log in to Headlamp on prod:
Get "https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration":
dial tcp 192.168.1.53:443: i/o timeout
ROOT CAUSE: OIDC_ISSUER_URL and OIDC_CALLBACK_URL were keys INSIDE the SOPS-encrypted
headlamp-oidc-credentials.sops.yaml. Flux's ${BASE_DOMAIN} postBuild substitution rewrites
the RENDERED manifest text and can never reach inside ciphertext, so both values were frozen
at the apex `home.keithjackson.dev` in BOTH environments. Dev IS the apex, so dev looked
correct; prod (BASE_DOMAIN = prod.home.keithjackson.dev) therefore pointed at dev's Keycloak
at 192.168.1.53 — on the dev LAN behind pfSense, unroutable from prod's 192.168.2.0/24.
FIX: an issuer URL and a callback URL are PUBLIC. Move both out of the Secret into the
HelmRelease's spec.values.env, written with ${BASE_DOMAIN} so substitution resolves them per
environment. The Secret keeps only what is genuinely secret or environment-invariant
(OIDC_CLIENT_SECRET, OIDC_CLIENT_ID, OIDC_SCOPES).
Exactly ONE source per variable: the chart renders `env:` after `envFrom:`, and an explicit
env entry silently WINS over the same name from envFrom — so the two sets must not overlap,
and they no longer do. The five load-bearing names the chart's $(...) args require
(OIDC_CLIENT_ID/CLIENT_SECRET/ISSUER_URL/SCOPES/CALLBACK_URL) are all still delivered;
verified by rendering platform/headlamp per overlay, `flux envsubst --strict`, and
`helm template` of chart 0.41.0 against the substituted values.
Side effect, deliberate: naming OIDC_CALLBACK_URL in `env:` makes the chart finally EMIT
`-oidc-callback-url=$(OIDC_CALLBACK_URL)` — it gates that arg on the value being set via
`env:`/`config.oidc`, which a Secret key alone never tripped, so the Secret's callback URL
had been dead weight. The value is byte-identical to the single redirectUri registered on the
Keycloak `headlamp` client (also ${BASE_DOMAIN}-templated), the only URI Keycloak accepts.
[parity] holds: prod's kube-apiserver oidc-issuer-url is an install-time k3s flag from
k3s_base_domain=prod.home.keithjackson.dev, which is exactly what Headlamp now uses.
Co-Authored-By: Claude Opus 5 <[email protected]>
infra/devcontainer/kubeconfig.sh was never staged into the delivery-A commit, so
the branch built on main carried main's single-cluster tooling and `--context
k3s-prod` failed with 'context does not exist' mid-incident. Not Flux-reconciled,
so it never affected the cluster — but it breaks prod diagnosis from a checkout of
this branch, which is when you need it most.
Carries the concurrency fix: the merged kubeconfig holds both contexts and NO
current-context, with each invocation getting a per-invocation selector file
prepended to KUBECONFIG. Two concurrent dev.sh runs with different CLUSTER values
previously overwrote each other's current-context mid-command — reproduced live,
a CLUSTER=dev reader returned k3s-prod nodes on 3 of 7 iterations.
Co-Authored-By: Claude Opus 5 <[email protected]>
The previous commit created the two halves. This one wires the choice, so dev renders MinIO
credentials and prod renders R2 credentials from the SAME base manifests (R-ENV-02: the
overlay patch is the entire difference between environments; no base manifest is forked).
cluster-vars BACKUP_CREDENTIALS_SUFFIX base "" | dev "" | prod "-prod"
velero HelmRelease credentials.existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}
3x CNPG Cluster s3Credentials.*.name: <db>-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
ONE SUFFIX, NOT FOUR SECRET-NAME KNOBS. Four independent knobs can be updated three-quarters
of the way, leaving one consumer reading the OTHER environment's key with nothing rendering
wrong — and StrictPostBuildSubstitutions cannot catch that: strict mode fails on an UNDEFINED
variable, never on a defined-but-stale one. With one suffix the four move together or not at
all. The cost is that the rendered Secret name is not greppable from the consumer, so both
kustomizations spell the two names out in full beside the resources.
THE BASE DEFAULT IS "" AND THAT IS LOAD-BEARING. It renders the UNSUFFIXED names — objects
that exist in every render, and that dev is running today. The alternative failure mode is
strictly worse than the shared-Secret state this replaces: velero's pod hangs in
ContainerCreating forever when `existingSecret` names a missing Secret (the volume never
mounts), and a CNPG Cluster whose s3Credentials Secret is absent stops archiving WAL — which
under archive_mode=on is a filling pgdata PVC, not a missing backup. So an unresolvable
default breaks the referencing cluster HARDER than a wrong key does. Both overlays state the
value explicitly anyway, so neither inherits it silently.
Prod's overlay change is a cluster-vars op only. The velero Flux Kustomization patch is
untouched (it stays decryption-only, and velero stays suspended on prod).
VERIFIED — `kubectl kustomize --load-restrictor LoadRestrictionsNone` on clusters/dev and
clusters/prod for the cluster-vars values, then on each of the four affected tiers (platform,
platform-config, platform/velero, apps/wave-0) piped through `flux envsubst --strict` with
that environment's variables:
strict envsubst PASSES for all 8 (env x tier) combinations
dev velero existingSecret velero-cloud-credentials
dev keycloak/nextcloud/semaphore-db <db>-backup-credentials (both key refs each)
prod velero existingSecret velero-cloud-credentials-prod
prod keycloak/nextcloud/semaphore-db <db>-backup-credentials-prod (both key refs each)
both all EIGHT Secret objects present in the rendered output (dev and prod alike)
no `${...}` survives in any rendered tier except gitea's three DELIBERATE `$${...}` shell
escapes in apps/wave-0, which are pre-existing and unrelated.
Read alongside the endpoints the same overlays already set, the two environments are now
isolated on credential AND target:
dev velero s3://velero (root) + cnpg s3://cnpg @ http://192.168.1.9:9000 region us-east-1
prod velero k3s-prod-backup/velero + barman prefix @ https://<acct>.r2.cloudflarestorage.com region auto
Co-Authored-By: Claude Opus 5 <[email protected]>
The four backup-credential SOPS Secrets are SHARED BASE manifests: platform/ and apps/ are
rendered by the same Flux tiers on both clusters, and the ONLY per-environment difference in
this repo is cluster-vars substitution (R-ENV-02). So writing the R2 token into all four
(0a6a862) did not "give prod credentials" — it REPLACED DEV'S. Merging that as-is would have
handed dev's Velero and all three dev CNPG Clusters an R2 key id to sign MinIO requests with:
403 on every WAL segment, and archive_mode=on turns that into a disk-full clock on the pgdata
PVC, not a degraded backup. Dev's PROVEN DR restore (R-STATE-03) is the thing that would have
broken.
This commit splits the material. Each credential is now TWO Secret objects, both applied to
BOTH clusters, differing only in name suffix:
dev -> velero-cloud-credentials keycloak/nextcloud/semaphore-db-backup-credentials
prod -> velero-cloud-credentials-prod ...-db-backup-credentials-prod
The consumer picks one by name; that wiring is the NEXT commit, so this one changes no
rendered reference and is a no-op for both live clusters.
WHY DEV KEEPS THE UNSUFFIXED NAME. Dev is live. Renaming its Secret would change
`s3Credentials.accessKeyId.name` on three running CNPG Clusters — which projects into the
instance pods' environment and therefore ROLLS keycloak-db, nextcloud-db and semaphore-db —
and would roll the velero Deployment, all to rename an object whose contents do not change.
This repo already makes exactly this call for dev's live identifiers (EXTERNALDNS_TXT_OWNER_ID
"default", VELERO_BACKUP_PREFIX ""): the live cluster keeps the name it is running, and the
NEW environment takes the new one. The asymmetry is deliberate and documented at each
kustomization.
WHERE THE MATERIAL CAME FROM. Neither side is invented:
- the four unsuffixed files are restored byte-for-byte from `main` (the MinIO material as
it stood before 0a6a862), so dev's Secrets are IDENTICAL to what dev is running;
- the four `-prod` files carry the R2 token from 0a6a862 (already on this branch as
9739e9b), re-encrypted only because metadata.name changed.
RE-ENCRYPTION, AND WHY A COPY+SED IS NOT ENOUGH. `.sops.yaml` sets
`encrypted_regex: ^(data|stringData)$`, so metadata.name stays cleartext — but SOPS MACs the
WHOLE document (there is no `mac_only_encrypted` here), so editing that cleartext line
invalidates the MAC and `sops -d` then refuses the file. The `-prod` files were therefore
produced inside the dev toolbox (infra/devcontainer, which has sops 3.9.4 and the age key
mounted) by decrypt -> rename -> re-encrypt, with the plaintext never leaving a container
tmpfile. Note for the next person: `sops -e --output NEW OLD` matches creation rules against
the INPUT path, so encrypting a tmpfile into a repo path needs `--filename-override NEW` or
it dies with "no matching creation rules found".
Verified: all eight files `sops -d` cleanly; the four dev files hash to dev's material and the
four prod files to R2's, and the two sets differ. `.sops.yaml` needed no new rule — the
existing `(platform|apps|catalog)/.*(secret|sops)\.ya?ml$` rule already matches
`*-prod.sops.yaml`, so the new files encrypt to the same age recipient.
apps/wave-0/kustomization.yaml is GENERATED, so render.py learns the `-prod` sidecar rather
than the golden being hand-edited: `<db>-backup-credentials-prod.sops.yaml` now places into
the DB's wave alongside its unsuffixed twin (both must exist before the Cluster that names
one), and the selftest asserts it.
Co-Authored-By: Claude Opus 5 <[email protected]>
Prod has been archiving WAL to `TBD-PROD-S3-BACKUP-ENDPOINT` since bootstrap.
Verified live 2026-09-07 on all three CNPG Clusters: ContinuousArchiving=False,
"unexpected failure invoking barman-cloud-wal-archive: exit status 4". Because
archive_mode is ON, that is not a degraded backup — it is a DISK-FULL CLOCK.
Postgres will not recycle a segment it has not archived, so keycloak-db's pgdata
PVC is at 579 MiB used / 4360 MiB free and climbing monotonically. When it fills,
the primary PANICs and takes SSO down, and with it kubectl OIDC, Grafana,
Headlamp, Gitea and forward-auth. Wiring the endpoint is what lets Postgres drain
the whole accumulated backlog in one pass.
THE KNOB LAYOUT (three new, one repurposed) and why it is shaped this way.
The operator created exactly ONE R2 bucket, `k3s-prod-backup`, but the two backup
tools want differently-SHAPED values: the Velero chart takes `bucket` and `prefix`
as separate fields, while CNPG's barmanObjectStore.destinationPath is one full
s3:// URL whose first path segment IS the bucket. So:
S3_BACKUP_ENDPOINT dev http://192.168.1.9:9000 prod https://<acct>.r2...
S3_BACKUP_REGION dev us-east-1 prod auto
VELERO_BACKUP_BUCKET dev velero prod k3s-prod-backup
VELERO_BACKUP_PREFIX dev "" prod velero
CNPG_BACKUP_PATH dev s3://cnpg prod s3://k3s-prod-backup/barman
There is deliberately NO separate `S3_BACKUP_BUCKET`: CNPG's bucket already lives
inside CNPG_BACKUP_PATH, and a second knob holding the same bucket name twice is a
rotation waiting to drift. Three knobs, each shaped for its own consumer, and no
two of them can contradict each other.
TENANCY. Inside k3s-prod-backup, velero and barman get disjoint TOP-LEVEL
prefixes — s3://k3s-prod-backup/velero/ and s3://k3s-prod-backup/barman/<cluster>/
— with the bucket ROOT left empty so a misrendered prefix lands somewhere
obviously wrong instead of silently colliding. This is not tidiness: Velero ADOPTS
every backup under its location and TTL-DELETES it on retention, so an overlapping
prefix would let it reap WAL segments; and interleaving two Postgres servers' WAL
into one prefix corrupts the chain outright. Dev↔prod isolation is doubly
independent — different endpoint AND different bucket names — so neither cluster
can reach the other's objects even with a mis-substituted prefix.
DEV IS UNCHANGED, BYTE-FOR-BYTE, and that is asserted rather than assumed. Base
defaults are dev's LIVE values (verified against the running cluster: BSL
bucket=velero with NO prefix, region us-east-1; all three CNPG Clusters
ContinuousArchiving=True at s3://cnpg on the MinIO), and the dev overlay now
restates all five explicitly for the same reason LONGHORN_DATA_PATH is restated —
a silent drift in base would ORPHAN the backups dev's PROVEN restore depends on.
The empty-prefix case was the one real risk, and it is settled empirically: with
prefix "" the chart omits objectStorage.prefix from the rendered
BackupStorageLocation ENTIRELY (helm template, chart 12.1.0), so dev's live BSL is
unchanged, not merely equivalent.
R2's TWO GOTCHAS, both wired up front:
- region must be exactly `auto`. head-bucket answers {"BucketRegion":"WNAM"},
which is a PHYSICAL LOCATION HINT, not a signing region. This was hardcoded
us-east-1 inside the Velero HelmRelease — correct-by-accident for MinIO, wrong
for anything else — so it is a knob now.
- checksumAlgorithm: "". velero-plugin-for-aws v1.14.2 bundles an AWS SDK with
flexible checksums on by default and sends x-amz-sdk-checksum-algorithm /
x-amz-checksum-crc32 on every PUT; R2 rejects those with HTTP 400
InvalidArgument, failing every backup and every kopia blob upload. It must be
PRESENT-but-empty (the plugin branches on the key existing), and the chart does
emit an empty value — verified by rendering it. Harmless against MinIO, so it
is unconditional rather than a fourth knob.
WHAT THIS DOES NOT DO. Velero stays SUSPENDED on prod. `spec.suspend: true` is
owned by field manager `flux` (verified: kustomize-controller's managedFields do
not include f:suspend), so server-side apply will not strip it, and nothing here
declares suspension — which would also trip R-ENV-03, since a prod overlay may
patch only cluster-vars and a Flux Kustomization's `decryption`. Unsuspending is
the operator's, after `standup.sh --s3-endpoint <r2> s3check` proves a round trip.
NO DNS KNOB IS TOUCHED. BASE_DOMAIN stays prod.home.keithjackson.dev and every
EXTERNALDNS_* value is untouched. The apex inversion on work/prod-standup-campaign
was deliberately NOT carried: this branch ports that branch's R2 reasoning knob by
knob, and merges nothing.
VERIFIED: ci/l0.sh green (flux-postbuild-substitution 272 assertions, 0 hazards,
33 knobs defined / 28 referenced, every new token resolved in BOTH overlays);
render-golden, lint, schema, spec-immutability green; touched requirement
R-STATE-03 covered.
Refs R-STATE-03, R-IDP-05, R-ENV-02, R-ENV-03.
Co-Authored-By: Claude Opus 5 <[email protected]>
TRACKED_YAML rglob'd from the repo root excluding only .git and _artifacts, so it
also walked .claude/worktrees/agent-*/ — the isolated worktrees created for agent
runs. The check counted 5 CiliumEgressGatewayPolicy objects where exactly 1 is
authored, failing R-NET-04 for a reason unrelated to egress steering. Now skips
dot-DIRECTORIES (covering .git, .claude, .cache) while still matching dot-FILES
such as .sops.yaml. .claude/ is gitignored, so nothing beneath it is authored
surface.
Co-Authored-By: Claude Opus 5 <[email protected]>
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
Written by platform/backup/set-r2-credentials.sh (operator, 2026-09-07T05:41:34Z)
into all four consumers, in the two shapes they need: velero wants an AWS INI
file, CNPG/barman wants discrete ACCESS_KEY_ID / ACCESS_SECRET_KEY. All four
verified to carry the SAME 32-character key id, so no consumer was missed.
Proven from a pod INSIDE prod against
https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com:
head-bucket on k3s-prod-backup returns BucketRegion WNAM, and PUT and DELETE both
succeed. Byte-integrity of GET is NOT yet proven (the aws-cli image has no cmp);
s3check is what will establish it.
Credentials only — no endpoint or bucket wiring. That is feat/r2-backup-prod.
Co-Authored-By: Claude Opus 5 <[email protected]>
OPERATOR DECISION 2026-09-06: "use the VIP as intended — a virtual address to reach
the cluster and provide fault tolerant HA. Without a change it will not work in HA."
THE DEFECT. Cilium's kube-proxy replacement dialled a SINGLE NODE. Re-confirmed live
before changing anything: all three prod cilium-agent pods carried
KUBERNETES_SERVICE_HOST=192.168.2.10 — k3s-prod-1's own address — while the kube-vip
VIP 192.168.2.9 was Running on all three nodes with a held lease. Lose prod-1 and the
CNI's control-plane path is dead on an "HA" cluster. clusters/prod's own comment said
the value must be the VIP and NOT .10, then called it "Not yet decided" — I treated
that stale comment as an open question when the evidence already settled it.
SPLIT, NOT REPOINT, because k3s_api_host had two consumers with opposite needs:
k3s_api_host unchanged — the FOUNDER's IP: the cold-bootstrap join
target and a tls-san. infra/tofu/outputs.tf GENERATES this
name, so renaming would ripple into tofu and the runbooks.
k3s_apiserver_endpoint NEW = {{ kube_vip_address }} — the HA address; what Cilium
consumes. Derived, not restated, so an environment overrides
ONE line and the VIP, the SAN and the CNI endpoint move
together. Prod needed no new override.
THE SECOND HAZARD, WHICH MY BRIEF DID NOT NAME and the agent found: pointing Cilium at
the VIP breaks a COLD bootstrap for the same reason the join would — site.yml ran
node-baseline -> k3s -> cilium, so Cilium would install against an address nothing
announced yet. That is the 2026-09-01 deadlock shape. Fixed by SEQUENCE, not value:
site.yml now imports kube-vip.yml BETWEEN k3s and cilium. It cannot go earlier
(kube-vip is a k3s auto-deploy manifest, so it needs a cluster) and it works there
because the DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 — needing
neither CNI nor kube-proxy. roles/kube_vip gained a read-only `wait_for` on the VIP so
a cold run stops THERE with a named cause instead of surfacing as unreachable agents.
Also fixed a pre-existing bug that only mattered once this play joined site.yml: the
role's read-only interface probe ran without `check_mode: false`, so under --check it
was skipped and the assert fired with a misleading "set kube_vip_interface explicitly".
VERIFIED BY ME, not just reported:
renders dev K3S_API_HOST=192.168.1.49 prod=192.168.2.9 (each environment's VIP)
cilium template consumes k3s_apiserver_endpoint, not k3s_api_host
cold join still `--server https://{{ k3s_api_host }}:6443` — the founder
order site.yml: node-baseline -> k3s -> kube-vip -> cilium -> verify -> flux
gate full CI gate PASS
Agent-side, additionally: in-memory render of cilium-values.yaml.j2 per host showing
k8sServiceHost=the VIP with the join still on the founder; k3s_tls_sans renders
['.10','.9','127.0.0.1','.11','.12'] so NO cert reissue and NO k3s restart is involved
— .9 already answers under full CA verification; and --check --diff of kube-vip.yml
against prod reported 6 ok / 0 changed, i.e. the role edits change nothing about the
kube-vip prod already runs.
ha-control-plane-as-code 92 -> 112 assertions, turning two escalations into machine
checks: the Cilium template must use the endpoint var; the endpoint must derive from
kube_vip_address and k3s_api_host must not; per environment K3S_API_HOST must equal
that environment's VIP and be none of its node IPs (the [parity] contract, now
machine-checked rather than a comment); site.yml must order k3s -> kube-vip -> cilium.
Every new assertion mutation-tested and bites. One honest negative recorded: reverting
only the BASE default is not caught, because both overlays patch the key — the check
reads effective per-environment values, so the base default is unguarded.
APPLY ORDER, and this is the operator's step — nothing was applied:
PROD FIRST, and it is the low-risk direction (prod is the broken one). Push and let
Flux reconcile platform/cilium: the value change rolls the cilium-agent DaemonSet
ONE NODE AT A TIME (maxUnavailable: 1 — the knob that bounded the 2026-09-01
incident), it will not proceed past an unready pod, existing datapath survives an
agent restart, and .9 is already live before the change lands.
DEV SEPARATELY, operator present. Same mechanics but live dev agents dial
192.168.1.50, and dev is mid-DNS-migration on dev-live. Do not fold it into that.
ANSIBLE: nothing to run for either live cluster; the Ansible half matters only for a
cold build. Remove the now-redundant standalone kube-vip.yml step from any runbook.
NEW NAMED GAP, unwritten: the L2 test cannot assert the running agents' live
KUBERNETES_SERVICE_HOST, which is the drift the L0 check cannot see. Recorded in that
test's gap list.
Refs R-PLAT-01, R-NET-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
Prod's UniFi VM has no disk. `DataVolume unifi/unifi-rootdisk` stuck in
ImportScheduled for hours, because Kyverno denied CDI's own generated pod:
validate.kyverno.svc-fail denied Pod/unifi/importer-prime-<uid>
require-auth-mode-label (R-IDP-02)
require-egress-class-label (R-NET-05)
Both policies match any Pod in an `app-namespace: "true"` namespace with no
exclusion, and CDI generates `importer-*` itself — it cannot carry repo labels.
A COLD-START defect, not a prod quirk: dev's identical import Succeeded 9 days
ago, before the policies enforced, so dev has never exercised it and would hit it
on any rebuild. Fourth bug of this shape in two days.
MECHANISM: a MUTATE ClusterPolicy that ADDS
`egress-class: internet` + `auth-mode: none` (add-if-absent anchor) to Pods on
CREATE in app namespaces WHEN the requesting subject is ServiceAccount
`cdi/cdi-sa`. Both validate policies are untouched — no exclude, no exception,
`background: true` preserved on them.
Why mutate beats exempting: the requirement stays literally true ("every pod in an
app namespace declares both labels") and the values are HONEST — the importer does
pull over the default WAN path, and `internet` is the least-reach class (it does
not grant the VPN path); a transient worker pod with no UI genuinely is
`auth-mode: none`.
THE MATCH IS IDENTITY, NOT A NAME. `subjects: ServiceAccount cdi/cdi-sa` is the
AdmissionReview userInfo, taken from the real prod denial
(`system:serviceaccount:cdi:cdi-sa`, name importer-prime-…, op CREATE). A name
prefix is attacker-choosable; a requesting ServiceAccount is not — and the test
proves it, with a fixture `importer-prime-forged-by-an-app` that copies both the
name prefix AND CDI's labels and is still DENIED. Covers every CDI worker kind
(importer/uploader/cloner) without enumerating names that change per release.
Confirmed on prod that cdi-sa is the only cdi-namespace SA that can create pods.
REJECTED, and the first one is a trap worth recording:
* PolicyException — the CRD EXISTS (v2/v2beta1, from the pre-applied bundle) but
all controllers run `--enablePolicyException=false` (verified live on prod:
admission, background and reports controllers all carry the flag). An exception
would apply cleanly through Flux and be SILENTLY IGNORED. Enabling the gate is a
cluster-wide capability change for no gain over the mutate.
* `exclude` scoped to cdi-sa — sound identity, worse consequences: any userInfo
match forces `background: false`, which would drop background scanning and
policy-report coverage of R-NET-05/R-IDP-02 for ALL EXISTING PODS to fix one pod
class. It also leaves the importer with no labels at all — the unlabelled
fall-through that require-egress-class-label's own header calls a fail-open leak.
* name-prefix / CDI-label matching — attacker-choosable, and proven walkable.
TWO-WAY PROOF, non-vacuous by construction — same resource, different requester,
opposite outcomes (verified by me, not just reported):
userinfo cdi-sa -> mutate pass, require-egress-class pass, require-auth-mode pass
userinfo ordinary -> mutate SKIP, require-egress-class FAIL, require-auth-mode FAIL
Kyverno L0 went 17 -> 29 tests passed, 0 failed. The mutate policy is deliberately
kept OUT of the existing tests/kyverno-test.yaml: that suite declares no userinfo,
and without userinfo the offline engine does not discriminate on subjects — it
mutates everything, which would have silently turned its negative rows into passes.
platform-config/kustomization.yaml is the live apply path for these ClusterPolicies;
without it the policy never reaches a cluster.
RESIDUAL TRADE-OFF, reversible by deleting one file: anything created by cdi-sa in
an app namespace now has these labels supplied rather than being rejected for not
declaring them — including a pod that is not a CDI worker. That identity already
holds `pods create/delete` cluster-wide via ClusterRole `cdi`, so it is not a new
capability; what changes is that for that one identity the declaration is assumed
rather than demanded. restrict-host-namespaces, the namespace default-deny baseline
and the CiliumNetworkPolicies all still apply.
FIRST resource-mutating policy on this cluster: kyverno-resource-mutating-webhook-cfg
holds 0 webhooks today; with autoUpdateWebhooks=true Kyverno registers
mutate.kyverno.svc-fail for Pods within seconds of this applying. Prod's
forceFailurePolicyIgnore=false makes it failurePolicy: Fail — same posture as the
validating webhook already in that path.
CDI retries importer creation on a loop, so the stuck import should proceed once
this lands. The detached prime-be3f9d1c-… 40 GiB scratch PVC is separate operator
cleanup; untouched, as is the DataVolume.
VERIFIED: full CI gate PASS. Nothing applied to any cluster.
Refs R-IDP-02, R-NET-05, R-PLAT-01.
Co-Authored-By: Claude Opus 5 <[email protected]>
`platform-config` applied the CDI CR *and* cdi-storageprofiles.yaml in ONE pass.
StorageProfile's CRD does not exist until cdi-operator reconciles that CR, and
Flux server-side dry-runs every object before applying any — so the unknown kind
aborted the tier, the CR was never applied, the CRD was never created, and the
tier could NEVER converge. Observed on prod: cdi-operator Available 12h, no CDI
CR, no storageprofiles CRD, every downstream tier blocked.
The profiles now live in their own tier ./platform-config-cdi with
dependsOn: platform-config and wait: true. Because dependsOn is REVISION-gated
and platform-config runs wait: true, the CDI CR must be applied AND healthy —
exactly the event that creates the CRD — before the profiles are applied. Cold
clusters order themselves. `standup.sh crds` is retained as a fallback for
clusters already stuck, and because it is idempotent.
THE GENERALISABLE RULE (recorded in clusters/README.md): a CR whose CRD is
created by a CONTROLLER AT RUNTIME is second-order and needs its own pass.
cdi-operator.yaml registers exactly ONE CRD (cdis.cdi.kubevirt.io); storageprofiles,
datavolumes, datasources and dataimportcrons appear only after it reconciles the
CDI CR. An audit found StorageProfile is the only second-order kind in
platform-config — KubeVirt, CDI, KeycloakRealmImport, CNPG Cluster,
VolumeSnapshotClass, ClusterIssuer, PrometheusRule, ServiceMonitor and the Cilium
L2 CRs all have their CRD in git or in a chart.
Moving the CDI CR up into `platform` was rejected: that would put CDI's health on
the tier the CNI gates and everything waits on — the mistake the oauth2-proxy and
headlamp splits exist to undo.
apps-vms gains dependsOn: platform-config-cdi. LOAD-BEARING, not cosmetic:
apps/vms/unifi/20-datavolume.yaml provisions on storageClassName: longhorn-vm and
its comment claimed platform-config guaranteed the profiles — which this move
would have silently falsified.
THE PRUNE TRANSIENT, MEASURED RATHER THAN ASSUMED. platform-config has
prune: true, so it deletes both profiles once, CDI recreates them with empty spec,
and the new tier re-applies the overrides ~30-60s later. Compared our override
against CDI's unaided inference for the same Longhorn provisioner on dev:
longhorn-vm (we override) spec RWX/Block -> status ReadWriteMany/Block
longhorn-static (no override) spec unset -> status ReadWriteOnce/Block
Both infer cloneStrategy: snapshot. So cloneStrategy is NOT at risk; accessModes
IS — unaided CDI gives RWO. A DataVolume provisioned on longhorn-vm inside the
window would be stamped RWO/Block and be PERMANENTLY non-live-migratable. Flux can
no longer cause it (apps-vms gates on the new tier); a human must not do it by hand.
Corroborating evidence that the recreate happens at all: dev has 5 StorageClasses
and 5 StorageProfiles, three with fluxOwner=NONE, spec unset and status populated
— CDI creating and inferring them with no Flux involvement.
MERGE PRECONDITION: `flux get kustomization platform-config` must be Ready on the
target first. If it is not, the profiles are pruned and sit at CDI defaults until
it recovers — and prod was in that state hours earlier. Fallback is the two-step
the Cilium move used (merge with prune: false, verify the ownership label moved,
restore prune: true).
substituteFrom: cluster-vars is declared even though the manifest has no ${TOKEN},
deliberately: conformance/tests/l0-static/flux-postbuild-substitution only audits
paths whose Kustomization declares it, so omitting it would move the file OUT of
the audited set — a blind spot for a future literal-token ship. Gate confirms it
stayed in (path=./platform-config-cdi files=2 defined=0 hazards=0). No SOPS
decryption stanza: the moved file is two plain CRs, so no overlay change was needed.
VERIFIED INDEPENDENTLY OF THE AGENT: platform-config renders 0 StorageProfile
while keeping the CDI and KubeVirt CRs; platform-config-cdi renders exactly the
two profiles; the graph is byte-identical across dev and prod with apps-vms
carrying both edges and no cycle. Full CI gate PASS.
KNOWN FOLLOW-UP: bootstrap/standup.sh TIER_ORDER does not yet include
platform-config-cdi, so `resume` will not wait on it. Another agent holds that
file; the PM will add it.
Refs R-GITOPS-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
Prod reconciled from http://192.168.2.132:3000 — a podman container on the
operator's WORKSTATION, holding a DHCP lease on WIFI. Production's source of truth
was a laptop: a lease change or a closed lid stalls all reconciliation. Dev was
pointed there too (deliberately, to break a strand risk during its DNS migration),
so BOTH clusters depended on it simultaneously.
Prod now reconciles from https://gitea.prod.home.keithjackson.dev/... — the Gitea
prod itself runs, which apps-wave-0 has been serving for 28h.
THE SELF-REFERENCE IS THE WHOLE DIFFICULTY, and it is why this is a commit rather
than a kubectl patch. The root Kustomization renders ./clusters/prod, which CONTAINS
the patch that sets GitRepository.spec.url — so a live patch is reverted on the next
reconcile unless the AUTHORED value moves with it. Dev hit the identical trap on
2026-09-05. The ordered cutover:
1. push the repo to prod's Gitea (18 branches, refs verified identical, 499 commits
of history — a real move, not a shallow copy);
2. fast-forward prod Gitea's `main` to c8c1d29, the EXACT commit prod was already
running, so swapping the source changed nothing else. Its main had been at
4809199, and repointing to that would have ROLLED PROD BACKWARDS, undoing the
per-environment BASE_DOMAIN work and sending prod at the apex — the collision
with live dev that two days of work exists to prevent;
3. live-patch spec.url to open a window (source changed, content identical:
Ready=True at the same revision on the first poll);
4. land THIS commit through that window so the authored value agrees.
VERIFIED FROM INSIDE PROD, not from the workstation: a throwaway pod resolved
gitea.prod.home.keithjackson.dev through in-cluster CoreDNS to the Gateway VIP
(192.168.2.20) and cloned main over the public Let's Encrypt cert with NO CA
supplied. That exercises split-horizon DNS, the Gateway, Gitea and TLS on the pod
network — the path Flux actually uses. A workstation clone proves none of it.
The bootstrap Gitea is retained as a recovery path, not deleted
(bootstrap/gitea/bootstrap-gitea.sh). Dev still reconciles from it on branch
`dev-live` and returns to its own Gitea at step D4 of the §10a migration runbook.
Refs R-GITOPS-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).
THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.
Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.
PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.
BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.
TWO THAT CANNOT BE TOKENIZED, flagged not forced:
- headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
anyway — client secrets are realm-scoped.
- kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
prod.home.
EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.
I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.
DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.
L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.
PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.
Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Autonomous prod bring-up, operator away. Three changes that together clear the last
blockers before `flux bootstrap`.
=== 1. TEMPORARY BOOTSTRAP GITEA — the last hard blocker, cleared ===
clusters/prod's source URL could not work: Gitea runs ON the dev cluster (the one slated for
decommission) and split-horizon DNS resolves gitea.home.keithjackson.dev to PROD's OWN
gateway. Stood up the existing bootstrap/gitea/bootstrap-gitea.sh on the operator's
workstation with HOST_NAT_IP=192.168.2.132; all three prod nodes verified reachable
(Gitea 1.22.6 via the API), no firewall change needed — prod is on the same /24, no NAT.
clusters/prod now points at http://192.168.2.132:3000/... under a boxed TEMPORARY banner
with a 5-step cutover checklist ending 'delete this whole block — dev's stanza is what done
looks like'. Repo verified PUBLIC with anonymous read two independent ways (API
"private":false; git ls-remote unauthenticated with GIT_TERMINAL_PROMPT=0), so NO secretRef
is needed — documented so a future auth failure is not misread as a network fault.
bootstrap-gitea.sh's header corrected: it documented HOST_NAT_IP=10.73.0.1, the RETIRED
libvirt NAT, and forbade agent use; the operator has now directed this for prod.
=== 2. WIRELESS DISABLED ON PROD (operator-reported) ===
k3s-prod-3 was on ethernet AND wireless, and the wifi DHCP client was overwriting
/etc/resolv.conf. Root cause of the silence: node_baseline's DNS `contended` detection
checked resolvconf and NetworkManager but NOT dhcpcd, so the role wrote resolvers, reported
success, and dhcpcd reverted them on lease renewal — the exact 'green run that fixed
nothing' the contended branch exists to prevent.
Now IaC, not a one-off: survey wireless hardware, **REFUSE to disable it if the path to the
node runs over the wireless interface** (the guard that meant an agent dying mid-apply cost
nothing), blacklist the driver so it never reloads, stop dhcpcd only where it holds a
wireless interface and nothing else, take links down and unload modules with NO reboot, then
report the resulting state. dhcpcd is now in the contended set without weakening fail-closed.
Applied and verified: node 3's wlp1s0 is GONE from /sys/class/net, dhcpcd inactive and its
unit not-found, ONE default route per node, all three on 8.8.8.8/1.1.1.1 resolving public
names. Safe and idempotent on nodes 1 and 2, whose wifi hardware was already down.
=== 3. flux_bootstrap PATH PARAMETERISED ===
bootstrap/flux/bootstrap.sh already read FLUX_ENV_PATH but the role never exported it, so it
ALWAYS bootstrapped clusters/dev — prod was impossible. Two behaviour-bearing lines: role
default clusters/dev + `environment: FLUX_ENV_PATH`, overridden to clusters/prod in
group_vars/k3s_prod. Proven per host with a read-only Ansible probe: prod hosts resolve
clusters/prod, localhost/hypervisor resolves clusters/dev. Only two definitions repo-wide.
DEV PROVEN BYTE-IDENTICAL for the source change: all 12 substituting Flux paths rendered for
both trees, 27/27 SHA-256 artefacts identical, with TWO POSITIVE CONTROLS (an overlay-level
canary and a deep-path content canary) confirming the harness detects change — and both
canaries proven restored. Honest scope note recorded: kustomize strips comments, so the
harness proves the built stream, not source comments.
L0 substitution check: 106 assertions, 0 hazards. ci/l0.sh and ci/lint.sh exit 0.
=== STILL BLOCKING A FULL PROD RECONCILE (operator decisions) ===
- S3_BACKUP_ENDPOINT remains TBD-PROD-*: prod CANNOT reach dev's versitygw at
192.168.1.9:9000 (verified UNREACHABLE from all three — the pfSense boundary), and a prod
backup target is a deferred operator decision. Deliberately NOT invented: a DR target on a
workstation container would create false confidence. Velero/CNPG backups will fail loudly.
- Prod's public resolvers mean the API server's oidc-issuer-url resolves PUBLICLY, so kubectl
SSO will not work on prod until split-horizon is solved for it (§4).
- The temp Gitea is a single container on the operator's workstation — a single point of
failure for prod reconciliation, acceptable for bring-up only.
Serves: R-GITOPS-01, R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Prod is being stood up now, without waiting for node 2's RAM upgrade. Three coupled
changes, all applied and verified on all three nodes.
=== 1. KERNEL >= 6.13 — now a PREREQUISITE, not a nice-to-have ===
A k3s SERVER that intermittently boots with 2.6 GiB is not a degraded node; it fails under
etcd plus its workload share, unpredictably, on any reboot. Fixed before k3s goes on.
roles/node_baseline gains a `kernel`-tagged block: asserts the suite matches the node's
Debian release, writes a deb822 trixie-backports source, installs, reads back the exact
version, writes a provenance/rollback note to the node. Scoped by
node_baseline_backports_kernel (default FALSE, true only in group_vars/k3s_prod) — chosen
over a hostname test so a fourth prod node inherits it from group membership. Verified:
--check against dev skips all 8 tasks on all 3 hosts. Two deliberate calls: `state: latest`
not `present` (linux-image-amd64 was already installed from main, so `present` would have
silently fixed nothing), and NO preferences.d pin (backports' NotAutomatic +
ButAutomaticUpgrades already gives 100/500; an explicit Pin-Priority: 100 would switch OFF
automatic security upgrades for the very kernel being installed).
Installed 7.1.8+deb13-amd64 on all three; 6.12.107 kept bootable.
THE DECISIVE PROOF: node 2 booted at MEMATTR=0xbeeb0018 — the EXACT address that gave it
2.6 GiB twice — and now reserves 634,904K instead of 13,958,980K. MemTotal 15.4 GiB. Same
address, handled correctly; not a lucky draw.
§0b's claim that a fixed kernel prints 'FW_BUG Corrupted...' is CONDITIONAL and was
corrected: >=6.13 has three rejection gates and only the third prints that line; all three
nodes hit gate 1 (version > 2). Proof instead is boot-independent — decompressing
/boot/vmlinuz on all three yields the FW_BUG and descriptor-size strings verbatim; neither
exists in 6.12.
TWO MORE BUGS IN MY OWN HEALTH CHECK, found because the fix couldn't otherwise be shown:
- MEMATTR_VERSION_RE matched (\d+) but the kernel prints that field with %d — SIGNED. Node
2's real value was -1569980416, which did not match, so state fell through to `clean` and
the node reported PASS "the kernel accepted it without complaint". A FALSE PASS on the
exact bug the check exists to find. Now (-?\d+).
- bad_version was WARN unconditionally, so a FIXED node would have stayed WARN forever.
Now keys on memattr_kernel_fixed: same line, WARN below 6.13, PASS at/above.
=== 2. DISK LAYOUT — Longhorn claimed, per-node paths DISCOVERED not hardcoded ===
Operator decision: reuse the ~424 GB partition; accept the shared-device I/O risk (dev's
etcd/Longhorn contention was on a spinning HDD, prod is SSD/NVMe).
Device paths differ per node (sda5 / nvme0n1p5 / sda5), so the role DISCOVERS the partition
via `findmnt --mountpoint` on a declared claim mountpoint rather than trusting inventory.
Per-host host_vars would have frozen today's hardware into inventory, where it goes stale on
the first re-image and the stale value's failure mode is mkfs on the WRONG partition.
Eleven fail-closed asserts before any write: real block device, not backing / /boot /usr
/var /home /tmp or active swap (built from the node's OWN gathered facts, never a hardcoded
list), already ext4, no foreign label, >= 100 GiB, and empty. Relabel, never reformat.
/srv verified empty first on every node (ls -A -> lost+found only; du 20K; the "5.1% used"
was ext4's default 5% root reserve).
=== 3. k3s DATA-DIR MOVED — /var was 11.8 GiB and would have killed etcd ===
Measured on the fully-deployed dev cluster: containerd 21-25 GB/node, /var total 58-70 GB.
Prod's /var is 11.8 GiB and held BOTH etcd and the image store, so prod would have filled
/var during its first image pulls — and a full /var means etcd cannot write its WAL. Cluster
death, not degradation, arriving during bring-up. Invisible when the disk decision was made,
because "etcd on root" sounded like the 30 GiB / rather than a separate 12 GiB /var.
Now: p5 -> /data (LABEL=k3s-data), with /data/longhorn and /data/k3s. Neutral names chosen
deliberately — the partition now has two tenants, and reusing /srv would make "claimed" and
"unclaimed" indistinguishable to the idempotency pivot, while a bare `data` label would make
a stranger's `data` volume look already-claimed and weaken an assert.
The transition off the existing LABEL=longhorn mount is DECLARED, not inferred
(node_baseline_data_retired_fs_label/_retired_mount, defaults ""), because changing the label
makes a claimed node answer "no" to the pivot and walk into the greenfield path. Self-retiring
once blkid -L k3s-data succeeds. THREE independent gates keep live dev out: retired vars
default ""; gated on longhorn_source == claim-mountpoint (dev is `device`); and the emptiness
assert fails closed on dev's 18 volumes even if both were defeated.
k3s_data_dir is one variable read by two roles (k3s_server --data-dir; node_baseline derives
the etcd mount). Role default is k3s's own default, so dev is unchanged and keeps its
dedicated etcd device. node_baseline_etcd_source becomes `shared-data-partition` and is
LOAD-BEARING: the role asserts per run that the filesystem backing the etcd dir really IS the
data partition — had the mount silently failed, etcd plus 21-25 GB of images would land back
on the OS filesystem invisibly until the disk filled.
LONGHORN_DATA_PATH is a new cluster-vars knob (R-ENV-02): base and DEV /var/lib/longhorn,
prod /data/longhorn.
DEV PROVEN UNCHANGED, which is the property that mattered — the live cluster has 18 Longhorn
volumes on that path. All 12 substituting Flux paths rendered through
`kubectl kustomize | flux envsubst --strict` for both trees: 12/12 per-file SHA-256 identical
and an identical aggregate SHA over every rendered dev byte. With a POSITIVE CONTROL showing
the same harness detects prod changing on exactly one line — an all-identical result is
worthless if the harness cannot see change. Live check after: dev's default-data-path setting
still /var/lib/longhorn, 18 volumes attached, 0 Kustomizations unready.
Idempotent: run 2 is changed=0 on all three nodes. Also fixed a real tag bug — include_tasks
does NOT propagate its own tags to included tasks (unlike import_tasks), so `--tags disks`
selected the include and ran nothing inside it while reporting success.
OPEN DEFECTS, tracked not forgotten: node 2's warm-reboot hang (one clean reboot since, NOT a
fix); and node 3 runs dhcpcd for its wifi interface, which rewrites /etc/resolv.conf on lease
renewal — the role's `contended` detection checks resolvconf and NetworkManager only, so node
3's decided resolvers will silently revert.
Gate: MERGE GATE: PASS. L0 substitution check 106 assertions.
Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-STATE-01, R-STATE-02, R-PLAT-01. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
THE MODEL WAS WRONG AND MY CHECK WITH IT. A live counter-example: k3s-prod-2 after its power
cycle shows MEMATTR=0xbeeb0018 and 'Failed to map ... @ 0xbeeb0018' — the SAME address and SAME
line as both of its 2.6 GiB boots — with MemTotal 15857732 kB, i.e. 15.1 GiB of 16 GB. FULL
MEMORY. The check reported FAIL 'THIS NODE HAS LOST MEMORY TO THE BUG' on a healthy node while
memory.physical_total correctly PASSed at 4.2%. The two checks contradicted each other.
REFINED MECHANISM — two kernel functions were conflated:
efi_memattr_init() does the memblock_reserve(). Catastrophic loss needs tbl_size NEGATIVE
(signed int -> sign-extended -> clamped to PHYS_ADDR_MAX). A garbage-but-POSITIVE tbl_size
reserves a modest wrong amount and costs nothing noticeable.
efi_memattr_apply_permissions() does the later memremap(), and THAT is what prints
'Failed to map'. It fails for ANY sufficiently wrong size, negative or merely large-positive.
So the line proves the table was corrupt AND TRUSTED, not that memory was lost. The cost is set
by the garbage VALUE, which is why one address gave 2.6 GiB twice and 15.1 GiB once.
THIS ALSO REFUTES THE ADDRESS->OUTCOME HYPOTHESIS OUTRIGHT — not untested, disproven.
STRUCTURAL FIX so the contradiction cannot recur: the shortfall is computed ONCE into a shared
fact (node_health_mem_shortfall) with worst-wins sub-verdicts; memory.physical_total IS that
verdict and firmware.efi_memattr READS it, measuring nothing itself. FAIL is gated on the shared
verdict. 'Failed to map' + normal memory is now WARN ('the bug fired and this boot survived it').
No honesty guard was weakened; one was strengthened (the measurement clause falls back to the DIMM
shortfall when zoneinfo is unreadable, and only claims 'could NOT be measured' when neither was
read). All five paths incl. the two unproducible-live SKIPs exercised against synthetic facts.
Verdicts: prod-1 WARN, prod-2 WARN (was FAIL), prod-3 PASS, dev unaffected.
SECOND FAULT, arguably worse for prod: NODE 2 HANGS ON A WARM REBOOT, STAYING POWERED. Its
journal ends cleanly mid-shutdown ('eno1: NIC Link is Down') then stops; the box sat powered but
off-LAN for ~38 min until the operator held the power button. A node that cannot survive an
unattended warm reboot cannot be rolled in an HA cluster, and node_health cannot see a machine
that is off the network. Leading candidate: node 2 is the ONLY node with 'Recover OS after Boot
Failure = Enable' — promoted from 'coincidence' in the BIOS diff to a live availability defect.
The reboot survey is NOT viable unattended on node 2 until this is understood.
SURVEY RESULTS (40 boots): prod-1 0/20, prod-3 0/20, prod-2 0 ROUNDS (hung on its first reboot),
pooled 0/40 -> p(bad) 95% CI 0.000-0.088. NOT elimination: 0/20 still permits 17% per node, and
node 2's prior 2-of-3 is unchanged. What it did establish: addresses are node-specific and highly
concentrated (node 1 used 3 distinct addresses in 20 boots, 15 at one; no address ever shared
between nodes); MEMATTR -> parse was a PERFECT function across all 7 repeated addresses over 37
boots; and the corrupting bytes are stable PER MACHINE (node 1 always 1030, node 3 always
1463964749), so the 'shared OS boot chain' inference needs refining.
Adds playbooks/memattr-survey.yml + roles/memattr_survey — reboots opt-in, resumable, artifacts
gitignored. Documents an Ansible defect worth knowing repo-wide: a looped include_tasks under the
default 'linear' strategy is flattened per host, so hosts run SERIALLY; the first attempt rebooted
node 1 twenty times while the others idled. Fixed with strategy: free + a forks preflight assert.
Serves: R-ENV-01, R-ENV-03, R-OBS-04, R-TEST-02. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
k3s-prod-3 online at the planned 192.168.2.12 (serial 8CC015307N, same chassis/BIOS/CPU),
enabled in inventory, vaulted become confirmed, package floor installed. Health: 0 FAIL /
4 WARN / 14 PASS / 1 SKIP — the only node with zero FAILs. WD Blue WDS500G2B0A SATA SSD at
100% life, 3213 h, 0 grown bad blocks, 0 erase fails. Matched Crucial 2x16 GB pair — the only
node with symmetric memory.
*** NODE 3 IS NOT IMMUNE TO THE §0b BUG. *** Its firmware DOES publish a MAT
(MEMATTR=0xbeece018); the ABSENCE of a memattr diagnostic line means the kernel read a VALID
table, which is the opposite of immunity. There are THREE outcomes, not two: bail-out on a
garbage version >2, a valid parse with a sane size, and the catastrophic path. Seven boots across
three machines show the MAT address MOVES BOOT-TO-BOOT on every node with settings constant, and
each node has been seen on more than one path. Blast radius widens from two nodes to ALL THREE;
all three need the >= 6.13 kernel.
THE CORRUPTING BYTES ARE THE OS BOOT CHAIN, not firmware and not DRAM: garbage version
1463964749 = 0x5742544D appeared on node 2 AND node 3 — two different machines, identical bytes.
The overwriter is the shared shim/GRUB/EFI-stub/initrd placement writing into the page the
firmware pointed at. So initrd size, module set and boot device are part of the causal chain.
BIOS-SETTING HYPOTHESIS: DISPROVEN, and the three-way diff cannot support it even in principle.
All 240 attributes read on all three; firmware identical fleet-wide. The discriminating set (both
working nodes agree, broken differs) is seven attributes, of which:
- Fast Boot is the ONLY live candidate, and only as a probability-shifter — HP documents it
'skips some preboot initialization steps', which changes the heap top and hence where the MAT
lands. Different mechanism from the previously-killed MRC-caching theory; that stays dead.
- Legacy Boot Options is AFFIRMATIVELY INERT: 'Configure Legacy Support and Secure Boot' =
'Legacy Support Disable and Secure Boot Enable' on ALL THREE, Option ROM Launch Policy =
'All UEFI' on all three. No CSM is built on node 2 either. A naive diff gets this wrong.
- the rest are coincidence. After Power Loss is ELIMINATED by node 3, which works and shares
node 2's Power Off — though it is now a §1 checklist violation on TWO nodes.
FATAL CONFOUND: the 1=3!=2 partition is perfectly collinear with node 2 being the only
single-DIMM AND the only NVMe-booting node. Adding node 3 did not break the confound, it DOUBLED
it. No setting can be the cause anyway: node 2 booted good once and bad twice with settings held
constant.
EXPERIMENT DESIGN RECORDED because the obvious approach is statistically worthless: p(bad) is
known only to n=3, 95% CI ~0.09-0.99, so even 8 clean boots prove nothing. Elimination needs ~20
boots/arm; an odds shift needs ~50. Zero-touch first step: 20 reboots of node 2 unchanged, logging
MEMATTR address vs outcome — both failures were at exactly 0xbeeb0018 and all five successes
elsewhere, so if outcome maps onto a narrow address range the mechanism is settled without
touching anything.
ROLE BUG FIXED: node 3's WD drive reports a SMART threshold for only 1 of 25 attributes, and
smartctl OMITS the key rather than emitting 0 — summary.txt.j2 dereferenced a.thresh bare in five
places and the report task died, so no artifact was written. Added a thr() macro rendering 'n/a'
(deliberately not 0, which would read as a real ATA cannot-trip threshold). Would have hit any
drive of this class.
Also: three nodes, three different disk models, only one NVMe, and every node has a single disk
with no unpartitioned spare — so §2's identical-layout requirement is unmet on all three.
Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Operator asked whether waiting is viable or whether a backports kernel is permanent. Answer: it
will NOT arrive on its own, but it CAN be made to, and the ask is unusually strong.
VERIFIED:
- NOT in linux-6.12.y. Newest commit touching memattr.c on that branch is 2024-07-12. In mainline
since v6.13 (2025-01-19) — ~19 months, ~97 point releases.
- NO stable tag and NO Fixes: tag on either commit. Structural, not an oversight: the unvalidated
arithmetic dates to the file's creation in 2016, so it is not a regression and there is no bad
commit to point Fixes: at. BOTH automatic feeds — Fixes:-scanning and AUTOSEL — had nothing to
grab. AUTOSEL never surfaced it for any branch.
- NOBODY has requested it: nothing on [email protected], nothing in all 8,027 open+archived
Debian bugs against src:linux.
- 6.12 LTS EOL is December 2028 — outlives Debian 13's support window, so not time-barred.
- Debian REBASES: trixie carries 6.12.94-1 in trixie-security ('New upstream stable update'), so an
upstream pickup would reach us via a normal apt upgrade.
THE DECISIVE CARD: the cherry-pick is BYTE-CLEAN. Debian trixie 6.12.94-1's memattr.c hashes to
164203429fa7a..., identical to the upstream patch's pre-image blob — the file has not diverged
between v6.12 and the commit's parent. cherry-pick applies with zero conflicts and zero adjustment,
which removes the usual reason such requests get declined. The commit also meets every
stable-kernel-rules criterion: mainline, 17 insertions, 'hardware quirk', 'a real bug that bothers
people'.
PLAN: backports now as explicitly provisional; email [email protected] (rules Option 2) CC
linux-efi and Ard Biesheuvel who authored AND committed it as EFI maintainer, leading with the
clean-apply proof and the three-boot reproduction with byte-identical e820; Debian BTS as secondary
under handbook §9.1.7; make the deviation permanent only if there is no movement in 4-6 weeks.
Only 8fbe4c49c0cc is needed by hand — 5d0faa8e8369 loosens a heuristic that was rejecting
LEGITIMATE tables and is not required for this symptom.
Serves: R-ENV-01, R-ENV-03. Task: T-7.1.
Surfaced by the new node_health DIMM reporting, unrelated to the memattr bug and NOT a fault:
DIMM1 is a Micron 8ATF2G64HZ-3G2E1 (DDR4-3200), DIMM3 a Micron 16ATF2G64HZ-2G6E1 (DDR4-2667).
Both 16 GB so capacity is right, but different rank organisation and rated speed mean the pair
runs at the slower 2667.
Recorded in §0 where the symmetry decision gets made, because §2 requires the three nodes to end
up with identical layouts and the previous '2x16 GB Micron' note hid the mismatch.
Serves: R-ENV-03. Task: T-7.1.
The fault is BOOT-TO-BOOT NON-DETERMINISTIC — the same machine booted correctly two boots
earlier — so it has to be detected on every health run, not diagnosed once. And node 1 is
exposed to the identical lottery on every reboot while currently looking healthy.
firmware.efi_memattr — reads the kernel ring buffer and distinguishes six states:
Failed to map EFI Memory Attributes table @ 0x.. -> FAIL (memory lost NOW)
Unexpected EFI Memory Attributes table version <N> -> WARN (corrupt table still published;
intact only by luck)
[Firmware Bug]: Corrupted EFI ... detected! -> PASS (the >=6.13 fix working)
no memattr line, log reaches early boot -> PASS (no table / accepted / not UEFI)
no memattr line, log does NOT reach early boot -> SKIP
dmesg unreadable -> SKIP
memory.physical_total — deliberately a SEPARATE check, not folded in: it measures the SYMPTOM
with no kernel log at all, so an unreadable-dmesg node is still not unmeasured. Two sub-verdicts,
worst wins: /proc/zoneinfo present-vs-managed (world-readable, works on metal and VMs — sum of
managed IS MemTotal) and MemTotal vs the DIMM total from dmidecode. Thresholds 10% WARN / 25%
FAIL, set from measured baselines: 2.0% prod-1, 2.1% workstation, 2.2% dev VM, vs 83.4% prod-2.
THREE GUARDS AGAINST A FALSE PASS, because "no memattr line" must never silently mean "fine":
1. dmesg failure yields available:false with the exact reason (rc, uid, kernel.dmesg_restrict) -> SKIP
2. the ring buffer WRAPS and every memattr line prints in the first fraction of a second, so the
collector records a boot_covered anchor (Linux version / BIOS-provided physical RAM map); no
memattr line WITHOUT that anchor is a SKIP, not a PASS
3. memory.physical_total needs no kernel log at all
Verified by forcing the unprivileged path: SKIP with the EPERM reason while zoneinfo still PASSed.
VERDICTS, confirmed live:
k3s-prod-1 WARN "EXPOSED, not immune ... survived only because its garbage version field read
1030, which is > 2 ... one unlucky boot from the failure state" + PASS 2.0%
k3s-prod-2 FAIL "THIS NODE HAS LOST MEMORY TO THE BUG ... at 0xbeeb0018" + FAIL 83.4%/83.6%
k3s-dev-* PASS both, NO false finding: "no UEFI config-table line ... did not boot via UEFI.
Log reaches back to early boot, so that absence is measured, not assumed."
Both failure texts carry the remediation verbatim, including that there is NO kernel cmdline
workaround because efi_memattr_init() runs before any efi= bail-out.
Root-gating follows the role's existing pattern: --privileged re-runs THE SAME collector under
become, only when the unprivileged pass actually failed to read something. Reimplementing the log
grep and SMBIOS parse in Jinja would have been a second source of truth for the same fact.
LATENT BUG FIXED in passing: the collector's run() used bare shutil.which(), so dmidecode (in
/usr/sbin) was invisible under the unprivileged k3s PATH and produced a FALSE "not installed"
skip reason. Now shares the sbin-aware which() collect_tools() already had.
INCIDENTAL FINDING recorded in §0b: node 1's DIMMs are NOT a matched pair — DIMM1 Micron
8ATF2G64HZ-3G2E1 (DDR4-3200), DIMM3 Micron 16ATF2G64HZ-2G6E1 (DDR4-2667). Both 16 GB so capacity
is right, but different rank organisation and rated speed, so the pair runs at 2667. Not a fault;
recorded because §2 requires the three nodes to end up symmetric and "2×16 GB Micron" hid it.
Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
A second research pass reached the same root cause by a different route and pinned the
arithmetic. Recording it because convergence from two independent lines is what turns this from
a good hypothesis into a settled diagnosis.
MECHANISM TO THE BYTE:
- the garbage 'version' node 2 read was 1463964749 = 0x5742544D = ASCII 'MTBW'. The kernel is
reading unrelated data. HP publishes the MEMATTR pointer into memory /sys/firmware/memmap calls
plain System RAM, so the pre-Linux boot chain overwrites it.
- tbl_size is a SIGNED int; num_entries * desc_size wraps mod 2^32 and goes NEGATIVE when bit 31
lands. memblock_reserve sign-extends it and memblock_cap_size clamps to PHYS_ADDR_MAX - base,
so EVERYTHING above 0xbeeb0018 is reserved.
- memblock_free_all then hands the allocator nothing above that line while present_pages (from
memblock.memory) is untouched -> present 3340288 / managed 0.
- 'Failed to map EFI Memory Attributes table' is emitted LATER by efi_memattr_apply_permissions();
it is positive PROOF the reservation already happened, a consequence rather than the cause.
CLEANEST A/B TELL, one dmesg line: NODE_DATA(0) on prod-2 ends at 0xbeeb0000 — pgdat forced
immediately BELOW the reservation — while on prod-1 it ends at 0x82f800000, the top of RAM.
Identical 0x2a980 allocation, opposite placement.
THERE IS NO KERNEL CMDLINE WORKAROUND. efi_memattr_init() is called from
efi_config_parse_tables(), and x86's efi_init() calls efi_config_init() BEFORE the
efi_runtime_disabled() bail-out — so noefi / efi=noruntime / efi=novamap do NOT prevent the
reservation. A fixed kernel is the only real fix.
EFI_MEMORY_SP / soft-reserved was the strongest competing theory and is EXCLUDED BY MEASUREMENT:
/sys/firmware/memmap/10/type reads 'System RAM'. Soft-reserved would print as 'Soft Reserved' and
is never memblock_add()ed, which would have left present at ~0 rather than 3340288.
Also records that the hardening landed exactly one release late for Debian 13 (bug present since
2016; 8fbe4c49c0cc first in v6.13; follow-up 5d0faa8e8369) and that a Debian BTS report against
src:linux is warranted — 6.12 is Debian 13's kernel for the release lifetime.
Serves: R-ENV-01, R-ENV-03. Task: T-7.1.
I framed this as an OPEN HARDWARE FAULT and blamed, in order: the DIMM, BIOS memory remapping,
and stale Fast Boot memory-training data. ALL THREE WERE WRONG. The hardware is fine.
ROOT CAUSE: HP's R21 firmware publishes a CORRUPT EFI Memory Attributes Table, and Linux 6.12
fails to validate it before reserving memory from its contents. drivers/firmware/efi/memattr.c
never validates num_entries or desc_size:
if (tbl->version > 2) { pr_warn(...); goto unmap; }
tbl_size = sizeof(*tbl) + tbl->num_entries * tbl->desc_size;
memblock_reserve(efi_mem_attr_table, tbl_size);
Garbage version > 2 -> kernel bails, memory intact. Garbage version <= 2 -> kernel multiplies two
garbage fields and reserves GIGABYTES from the table address to the top of RAM.
THE EVIDENCE THAT SETTLES IT: the same machine — same DIMM, same BIOS, same kernel, same cmdline —
booted with the FULL 15.9 GiB two boots earlier, with a byte-identical BIOS-e820 map. Boot -2:
15,917,744K available, 615,208K reserved, 'Unexpected EFI Memory Attributes table version
1463964749'. Boots -1/0: 2.6 GiB, 13.9 GiB reserved, 'Failed to map ... @ 0xbeeb0018'. No hardware
fault is intermittent in that shape or fixed by a reboot. MemTotal = DMA(3840) + DMA32(684481)
pages = 2,753,284 kB exactly, Normal contributing zero.
BOTH PROD NODES ARE AFFECTED. k3s-prod-1 prints 'Unexpected EFI Memory Attributes table version
1030' — also garbage, but >2, so the kernel bails and it keeps its 32 GiB BY LUCK, NOT IMMUNITY.
It is one unlucky boot from the identical fault. Dev nodes publish no memattr table at all
(Proxmox OVMF) and are unaffected.
FIX: kernel >= 6.13 — upstream 8fbe4c49c0cc 'efi/memattr: Ignore table if the size is clearly
bogus', whose commit message describes this verbatim. Verified absent from v6.12 and from current
linux-6.12.y. On Debian 13: linux-image-amd64 from trixie-backports.
DO NOT buy RAM, move the DIMM, RMA the board, or run memtest expecting failures. Rebooting,
disabling Fast Boot, clearing CMOS and unplugging USB are all the same lottery — they move where
the table lands and may LOOK like fixes.
Records every killed hypothesis with its disproof so none of them get re-run.
Serves: R-ENV-01, R-ENV-03. Task: T-7.1.
Operator asked for the default BIOS password. ANSWER: there is none, and no factory default
exists to look up. HP business desktops ship with no Setup and no Power-On password; the
is_enabled=0 we measured IS the factory state. Recovery for a forgotten password is the PSWD
jumper — and 'stringent security' must NOT be enabled, since HP documents that it removes the
physical bypass and turns a 5-minute fix into a signed per-serial HP service case.
CORRECTION, and the kernel driver misled me: I recorded that the firmware refused our write for
lack of BIOS-admin authentication. Wrong. hp-bioscfg passes the return of HP's WMI SetBIOSSetting
through hp_wmi_error_and_message(), which is THE ERROR TABLE FOR A DIFFERENT INTERFACE (the
"SECU" commands under HP_WMI_BIOS_GUID). Per HP's Client Management Interface white paper,
SetBIOSSetting 4 = 'Failed' (generic) and 6 = 'Access Denied'. We got 4, not 6 — and the driver
has a separate BIOS_ADMIN_NOT_SET = 0x38 -> -EPERM for the no-password case, which we also did
not get. 'Invalid command type' is the driver looking 4 up in the wrong table: a genuine upstream
bug worth reporting to platform-driver-x86.
Most likely real cause (inferred, testable): with an empty password the driver emits an UNTAGGED
empty UTF-16 string, while every HP-documented caller passes Password="<utf-16/>" even when
blank — the driver only prepends that prefix for a non-empty string. Setting a password may fix
writes, but by flipping onto the tagged code path, NOT because 0x4 meant unauthorised.
Also verified from source: hp-bioscfg CANNOT set a password. current_password and new_password
both funnel into store_password_instance(), which only strscpy's into driver RAM; new_password is
write-only into a buffer nothing ever reads.
Count corrected: 241 attributes exposed, 239 with a current_value, 215 non-empty. I had quoted
215 as the total; it was the non-empty subset.
Recommendation recorded: land read-only drift detection now (zero risk, no new credential, and it
already caught real drift); treat a Setup password as a deliberate experiment on node 3 only;
hp-repsetup as fallback but it needs Secure Boot off or a self-signed module; skip Sure Admin/BCU
at this scale.
Serves: R-ENV-03, R-SEC-01. Task: T-7.1.
Operator asked whether node 2's missing memory could be reserved for integrated graphics.
TESTED, and no: /proc/iomem shows 'c8800000-cc7fffff : Graphics Stolen Memory' = EXACTLY 64 MiB,
matching the BIOS Video Memory Size=64 MB setting, and it sits BELOW the 4 GiB line. The UHD 630
(i915 bound) takes precisely what it is configured to take. It cannot account for 13.3 GiB and is
not even in the affected address range.
CORRECTION to my own analysis, prompted by checking that hypothesis properly: I cited the
'last_pfn revised DOWN from 0x42f800 to 0xc3c10' line as evidence of a memory cap. That was a
MISREADING. x86 boot prints last_pfn twice — first max_pfn from the full e820, then max_low_pfn,
the end of RAM below 4 GiB. The e820 low usable region ends at 0xc3c0ffff, so the next PFN is
exactly 0xc3c10, which is the second value observed. It matches to the byte and is normal on any
machine with more than 4 GiB. I presented a routine boot message as a smoking gun, and the
'memory remapping above 4 GB not being applied' signature claim rested partly on it.
The real anomaly is narrower than first stated: the Normal zone is 'present' but 'managed 0',
13.3 GiB is reserved, and nothing in the boot log explains by what. Ruled out so far: DIMM health,
firmware inventory, memory-remap settings (none exist on this platform), MTRR trim, iomem
registration, and now integrated graphics. Leading remaining hypothesis is stale memory-training
data cached by Fast Boot — testable with one F10 toggle.
Serves: R-ENV-03. Task: T-7.1.
Records node 2's as-found hardware alongside node 1 and removes two bullets that my earlier
edit left duplicated/obsolete (the BIOS comparison is done; clear-CMOS was listed twice).
Makes the hardware ASYMMETRY explicit in the table rather than buried in prose: node 1 is a
~2011 Micron C400 SATA SSD at 69% life with 15 grown bad blocks and 187 erase failures; node 2
is a Samsung PM9A1 NVMe at 96% life, PASSED. Same chassis, same BIOS, same CPU, same partition
layout — very different storage. §2 requires an identical disk layout across all three nodes
because Longhorn replica placement assumes symmetry, so this belongs in the inventory where the
disk decision will be made.
Serves: R-ENV-03. Task: T-7.1.
I recorded that the hp-bioscfg attributes were 'writable from the OS' because current_value
has mode rw and no BIOS passwords are set. That was WRONG, and the firmware said so when the
write was actually attempted:
hp_bioscfg: Returned error 0x4, "Invalid command type" (surfaces as printf: I/O error)
HP requires BIOS-admin authentication for programmatic changes and neither credential is
provisioned on these nodes — Setup Password is_enabled=0 (role bios-admin), SPM
key_mechanism=not provisioned. Secure Boot is also enabled. File mode is not authority; the
firmware is.
What survives the correction, and it is the useful half: READS WORK. Diffing 215 attributes
between node 1 and node 2 is exactly how the drift in §0b was found — nobody spots
'After Power Loss=Power Off' by eye. So the §1 firmware checklist IS machine-CHECKABLE today
with reads alone, and node_health should assert it.
What does NOT survive: machine-ENFORCEABLE. Converging BIOS settings automatically would
require provisioning a BIOS Setup Password on every node — a stored credential that can rewrite
firmware. That is a real security decision and the operator's call, not something to default to.
Consequence for the node-2 memory fault: setting Fast Boot=Disable and After Power Loss=Power On
must be done in F10 on the box. Updated §0b and §1 accordingly.
Serves: R-ENV-03. Task: T-7.1.
Both nodes load the hp-bioscfg driver, exposing 215 BIOS attributes read/write at
/sys/class/firmware-attributes/. No BIOS passwords are set, so they are writable FROM THE OS.
RULED OUT as the cause of node 2's 2.6 GiB limit: firmware inventory is correct (BIOS itself
reports Memory Size=16384 MB); no memory-remap/above-4G attribute exists on this platform at
all; MTRRs are effectively identical with no trim warning; /proc/iomem registers the full
100000000-42f7fffff as System RAM and 127 memory blocks report online.
REAL DIFFERENCES FOUND (node1 works / node2 broken):
After Power Loss Power On / Power Off <- VIOLATES this document's own §1
Fast Boot Disable / Enable <- leading hypothesis
Legacy Boot Options Disable / Enable
TPM Activation Policy F1 to Boot / Allow user to reject
MS+Windows UEFI CA 2023 Enable / Disable
Leading hypothesis: HP Fast Boot reuses CACHED MEMORY-TRAINING DATA rather than retraining each
boot. A DIMM fitted while Fast Boot was on is a plausible route to memory being present,
inventoried, handed to the kernel — and then reserved. It is also the only memory-relevant
setting differing from the working node. NOT TESTED: changing it is a firmware write plus a
reboot and needs operator sanction.
BROADER FINDING for T-7.1: the §1 firmware checklist is now MACHINE-CHECKABLE rather than an
eyeball-at-F10 exercise, and machine-ENFORCEABLE too. That is precisely how node 2 came to have
After Power Loss=Power Off while node 1 has Power On — nobody could see the drift. Follow-up
recorded: extend node_health to assert the firmware checklist, add a role to converge it.
Raw BIOS dumps deliberately NOT committed — they carry serials, UUIDs and MAC addresses, same
reason node-health artifacts are gitignored. Only the findings are recorded.
Serves: R-ENV-03. Task: T-7.1.
k3s-prod-2 (192.168.2.11) came online 2026-09-01. Enabled in the inventory, baseline package
floor installed, first health run completed. Vaulted become works on it unchanged.
*** OPEN FAULT: the node can only address ~2.6 GiB of a healthy, correctly-installed 16 GB
DIMM. *** Recorded in PROD-PROVISIONING §0b.
dmidecode 16 GB Samsung M471A2K43DB1-CTD in DIMM1 @ 2667 MT/s (DIMM3 empty)
BIOS e820 reports ~16 GiB usable, incl. 0x100000000-0x42f7fffff
kernel Memory: 2568544K/16546088K available (13958980K RESERVED)
last_pfn revised DOWN 0x42f800 (~16.7 GiB) -> 0xc3c10 (~3.06 GiB) in early boot
zones Normal present 3340288 pages (~12.7 GiB) but MANAGED 0
MemTotal 2,753,284 kB = 2.6 GiB
The DIMM is NOT the problem: it is present, correctly identified, and at rated speed. The
kernel sees all 16 GB and then reserves 13.3 GiB. Everything below ~3 GiB usable, everything
above 4 GiB present-but-unmanaged — the signature of memory remapping above 4 GB not being
applied, not of failing memory. Node 1 is the control: identical chassis, identical BIOS R21
Ver. 02.26.00, identical CPU, and it addresses its full 31 GiB. Per-machine state, not a
firmware-version defect.
WARNING RECORDED: do NOT assume the incoming 32 GB fixes this. If the fault is the slot, the
board or a BIOS setting rather than the DIMM, 32 GB presents identically and the money is
wasted. Verify MemTotal after any RAM change.
ALSO FOUND — the prod nodes are NOT hardware-symmetric:
node 1 Micron C400 512GB SATA SSD (~2011) — 69% life, 15 grown bad blocks, 187 erase
failures, 32 unexpected power losses
node 2 Samsung PM9A1 512GB NVMe — 96% life, 2142 power-on hours, 34 unexpected
power losses, overall-health PASSED
Same chassis, same BIOS, same CPU, same partition layout — very different storage. §2 requires
an identical disk layout across all three nodes because Longhorn replica placement assumes
symmetry, so this asymmetry is now a live input to the disk decision rather than a footnote.
Node 2's NVMe is the drive node 1 should have had.
Node 2 otherwise: 0 FAIL / 3 WARN / 12 PASS / 2 SKIP. Same three WARNs as node 1 (swap still
active pending full baseline, single IPv4 resolver, and the NVMe's 34 unexpected power losses),
plus the same unknowable-ECC SKIP.
Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The second requirement ever proven on this project, and the one the operator had attested
was working but the suite could not attest to. Test authored 2026-08-31 carrying a
`conformance.home.k3s/never-executed` annotation; executed for the first time tonight.
test dr-restore run 20260902T001846Z-fdc3265
req R-STATE-03 (safety-critical, DR) junit passed / 0 failed / 0 skipped
sentinel DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK proof proven
sha fdc3265 (clean tree) kube v1.36.4+k3s1
WHAT IT ACTUALLY PROVED, in three steps:
1. The backup target is genuinely OFF-CLUSTER. BSL velero/default phase=Available at
http://192.168.1.9:9000 — endpoint resolved and asserted to be none of: loopback, any
node Internal/ExternalIP, any Service ClusterIP, any Pod IP, inside any node podCIDR, or
inside any ServiceCIDR.
2. Backups are SCHEDULED, not merely possible — velero-daily and
velero-daily-resources-only, both phase=Enabled, last run 2026-09-01T03:00:27Z.
3. The drill: seed a nanosecond-unique canary onto a longhorn PVC; Velero Backup to
Completed with a kopia PodVolumeBackup carrying a real snapshotID (volume DATA reached
the remote repository, not just resource JSON); ANNIHILATE the source — delete the
namespace, poll until gone, force-delete the PV, FAIL if the original PV survived;
Restore into a clean namespace; assert the restored PVC bound a DIFFERENT PV and the
canary bytes are byte-identical.
The bytes could only have come from outside the cluster.
VALIDATES THE PARSER FIX FROM 57d4df8. This log contains SEVEN occurrences of
DR_RESTORE_SKIPPED_OK — every one of them inside a `=== COMMAND` script echo — and the pass
token twice in real `=== STDOUT`. Under the pre-fix scanner this genuine pass would have been
recorded as `skipped`, exactly as node-loss was. The section-parsing rule is doing real work,
not theoretical work.
CLEANUP VERIFIED against a pre-run baseline: Velero backups back to 7 (its own
chainsaw-*-dr-backup deleted via DeleteBackupRequest), 0 chainsaw backups, 0 restores, 0
orphan Released/Available PVs, both scratch namespaces gone, 13/13 Kustomizations Ready, 0
pods not Running. It wrote into and cleaned out of the SAME bucket and kopia repository the
live nightly schedule uses, and left it as it found it.
SEPARATE FINDING, cleaned up: `chainsaw-stable-worm` was still Active from 2026-08-25 —
empty residue from an earlier conformance session that never tore its namespace down. Deleted.
Worth knowing that Chainsaw namespace leaks can outlive a run by a week unnoticed.
NOT proven by this test, stated in its own header rather than implied: off-HOST DR (the dev
nodes are VMs on the same Proxmox host as the S3 target, so this does not survive hypervisor
loss); no direct S3 client read; real service data (Nextcloud/Gitea/CNPG drills remain
operator exercises); bare-metal rebuild DR; encryption-at-rest; retention/GC.
Serves: R-STATE-03, R-TEST-03, R-TEST-05.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Committing the first PROVEN row immediately degraded it to `stale-proof`. Cause: a
self-defeating loop.
`IMPL_EXCLUDE_PREFIXES` excluded `conformance/tests/` and `ci/tests/` but NOT
`conformance/runs/`. The ledger and the preserved run logs CONTAIN the requirement IDs they
are evidence FOR — `ledger.jsonl` literally holds `"requirements":["R-STATE-02"]` — so they
were scanned as IMPLEMENTING evidence for R-STATE-02. That made `last_change` for the
requirement move to the ledger commit itself, which the row's `git_sha` necessarily predates,
so the freshness check failed.
The consequence was total, not marginal: **recording a proof destroyed it, so NO ledger row
could ever have been non-stale.** The ledger would have reported `PROVEN: 0 (stale N)` forever
while looking like it was working — the same shape of silent-uselessness as the COMMAND-echo
bug in the previous commit, and found the same way: by actually running the thing and
disbelieving a green-looking result.
Fix: exclude `conformance/runs/`. The ledger is the proof RECORD; it must not participate in
the freshness computation about itself.
Verified: `PROVEN: 1 (stale 0, self-skipped 0, failed 0, unknown 0) from 1 ledger row(s)`;
R-STATE-02 reads **proven**. status.py --selftest PASS. The staleness machinery is otherwise
untouched — a genuinely stale row (impl changed after the run) still degrades, which the
selftest continues to assert.
Serves: R-TEST-01..04.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Records the first requirement this project has ever PROVEN, in the only sense its own
definition of done accepts: a green, requirement-tagged, reproducible behaviour test,
executed against a real cluster, with the evidence committed.
test node-loss run 20260901T235117Z-e9a4b66
req R-STATE-02 (safety-critical) junit passed
sentinel NODE_LOSS_VOLUME_AVAILABLE_OK proof proven
sha 57d4df8 (clean tree) kube v1.36.4+k3s1
What actually ran: seed a 3-replica longhorn volume, cordon the node the writer landed on,
evict only its own pod, assert the volume stays available and a reader on another node
re-attaches and reads the data back. 131s. It deliberately does NOT drain, so kubelet and
the etcd member stay up and quorum is never touched — which is what retired the old
'defer node-loss to a maintenance window' reason as unfounded rather than by fiat.
Recorded against the clean tree of the preceding commit so git_sha resolves and the
freshness check passes. The test was NOT re-run for this: the preserved run.log is the
evidence, and re-running would have proved the parser rather than preserving the run.
Serves: R-STATE-02, R-TEST-03, R-TEST-05.
The run ledger mis-classified the FIRST real L3 execution — a genuine PASS recorded as
`skipped`. Diagnosis, from the preserved log, section-classified:
line 51 COMMAND echo NODE_LOSS_SKIPPED_OK <- script SOURCE
line 190 STDOUT NODE_LOSS_VOLUME_AVAILABLE_OK <- the only REAL emission
Chainsaw echoes each script into a `=== COMMAND` block before running it, and every
skip-gated test contains `echo <TEST>_SKIPPED_OK` as source text. record_run.py scanned the
raw log without distinguishing sections, so it matched the skip token in the echo.
NODE_LOSS_SKIPPED_OK never appeared in output at all.
SEVERITY: since every skip-gated test embeds its own skip token as source, and skip outranks
pass, THE LEDGER COULD NEVER HAVE RECORDED `proven` FOR ANY TEST WITH A SKIP GATE. It would
have silently recorded nothing forever while appearing to work. It failed CLOSED — the right
direction — but a ledger that always fails closed is an expensive way to write "unknown".
FIX: classify_sections() splits the log on indented `=== NAME` markers (a section runs to the
first non-blank line indented LESS than its marker). Verified against the real log's actual
markers, including that Go's column-0 `=== RUN/PAUSE/CONT` lines carry a test name and must
never open a section. Then:
PASS tokens are read from STDOUT ONLY — exactly where Chainsaw's own contains($stdout,…)
looks, so a source echo or stderr can never manufacture a proof.
SKIP tokens are read from every section that is NOT COMMAND, so an unknown future section
name fails closed in BOTH directions.
Lines outside any section prove nothing — this also closes a second false-positive vector:
a FAILING Chainsaw run prints the assertion expression, which contains both tokens.
A log with no markers is scanned whole exactly as before (such logs have no COMMAND echo).
SECOND BUG, same root: _LOG_HEADER_RE never matched a real Chainsaw log — the lines are
` l.go:52: <BS>| 23:51:18 | <ANSI>node-loss<ANSI> | …`, not `| … |`. EVERY genuine run was
silently falling into DEGRADED whole-log mode. Now strips ANSI/backspace noise first and
accepts the Go source prefix, so runs attribute per-testcase as designed.
Safety properties re-asserted, not assumed: skip still beats pass WITHIN real output; unknown
layers still require a sentinel; JUnit failed/error/skipped still outrank everything; missing
JUnit or log still exits 3. Negative control: with scan_texts forced section-blind in a
throwaway copy, exactly the new cases fail — they do catch this bug.
Includes the preserved run directory 20260901T235117Z-e9a4b66 (report.junit.xml + run.log)
for R-STATE-02 / node-loss — 131s, cordon -> evict -> volume stayed available -> reader
re-attached -> clean teardown, the first L3 fault-injection test ever executed on this
cluster. The ledger row is recorded in the FOLLOWING commit, deliberately: git_sha is sampled
at record time and a dirty tree records `-dirty`, which is unresolvable and correctly degrades
to stale-proof. Recording against this clean commit is the honest way to get a resolvable sha
— NOT a --git-sha override, which would be a lever for recording a clean sha over a dirty
tree and was deliberately not added.
Serves: R-TEST-01..04, R-STATE-02.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
status.py has always reported PROVEN: 0 for all 51 requirements, and named the fix in its
own header: "a committed, machine-readable run ledger". There was none — _artifacts/ is
gitignored, the root .gitignore kills *.junit.xml, and a stray JUnit was surfaced only as
a quarantined mtime advisory. Built BEFORE the first real L3 execution so that run is
captured rather than reconstructed from memory.
THE TRAP THIS EXISTS TO AVOID. These tests exit 0 and Chainsaw records `passed` on a
SELF-SKIP exactly as on a real run, and Chainsaw's JUnit carries no stdout — only
properties. So committing the JUnit as proof would let DR_RESTORE_SKIPPED_OK be recorded
as proof of R-STATE-03: authoritative-looking and wrong, which is worse than no ledger.
The sentinel token is the ONLY thing distinguishing proven from skipped, and it lives
solely in the run log. The ledger therefore records the LOG alongside the JUnit and
derives proof from the sentinel, never from the JUnit result.
Sentinel tokens are DERIVED, not hand-listed: record_run.py parses every
contains($stdout,'TOKEN') out of each test's own chainsaw-test.yaml, so the list cannot
drift from the tests. 23 of 35 tests declare no sentinel at all.
decide() is the single place `proven` is emitted, ordered WORST-FIRST:
JUnit failure/error -> failed
JUnit <skipped/> -> junit-skipped
ANY declared skip token in the log -> skipped (beats a green JUnit AND beats a pass
token present in the same log)
no sentinel declared -> proven ONLY for L0/L1 (whose JUnit is truthful on its own, since
run_l1_offline.py maps exit 3 to <skipped/>); written as an
explicit exempt-set so an unknown/blank layer fails CLOSED
EVERY declared pass token present -> proven
otherwise -> unknown-sentinel-absent
Log attribution is a state machine over Chainsaw's per-test headers; a log with no
recognisable header falls back to whole-log scanning restricted to GLOBALLY-UNIQUE tokens,
so DNS_OK (owned by two tests) can never prove either. record_run.py exits 3 if the JUnit
or the log is missing — nothing recorded, never green.
STALENESS: a proven row is honoured only while `git merge-base --is-ancestor` holds
between the requirement's last implementing/test change and the row's git_sha; otherwise
it degrades to stale-proof. A -dirty sha, an unknown rev, or an uncommitted test path all
fail ancestry the same way. A requirement reads proven only when EVERY verifying test
does (worst-of), and the latest row per test wins — not best-ever.
REFUSAL POSTURE PRESERVED: with an empty ledger the report is unchanged in intent — every
requirement not-derivable, headline "PROVEN: 0 — NOT DERIVABLE". This replaces "not
derivable" with real evidence only where evidence exists, never with optimism.
--record is explicit, never implicit (a run that silently records itself is how a bad run
becomes permanent false proof), is rejected for --status/--coverage-only/--lint, and
re-execs under tee returning ${PIPESTATUS[0]} so recording can never soften a gate. It
rm -f's a stale _artifacts/report.junit.xml first — a hazard hit live during development,
where a stale JUnit got paired with a fresh log.
Selftests: record_run.py 23 checks (incl. end-to-end Chainsaw-passed + NODE_LOSS_SKIPPED_OK
-> proof "skipped"), status.py +16 against a throwaway git repo (self-skip -> skipped;
proven-at-older-sha -> stale-proof; dirty/unresolvable -> stale-proof; worst-of;
latest-row-wins; malformed lines dropped). No regressions: --coverage-only, --status,
--lint, --strict-coverage and the merge-gate shape all rc=0 and unchanged; nothing in ci/
references status.py.
Root .gitignore needed no change — conformance/runs/ is not itself excluded, so the deeper
negations override the root's global *.junit.xml. Verified with git check-ignore.
Serves: R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Closes the transition opened in merge 1 (63f98aa). `prune: true` is restored on the
`platform` Kustomization now that the ownership transfer is VERIFIED on the live cluster,
not merely expected:
13/13 Kustomizations Ready, `cilium` among them, at main@sha1:23aa2854
Namespace/kube-system label kustomize.toolkit.fluxcd.io/name = cilium
HelmRelease/cilium label ... = cilium
HelmRepository/cilium label ... = cilium
platform inventory: 0 entries matching kube-system__Namespace | kube-system_cilium_*
cilium inventory: exactly those three objects
Cilium pods UNCHANGED from the pre-merge baseline —
cilium-2f8h8 r=0 2026-09-01T05:55:27Z
cilium-425m8 r=1 2026-08-30T13:25:10Z
cilium-54cww r=0 2026-09-01T05:55:26Z
HelmRelease still v11
So the relabel deterministically preceded any prune, exactly as the revision-gated
dependsOn edge predicted, and the ownership move cost zero pod restarts.
The in-file comment now records the lesson rather than the temporary state: before moving
ANY platform/* component in future, check whether it is the sole declarer of a shared
Namespace. platform/cilium/namespace.yaml was the only declaration of Namespace/kube-system
under ./platform, which would have turned a component-scoped prune into
`delete namespace kube-system` — taking coredns, metrics-server, local-path and Cilium.
L0 substitution gate: 103 assertions, 0 hazards.
Serves: R-PLAT-01, R-GITOPS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Cilium was one of ~12 components inside a single `platform` Kustomization with wait:true,
so the CNI's health was indistinguishable from eleven unrelated components in the tier's
Ready condition, and the only emergency valve was freezing the whole platform tier. This
repo already set the precedent in reverse (oauth2-proxy and headlamp were split OUT so
unhealthy leaves would not gate a tier); Cilium is the mirror image — the one component
that legitimately SHOULD gate everything.
New clusters/base/cilium.yaml: path ./platform/cilium, wait:true, prune:true, interval 1h
(a CNI does not need a 10m re-render cadence; kustomize-controller is source-triggered so
a push still reconciles immediately — the interval is only the drift-correction floor),
timeout 10m (a 3-node DaemonSet rollout with wait exceeds the 5m default),
dependsOn:[flux-system], and postBuild.substituteFrom: cluster-vars — the last is
LOAD-BEARING, not decorative: helmrelease.yaml references ${K3S_API_HOST} and
kustomize-controller now runs --feature-gates=StrictPostBuildSubstitutions=true, so
without it this Kustomization would fail immediately.
*** THE HAZARD WAS FAR WORSE THAN "the CNI gets pruned" ***
platform/cilium/namespace.yaml is the ONLY declaration of Namespace/kube-system anywhere
under ./platform, and the live object carries kustomize.toolkit.fluxcd.io/name=platform.
With prune:true, the moment ./platform/cilium leaves platform's path the stale set is not
"the Cilium HelmRelease" — it is `kubectl delete namespace kube-system`, taking coredns,
metrics-server, local-path AND Cilium. A full cluster kill, not a CNI kill.
TWO INDEPENDENT PROTECTIONS, deliberately not trusting either alone:
1. prune:false on `platform` for this transition merge — Flux's documented Procedure 1 for
moving resources between Kustomizations. Chosen over the per-resource
`kustomize.toolkit.fluxcd.io/prune: disabled` annotation because that needs THREE merges
and leaves the new Kustomization owning resources it can never prune.
2. The pruner's own owner-label filter. In v1.4.0, prune() passes Inclusions:
GetOwnerLabels(obj.Name...), so platform only deletes objects whose LIVE labels still
say name=platform. The new cilium Kustomization relabels all three to name=cilium on
apply, and platform.dependsOn:[cilium] is REVISION-GATED, so platform provably cannot
reconcile revision N until cilium has APPLIED revision N. The relabel deterministically
precedes the prune.
Protection 2 alone would probably have sufficed. Not betting kube-system on "probably".
NOTED FOR THE FUTURE, in-file: anyone moving a platform/* component must first check
whether it is the sole declarer of a shared Namespace — that turns a component-scoped
prune into a cluster-scoped one.
Verified: `kubectl kustomize platform` goes 88 -> 85 documents, 3 removed / 0 added /
0 CHANGED, and `kubectl kustomize platform/cilium` reproduces those 3 byte-identical
(sha256-matched). A pure ownership move; no Cilium value touched. L0 substitution gate
103 assertions (was 102), new cilium row, 0 hazards. Server-side dry-run against the live
cluster accepted both CRs. clusters/prod renders clean off the same base.
MERGE 2 (restore prune:true) follows only after verifying: 13/13 Ready; kube-system, the
HelmRelease and the HelmRepository all relabelled to name=cilium; platform's inventory no
longer lists them; and Cilium pod ages/restarts UNCHANGED from
cilium-2f8h8 r=0 2026-09-01T05:55:27Z / cilium-425m8 r=1 2026-08-30T13:25:10Z /
cilium-54cww r=0 2026-09-01T05:55:26Z, HelmRelease v11.
Serves: R-PLAT-01, R-GITOPS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Two bugs stacked and the outer one hid the inner. Fixing the dead OIDC guard made the
block execute for the first time in these apps' deployed lifetime, which immediately
exposed a latent defect: the hook's last statement was `[ cond ] && action`, so on the
SUCCESS path it returned 1 and became the hook's exit status — and a non-zero postStart
exit makes the kubelet kill the container.
Records the pattern for the next PM: fixing dead code is not a safe no-op. It is the first
execution of never-tested code and deserves the caution of new code, not of a
one-character escape.
Also records what is verified live (Cilium maxUnavailable applied with pod ages UNCHANGED,
proving nothing rolled), and that strict substitution is committed but NOT ACTIVE — it
needs bootstrap.sh step 1 because the controllers sit outside Flux's inventory.
LIVE REGRESSION, caught on the reconcile after merging #5. nextcloud went
CrashLoopBackOff with FailedPostStartHook, 7 restarts, and apps-wave-1/2 stalled.
ROOT CAUSE — a latent bug that could never fire until now. The final statement of
nextcloud's postStart script was:
[ "$p" -ge 15 ] && nextcloud_oidc_fail "could not promote ... after retries"
On the SUCCESS path (p < 15) the test returns 1, and as the script's LAST command that
becomes the hook's exit status. A non-zero postStart exit makes the kubelet KILL the
container. This was harmless for as long as the OIDC guard above it was dead code that
exited 0 first (see 3484955) — fixing that guard made the block execute for the first
time in these apps' deployed lifetime, and the bug fired immediately.
That is the honest sequence: the previous commit did not introduce this defect, it
UNCOVERED it. Two bugs stacked, and the outer one hid the inner one.
FIX: convert to a plain `if`, then `exit 0` explicitly, with a comment recording why.
Also made gitea's hook explicit. Its last statement already returned 0 on both paths, so
it was correct by accident of ordering; leaving one hook implicitly-safe and one
explicitly-safe invites exactly this class back. Both now end with an unconditional
`exit 0`, and both say why in-file.
Every failure path in both hooks still logs loudly to stderr AND writes a marker file on
the persistent volume, and deliberately does NOT crash the pod — the break-glass local
admin must stay usable. That contract is now actually honoured; before, the success path
was the one that crashed.
Fixed in catalog/ (the ServiceDefinitions are the source of truth) and goldens
regenerated; render==golden green, renderer selftest PASS.
Serves: R-IDP-02, R-IDP-06, R-GITOPS-02.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Operator decisions, 2026-09-01:
1. Prod uses the SAME SOPS age key as dev.
2. Prod tracks the SAME BRANCH as dev (`main`). Dev is a temporary instance and will be
decommissioned once prod is deemed stable.
SOPS: eight `decryption: {provider: sops, secretRef: sops-age}` patches mirroring dev
exactly — platform, apps-wave-0/1/2, oauth2-proxy, headlamp, crafty, velero. The sops-age
Secret is still created OUT OF BAND on prod (bootstrap/sops/install-age-key.sh); the
overlay references it by name only, never inlines key material.
RECORDED CONSEQUENCE of the shared branch: once prod is live, EVERY MERGE TO main
RECONCILES BOTH CLUSTERS. There is no promotion step and no soak window — dev and prod
move together. That is a deliberate consequence of dev being temporary, not an oversight,
but it removes the "prove it on dev first" safety net that the last few days have leaned
on heavily. Worth revisiting if dev outlives its expected life.
*** BOOTSTRAP BLOCKER recorded, NOT papered over *** — the source URL cannot work on a
cold prod cluster, for two independent reasons found while wiring this:
1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER — the one slated for
decommission. Prod syncing from it means prod's source of truth lives inside the
cluster being retired. Gitea must move to prod (or off-cluster) BEFORE dev is torn
down. This makes Gitea migration the FIRST task of the decommission plan, not the last.
2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml
answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP} via a template
plugin. On prod that is 192.168.2.20 — PROD's own gateway, where no Gitea listens. Prod
Flux would resolve gitea.home.keithjackson.dev to itself and never reach a repository.
This is the same split-horizon mechanism that makes dev work, inverted against us.
Bring-up therefore needs a TEMPORARY source (the bootstrap host-Gitea, or an off-cluster
mirror) with a cutover to the in-cluster URL once Gitea actually runs on prod — precisely
the sequence docs/05 describes for dev's own Phase-6 cutover. The branch is settled and
correct; only the host is blocked. Documented at the patch site and in
PROD-PROVISIONING.md §7.
Gate: MERGE GATE: PASS.
Serves: R-SEC-01, R-GITOPS-01, R-ENV-02, R-ENV-03. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
P0 — clusters/base/platform.yaml gains `dependsOn: [flux-system]`. This is the actual fix
for the incident, not the feature gate. Verified against kustomize-controller v1.4.0
`kustomization_controller.go :: checkDependencies` (our version): dependsOn is REVISION-
gated, not merely Ready-gated — when the dependency shares a GitRepository, the dependent
cannot reconcile revision N until the dependency has APPLIED revision N. Applying revision
N is exactly what writes a newly-added cluster-vars knob.
Graph verified mechanically, not by eye: built clusters/dev and clusters/prod and walked
the resulting objects. `platform` was the ONLY Kustomization without a dependsOn, and all
11 substituting Kustomizations chain transitively through it (platform-config -> platform;
the waves, apps-vms, oauth2-proxy, headlamp, homepage, crafty, velero -> platform-config).
The "not chaining through platform" set is EMPTY, so this one edge protects the whole
graph. DFS over the full edge set after the edit: no cycles. The root (clusters/dev/
sync.yaml) runs wait: false, which is what makes the edge deadlock-free — it reaches Ready
on apply success without health-gating its children.
Accepted trade-off, documented in-file: the root becomes a gate. A malformed manifest
anywhere under clusters/<env> — including clusters/base — now stalls platform and
everything downstream, where today it would not. Fail-closed, per doctrine.
P1 — StrictPostBuildSubstitutions. MY PREMISE WAS WRONG about where this belongs and it
matters: this repo deliberately does NOT use `flux bootstrap` (see bootstrap/flux/README).
There is no gotk-components.yaml in git, and the live kustomize-controller Deployment
carries NO Flux ownership labels — it was installed imperatively by `flux install` from
bootstrap.sh and is not in Flux's inventory. `flux-system` self-manages the CR graph, NOT
the controllers. Nothing under clusters/** can reach a controller flag.
So it uses Flux's documented mechanism for hand-installed clusters: a new
bootstrap/flux/components/kustomization.yaml wrapping `gotk-components.yaml` with a
JSON-6902 patch appending --feature-gates=StrictPostBuildSubstitutions=true to
kustomize-controller. JSON-6902 `add /args/-` deliberately, NOT strategic merge — a
container's args is atomic and a merge patch would replace all five install-generated
args. bootstrap.sh step 1 now renders (`flux install --export` into a temp dir, apply the
patch, ASSERT the gate string is present so a future Flux bump that breaks the patch
target fails loudly) then applies. FLUX_INSTALL_PLAIN=1 is a loud escape hatch. Nothing is
vendored, so components can never drift from the pinned FLUX_VERSION.
Rejected: vendoring gotk-components.yaml under the root's path (the root's prune: true
would put Flux's own controllers and CRDs in a prune inventory), and shipping a partial
Deployment as a Flux-managed resource (a future prune would delete kustomize-controller).
ROLLOUT ORDER MATTERS. P0 arrives automatically on the next reconcile. P1 does NOT arrive
on a git push — the controller Deployment is outside Flux's inventory, so an operator must
re-run bootstrap.sh step 1, which restarts kustomize-controller (~10-20s; Flux is
level-triggered so an interrupted reconcile is simply retried). P0 must be reconciled
BEFORE P1 is rolled out: with the gate on but the ordering edge absent, the same race
turns a transient empty render into a transient hard Kustomization failure.
ALSO FIXES A PHASE-7 BLOCKER THIS CHANGE WOULD OTHERWISE HAVE CREATED. clusters/prod had
no root Kustomization — dev/sync.yaml even carried the note "a prod overlay needs its own
sync.yaml -> ./clusters/prod". With the new dependsOn, a prod bootstrap would have left
platform blocked forever on a flux-system Kustomization that does not exist, and nothing
downstream would ever reconcile. clusters/prod/sync.yaml is now authored, mirroring dev.
Two further prod-overlay gaps found and DOCUMENTED, not guessed (PROD-PROVISIONING §7):
prod has no SOPS `decryption` patch (every encrypted Secret would fail to decrypt — needs
the prod age-key decision, a security choice not a copy), and no GitRepository source URL
patch (needs a decision on whether prod tracks the same branch as dev, which would make a
dev-facing merge reconcile prod).
Substitution gate re-run: check OK, 102 assertions, 0 hazards. Gate: MERGE GATE: PASS.
Serves: R-ENV-02, R-GITOPS-01, R-PLAT-01, R-ENV-03.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
During the substitution incident a bad render (k8sServiceHost: "") took down TWO of three
Cilium agents simultaneously. The agents' `config` init container falls back to the API
ClusterIP, which only the Cilium agent itself can translate — and that agent is the one
being replaced — so it deadlocks on `dial tcp 10.43.0.1:443: i/o timeout`.
Root cause of the 2-at-once: `updateStrategy.rollingUpdate.maxUnavailable: 2` is the
UPSTREAM CHART DEFAULT, confirmed by downloading cilium-1.20.1.tgz and reading
values.yaml:437-444 — not a local choice, and not overridden anywhere in this repo.
maxUnavailable: 1 makes a rollout replace ONE agent at a time and stop at the first pod
that fails to become Ready. The whole value of the setting is that the rollout halts on
the first bad node instead of taking the majority of the dataplane with it, and that
guarantee only exists at 1. It matters far more on a cold PROD bootstrap, where there is
no healthy peer to recover from — dev only self-healed because k3s-dev-3 kept a working
agent throughout.
Set as a [parity] field in ALL THREE places, per this repo's documented contract that the
Ansible bootstrap values and the Flux HelmRelease values must agree on settings that
define cluster networking (Ansible installs Cilium at bootstrap; the HelmRelease adopts
it; the two layers fight otherwise):
platform/cilium/helmrelease.yaml (authoritative for Flux)
infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 (authoritative for bootstrap)
platform/cilium/values.yaml (reference mirror, kept in sync)
Verified by rendering the .j2 with roles/cilium/defaults and diffing full value sets: every
[parity] field is byte-equal across all three; only k8sServiceHost differs, as designed
(${K3S_API_HOST} vs {{ k3s_api_host }}).
MERGING THIS IS NON-DISRUPTIVE. updateStrategy lives at DaemonSet.spec.updateStrategy — a
SIBLING of spec.template (chart daemonset.yaml:49-53). The DaemonSet controller decides a
pod is out of date by hashing spec.template alone, so the pod-template hash does not move,
no new ControllerRevision is created, and zero pods are recreated. Helm patches the field
in place. (If it were disruptive it would be self-defeating: rolling agents under the old
maxUnavailable: 2 in order to install maxUnavailable: 1.) Verify after reconcile with
`kubectl -n kube-system get ds cilium -o jsonpath='{.spec.updateStrategy.rollingUpdate.maxUnavailable}'`
plus unchanged pod ages — that is the cleanest proof nothing rolled.
Trade-off recorded in the comments: upgrades serialise, roughly two waves to three at 3
nodes (a minute or two). On a large cluster this becomes painful and a percentage such as
"10%" would be right — noted so a future larger cluster revisits it deliberately.
SCOPE: this bounds blast radius, it does NOT close the substitution race. A bad render
still ships; it now breaks one node and stalls rather than taking the majority.
platform.dependsOn: [flux-system] (P0) is the fix for the cause.
Serves: R-PLAT-01, R-NET-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Records three parallel research passes and, importantly, TWO CORRECTIONS.
Correction 1 (mine): StrictPostBuildSubstitutions alone is INSUFFICIENT — it fixes nothing
about ordering. The actual fix is one line: platform.dependsOn: [flux-system]. Verified
against kustomization_controller.go::checkDependencies at our exact version — dependsOn is
REVISION-gated, not merely Ready-gated, so a dependent sharing a GitRepository cannot
reconcile revision N until the dependency has APPLIED revision N.
Correction 2 (a research agent's): the claim that Helm considered the bad release healthy,
so remediation never fired, is WRONG — measured on the live cluster, the Helm release
secrets show v9 = failed. The machinery did fire. maxUnavailable: 2 of 3 IS confirmed, so
the real harm was blast radius, not missed detection.
Adds a MEASURED substitution-semantics table, including the two anti-patterns nobody had
written down: ${VAR:?msg} renders the MESSAGE as the value, and ${VAR:=default} silently
ships the dev value to prod. Both are exempt from strict mode by design — which is exactly
why strict would never have caught the gitea/nextcloud dead-guard bug.
Also corrects repo doctrine: escaping is TOTAL ($${ARR[0]} is safe); only the unescaped
indexed form is fatal. And qualifies an earlier overstatement in this log: kustomize strips
structural YAML comments before envsubst, so the hazard is block-scalar bodies, not all
comments.
Notes Flux 2.4.0 is out of support and that strict became default-on in kustomize-controller
v1.9.0 — so the audit is a prerequisite for the upgrade regardless.
Ranked plan P0-P5 with what is done, what is ready, and what needs an operator call.
Adds conformance/tests/l0-static/flux-postbuild-substitution/ — an offline, stdlib-only
L0 check tagged R-ENV-02 + R-GITOPS-01 (deliberately NOT R-TEST-01, which would
overclaim). It reuses lib/status.py's `resources:` walk rather than forking it, so "what a
path renders" has exactly one definition. It asserts: every ${...} on a substituted path
is defined-or-escaped (comments included); no default-operator form on an undefined name;
no unparseable expansion; the rendering path actually substitutes cluster-vars; every
overlay-patched /data/KEY has a base default; and that the gate itself is load-bearing.
WIRED AS AN UNCONDITIONAL STAGE (3c) in ci/l0.sh, and that matters: stage 3b scopes to
touched requirements, and ci/touched-requirements.sh maps only clusters/* to R-ENV-02 —
so an edit to a substituted manifest under apps/ or platform/ would have filtered this
check out of exactly the change that introduces the bug. l0 fail -> BLOCK=1 in gate.sh.
Mutation-tested all five assertion classes (undefined token, indexed expansion, :- form,
missing substituteFrom, un-wired gate); each fails as intended, baseline green.
STRICT MODE IS PROVEN SAFE TO ENABLE — measured, not inspected. For all 11 substituted
paths, `kubectl kustomize --load-restrictor LoadRestrictionsNone <path> | flux envsubst
--strict` exits 0 against cluster-vars base defaults. Negative control confirms the
incident: with K3S_API_HOST unset, non-strict renders k8sServiceHost empty while strict
exits 1 with `variable not set (strict mode)`. The feature gate itself is NOT flipped
here — that is a live-cluster change and the operator's call.
Escapes the six comment hazards that stood in the way (apps/vms/unifi x4,
platform/longhorn x2). All were prose ABOUT the escaping convention; since the repo's own
contract is that $${...} emits a literal ${...}, escaping them is both precise and
self-consistent. kustomize strips these particular comments today, but comment survival is
an implementation detail — a body comment carrying $ WAS observed surviving into built
apps/wave-1 — and a survivor would kill the whole Kustomization.
DOCTRINE CORRECTED (measured against flux 2.4.0): clusters/base/apps-vms.yaml claimed an
indexed expansion is fatal "escaped or not". Escaping is TOTAL — $${ARR[0]} emits the
literal text safely; only the UNESCAPED indexed form is fatal. Corrected in place.
Also empties DOCUMENTED_GAPS in conformance/run.sh with rationale: R-ENV-02 and
R-GITOPS-01 now have a real verifying test, which run.sh's own comment named as the
trigger for removing them.
Advisory output (not failures): 6 dead cluster-vars knobs — BASE_DOMAIN, CLUSTER_ENV,
CLUSTER_POD_CIDR, PIA_REGION, PLATFORM_NODE_SELECTOR_{KEY,VALUE}. The last four are
documented reference-only; BASE_DOMAIN and CLUSTER_ENV look like genuine rot (hostnames
are hardcoded elsewhere) and are worth a follow-up.
Gate: MERGE GATE: PASS. 102 assertions, 0 hazards.
Serves: R-ENV-02, R-GITOPS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Both ServiceDefinitions guarded their Keycloak OIDC registration with an UNESCAPED
braced expansion:
if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then # gitea
if [ -z "${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then # nextcloud, then exit 0
apps-wave-0/1 both declare postBuild.substituteFrom: cluster-vars, and neither name is a
cluster-vars key — so Flux's envsubst substituted them away AT RECONCILE TIME, long
before the container ever ran. Verified in the LIVE cluster: the deployed Deployments
literally contain
if [ -z "" ]; then
which is always true. The entire OIDC registration block has been skipped on EVERY boot
of both apps.
WHY NOBODY NOTICED: both apps DO have the provider registered — `gitea admin auth list`
shows keycloak/OAuth2/true and `occ user_oidc:provider` shows the keycloak provider.
Those live in each app's own database, persisted from an earlier successful registration.
SSO works today and DECISIONS-LOG is not wrong. But the idempotent re-registration is
dead, so any PVC rebuild, DB restore, or fresh cluster build comes up with NO OIDC and
only the break-glass local admin. A PROD BUILD FROM THIS REPO WOULD SILENTLY HAVE NO SSO
on gitea or nextcloud.
The irony: the surrounding code was hardened twice — see its own comments about a "silent
first-boot RACE", the verify-then-retry loops, the marker files. All of that hardening was
unreachable, because the guard above it always evaluated true.
Not a regression from the recent R-ENV-02 tokenization — introduced 2026-08-27 (8738871).
FIXED AT THE RIGHT SEAM: apps/wave-0/gitea.yaml and apps/wave-1/nextcloud.yaml are
RENDER-GENERATED, so the fix is in catalog/*.yaml and the goldens are regenerated.
Escaped to $${...}, the convention documented in clusters/base/apps-vms.yaml. Post-envsubst
the container now receives the intended `${VAR:-}` instead of `""`.
STRICT MODE WOULD NEVER HAVE CAUGHT THIS. Flux deliberately exempts default-providing
operators (-, :-, +, :+, =, :=, ?, :?) from StrictPostBuildSubstitutions, because they
"intentionally handle the unset case". So a targeted check was required.
MADE UN-REPEATABLE IN THE RENDERER: catalog/render.py gains a substitution-safety section
wired into emit(), so every render path refuses to emit a manifest containing a braced
expansion that is not exactly ${KNOWN_KEY} for a key read LIVE from cluster-vars (no
hardcoded copy to drift; unreadable => empty set => fail-closed). Dollar-runs are counted
properly ($$ is a literal $ to envsubst, so odd runs are flagged and even runs are not),
and indexed forms get their own hard-error message. Nine new --selftest assertions,
including the exact regression. Proven to bite: rendering the pre-fix catalog files from
HEAD now exits 1 with the precise diagnosis.
Sweep: the complete substituted surface was enumerated from every Kustomization's path.
catalog/ contains exactly TWO braced expansions and both were these bugs; every other
braced expansion on a substituted path is a declared cluster-vars key (zero typos, zero
undeclared names); existing $${UOS_URL} escapes verified intact.
Verified: --selftest PASS; canonical render is byte-identical to the committed apps/ for
all 10 onboarded services; the three L1 render checks pass.
NEXT RECONCILE: both pod templates change, so gitea (wave 0) and nextcloud (wave 1) roll —
strategy: Recreate on RWO Longhorn PVCs, so expect a brief single-pod outage each. The
OIDC block then executes for the first time in these apps' deployed lifetime. Both paths
are idempotent (gitea takes auth update-oauth on the existing id; nextcloud's
user_oidc:provider is create-or-update), so the expected result is a no-op re-write plus
the verify loop logging success. If the SOPS client secret ever diverged from Keycloak's,
this is when it gets corrected instead of silently left stale. Tail both pods after merge.
Serves: R-IDP-02, R-IDP-06, R-ENV-02, R-GITOPS-02.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Merging PR #4 put cilium agents into Init:Error on 2/3 dev nodes for ~6 minutes. Self-healed;
cluster fully green. Recorded because the mechanism is systemic and far more dangerous on a
cold prod bootstrap.
Root cause: cluster-vars (which DEFINES a knob) is owned by the flux-system Kustomization,
while platform/ (which REFERENCES it) is owned by the platform Kustomization, with no
dependency ordering between defining a variable and using it. Flux post-build substitution is
NON-STRICT by default, so during the window where platform rendered the new manifest against
the old cluster-vars, ${K3S_API_HOST} became the EMPTY STRING. Cilium upgraded with
k8sServiceHost: "" and fell back to ClusterIP 10.43.0.1 — which only the Cilium agent can
translate, and that agent was the thing restarting. Its config init container then could not
reach the API server at all.
Proven by the HelmRelease config digest going e83bf695 -> eebd6c88 -> e83bf695: the config
changed and then changed back to BYTE-IDENTICAL. The pre-merge byte-identity verification was
correct — the steady state is identical; the damage was a transient intermediate render.
optional: false does NOT protect against this: it requires the ConfigMap to exist, not that
every referenced key is defined.
Fix recommended, NOT applied unilaterally: enable the StrictPostBuildSubstitutions feature gate
on kustomize-controller so an undefined variable FAILS the Kustomization instead of silently
emptying it — fail-closed, per this project's own doctrine. Requires first proving every ${VAR}
in every substituted path is defined or correctly escaped, or it would fail legitimate embedded
shell expansions. Interim discipline: add a knob in one commit, reference it in a later one.
Serves: R-ENV-02, R-GITOPS-01, R-PLAT-01.
| 1 Cluster bring-up | 3-node k3s HA, embedded etcd, kube-vip VIP, Cilium, Flux, SOPS. Originally Proxmox VMs; **now 3× mini-PC bare metal** (`k3s-prod-1/2/3`, `192.168.2.10/.11/.12`, k3s v1.36.4+k3s1, Debian 13) |
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the authoritative LAN resolver** for `${BASE_DOMAIN}` on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files |
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero still blocked on OQ-BKP**; image-automation authored but disabled |
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero unblocked 2026-09-13** — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has **no** backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled |
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** |
| **7 Production readiness** | **IN PROGRESS** (DR restore proven 2026-09-14; node-loss outstanding). Prod live on bare metal serving the apex; dev decommissioned; CNPG HA 3 instances with primaries spread one per node; NFS bulk media + legacy Nextcloud mounted; VPN egress on prod. Alerting made real 2026-09-13 (Longhorn, cert-manager, Flux and twelve app SLIs had all been unable to fire). **Open: `node-loss` (R-STATE-02) re-proven on prod.** |
> - VM — **Running** on **`k3s-prod-1`**, guest agent connected, Ubuntu 24.04.4 LTS,
> `LiveMigratable=True` / `StorageLiveMigratable=True`. VMI age 34h, VM 8d.
> - L4 LoadBalancer VIP **`192.168.2.21`** — allocated and held on both Services.
> - **UniFi OS Server IS INSTALLED.** This reverses the previous entry here. See below.
> - **The guest has no IP address.**That — not a missing install — is why `:11443`, `:8443` and
> `:8080` refuse connections.
## What is actually wrong (2026-09-14)
The earlier note in this file said "UOS is NOT installed; the guest is a bare, prepped Ubuntu".
That was wrong, and acting on it would have meant reinstalling over a working controller.
Evidence, all read-only, from the live cluster:
| Claim | How it was checked | Result |
|---|---|---|
| UOS is installed | `kubectl -n unifi logs virt-launcher-unifi-… -c guest-console-log` | the current boot starts **`uosserver.service` — UniFi OS Server Service** and **`uosserver-updater.service`** |
| …and it is a real install, not a stub | KubeVirt guest-agent `filesystemlist` subresource | `/` is **6.3 GB used of 40 GB** — a bare Ubuntu cloud image with our packages is ~2 GB |
| the guest has no IP | `kubectl -n unifi get vmi unifi -o json` → `.status.interfaces` | `enp1s0`, `linkState: up`, **no `ipAddress` field at all**; the `kubectl get vmi` IP column is empty |
| …confirmed at the link layer | `kubectl -n unifi exec virt-launcher-… -c compute -- ip neigh` | **`10.0.2.2 dev k6t-eth0 FAILED`** — the guest does not answer ARP for its masquerade address |
| the ports are dead end-to-end | `/dev/tcp` probes from inside virt-launcher, to the pod IP **and** to `10.0.2.2` directly | 11443 / 8443 / 8080 / **22** all refused — i.e. the VIP and the CNP are exonerated |
So the controller is installed and its units start; nothing downstream of the guest's own network
stack can work. The interface was fine at boot (the console log reaches `network-online.target` at
~10s and cloud-init completes at 18.5s), so the address was lost afterwards — the obvious suspect is
UOS taking over host networking, but **that has not been proven and cannot be, from outside the
guest.** Diagnosing it needs a shell, which is the second problem below.
**There is currently no way into this guest.** The shipped cloud-init installs no SSH key and sets
no password, so the stock `ubuntu` account is locked and a serial console reaches a login prompt
nobody can answer. SSH would not rescue it either: the masquerade interface forwards **only** the
ports declared in `30-virtualmachine.yaml`, and 22 is not one of them — the probe above shows the
guest RSTs on 22 exactly as it does on 11443. Closing that hole is what `console-password` in the
new Secret is for.
## What this is
@@ -29,24 +59,39 @@ datastore** inside a Podman container stack — so **no separate MongoDB compani
| `10-default-deny.yaml` | R-NET-01 baseline: deny all, allow only DNS |
| `20-datavolume.yaml` | CDI **http** import of the Ubuntu 24.04 cloud image onto a `longhorn-vm`**RWX/Block** disk (**40Gi**). First real CDI http import on the cluster |
| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, a **pinned firmware UUID** (= the cloud-init instance-id), the **UOSDATA secret disk**, and a minimal **inline** cloud-init that installs `install-uos` + a per-boot systemd unit |
- "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n i=$((i + 1))\n [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n n=0\n until su-exec git gitea admin user create \\\n --username \"$GITEA_ADMIN_USERNAME\" \\\n --password \"$GITEA_ADMIN_PASSWORD\" \\\n --email \"[email protected]\" \\\n --admin --must-change-password=false; do\n n=$((n + 1))\n [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n sleep 3\n done\n echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\nif [ -z \"${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n d=0; disc_ok=0\n while [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1; break\n fi\n d=$((d + 1)); sleep 2\n done\n [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n r=0; reg_ok=0\n while [ \"$r\" -lt 10 ]; do\n oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n if [ -n \"$oidc_id\" ]; then\n su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n else\n su-exec git gitea admin auth add-oauth \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name\
\ groups --admin-group platform-admins >/dev/null 2>&1 || true\n fi\n # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\n done\n if [ \"$reg_ok\" = 1 ]; then\n echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n else\n gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n fi\nfi\n"
- "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n i=$((i + 1))\n [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n n=0\n until su-exec git gitea admin user create \\\n --username \"$GITEA_ADMIN_USERNAME\" \\\n --password \"$GITEA_ADMIN_PASSWORD\" \\\n --email \"[email protected]\" \\\n --admin --must-change-password=false; do\n n=$((n + 1))\n [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n sleep 3\n done\n echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not\n# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS TRUE\n# \u2014 making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on\n# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was\n# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.\n# Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator forms by design.\n# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.\n# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why the\n# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n d=0; disc_ok=0\n while [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1;\
\ break\n fi\n d=$((d + 1)); sleep 2\n done\n [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n r=0; reg_ok=0\n while [ \"$r\" -lt 10 ]; do\n oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n if [ -n \"$oidc_id\" ]; then\n su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n else\n su-exec git gitea admin auth add-oauth \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n fi\n # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\n done\n if [ \"$reg_ok\" = 1 ]; then\n echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n else\n gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n fi\nfi\n# A postStart hook MUST exit 0 \u2014 a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff). This hook already ends 0 on both paths, but\n# it is made EXPLICIT so the guarantee does not depend on whatever statement happens to\n# be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,\n# the moment its OIDC guard was fixed and the block actually ran for the first time.\nexit 0\n"
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): gitea-allow-internet-egress
apiVersion:cilium.io/v2
kind:CiliumNetworkPolicy
@@ -207,7 +228,7 @@ spec:
namespace:gateway
sectionName:https
hostnames:
- gitea.home.keithjackson.dev
- gitea.${BASE_DOMAIN}
rules:
- matches:
- path:
@@ -217,26 +238,6 @@ spec:
- name:gitea
port:3000
---
# ServiceMonitor (R-OBS-01)
apiVersion:monitoring.coreos.com/v1
kind:ServiceMonitor
metadata:
name:gitea
namespace:gitea
labels:
release:kube-prometheus-stack
app.kubernetes.io/name:gitea
spec:
selector:
matchLabels:
app.kubernetes.io/name:gitea
namespaceSelector:
matchNames:
- gitea
endpoints:
- port:metrics
interval:30s
---
# PrometheusRule SLI alerts (R-OBS-01)
apiVersion:monitoring.coreos.com/v1
kind:PrometheusRule
@@ -251,11 +252,11 @@ spec:
- name:gitea.slis
rules:
- alert:gitea-down
expr:up{job="gitea"} == 0
expr:kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"} == 0 or absent(kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"})
for:5m
labels:
severity:critical
service:gitea
annotations:
summary:gitea is down
description:'gitea has been unreachable for 5m (SLI:up).'
description:'gitea:its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI:up, via kube_deployment_status_replicas_available).'
expr:kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"} == 0 or absent(kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"})
for:5m
labels:
severity:critical
service:jackett
annotations:
summary:jackett is down
description:'jackett has been unreachable for 5m (SLI:up).'
description:'jackett:its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI:up, via kube_deployment_status_replicas_available).'
expr:kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"} == 0 or absent(kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"})
for:5m
labels:
severity:critical
service:pihole
annotations:
summary:pihole is down
description:'pihole has been unreachable for 5m (SLI:up).'
description:'pihole:its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI:up, via kube_deployment_status_replicas_available).'
expr:kube_deployment_status_replicas_available{namespace="plex", deployment="plex"} == 0 or absent(kube_deployment_status_replicas_available{namespace="plex", deployment="plex"})
for:5m
labels:
severity:critical
service:plex
annotations:
summary:plex is down
description:'plex has been unreachable for 5m (SLI:up).'
description:'plex:its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI:up, via kube_deployment_status_replicas_available).'
- "# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the\n# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.\n#\n# THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.\n# apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over\n# this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;\n# enforced by render.py's assert_substitution_safe) makes such a form correct ON THE\n# CLUSTER \u2014 but only there:\n# envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl\n# apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during\n# development) keeps the doubled dollar, which bash reads as its own PID. Not hypothetical;\n# see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`\n# are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY\n# however it is run.\nset -u\nCONF=/config/qBittorrent/qBittorrent.conf\nSVC=/run/service/svc-qbittorrent\n# The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this\n# ServiceDefinition \u2014 which is the single place the renderer takes the containerPort, the\n# Service port and the HTTPRoute backend port from \u2014 and the image exposes no WEBUI_PORT\n# env var for a lookup to read anyway.\n# It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and\n# that cost real time: run without envsubst, bash read the doubled dollar as its own PID,\n# so PORT became \"<pid>{WEBUI_PORT:-8080}\" \u2014 the brace never expanded, the digits never\n# matched. Every curl below then got an unparseable URL, returned nothing, and the hook\n# logged \"the WebUI rejected the durable\n# credential\" \u2014 having never actually asked it. That MISDIAGNOSIS is preserved in this\n# service's own marker log at 2026-09-13T05:35Z (three \"rejected\" lines, then GAVE UP,\n# against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.\nPORT=8080\n# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` \u2014 so a hook\n# that only echoes is effectively silent (gitea's hits the same wall and writes a marker\n# file for it). Every outcome therefore also lands, timestamped, on the persistent config\n# volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the\n# credential from git was asserted and accepted on this start.\nlog() {\n echo \"[qbt-webui-cred] $*\"\n printf '%s [qbt-webui-cred] %s\\n' \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\" \"$*\" \\\n >> /config/qbittorrent-webui-cred.log 2>/dev/null || true\n}\n\n# (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC\n# before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of\n# racing to create one it would then not overwrite.\nn=0\nwhile [ ! -s \"$CONF\" ] && [ \"$n\" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done\n\nwaited=0\nattempt=0\nwhile [ \"$attempt\" -lt 3 ]; do\n attempt=$((attempt + 1))\n\n # (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),\n # 10 = rewritten, anything else = could not write.\n python3 - \"$CONF\" <<'PY'\nimport base64, hashlib, hmac, os, stat, sys\n\nconf = sys.argv[1]\nuser = os.environ.get(\"QBT_WEBUI_USERNAME\", \"\")\npw = os.environ.get(\"QBT_WEBUI_PASSWORD\", \"\")\nif not user or not pw:\n sys.stderr.write(\"[qbt-webui-cred] credential env is empty - refusing to touch the conf\\n\")\n raise SystemExit(1)\n\n# qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000\n# iterations, 16-byte random salt, 64-byte digest, stored as\n# WebUI\\Password_PBKDF2=\"@ByteArray(<base64 salt>:<base64 digest>)\"\n# Get any one of them wrong and the app simply rejects every login.\nITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16\nUKEY, PKEY, SECTION = \"WebUI\\\\Username\", \"WebUI\\\\Password_PBKDF2\"\
, \"[Preferences]\"\n\ndef matches(raw):\n \"\"\"True iff the stored digest is one OUR password produces (salt is read from it).\"\"\"\n v = raw.strip().strip('\"')\n if v.startswith(\"@ByteArray(\") and v.endswith(\")\"):\n v = v[len(\"@ByteArray(\"):-1]\n try:\n salt_b64, digest_b64 = v.split(\":\", 1)\n salt = base64.b64decode(salt_b64, validate=True)\n digest = base64.b64decode(digest_b64, validate=True)\n except Exception:\n return False\n if not salt or not digest:\n return False\n return hmac.compare_digest(\n hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, len(digest)), digest)\n\ntry:\n lines = open(conf, encoding=\"utf-8\", errors=\"replace\").read().splitlines()\nexcept FileNotFoundError:\n lines = []\n\ndef value_of(key):\n for i, line in enumerate(lines):\n if line.startswith(key + \"=\"):\n return i, line.split(\"=\", 1)[1]\n return -1, None\n\nui, uval = value_of(UKEY)\npi, pval = value_of(PKEY)\nif uval == user and pval is not None and matches(pval):\n raise SystemExit(0) # converged - leave the file (and its salt) alone\n\nsalt = os.urandom(SALT_LEN)\ndigest = hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, DIGEST_LEN)\nstored = base64.b64encode(salt).decode() + \":\" + base64.b64encode(digest).decode()\ndesired = [UKEY + \"=\" + user, PKEY + '=\"@ByteArray(' + stored + ')\"']\n\nout = [line for i, line in enumerate(lines) if i not in (ui, pi)]\nif SECTION in out:\n at = out.index(SECTION) + 1\nelse:\n if out and out[-1].strip():\n out.append(\"\")\n out.append(SECTION)\n at = len(out)\nout[at:at] = desired\n\ntry:\n os.makedirs(os.path.dirname(conf) or \".\", exist_ok=True)\n tmp = conf + \".webui-cred.new\"\n with open(tmp, \"w\", encoding=\"utf-8\") as fh:\n fh.write(\"\\n\".join(out) + \"\\n\")\n fh.flush()\n os.fsync(fh.fileno())\n # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would\n # cost the app the ability to save its own settings, so carry the ownership over.\n try:\n st = os.stat(conf)\n os.chown(tmp, st.st_uid, st.st_gid)\n os.chmod(tmp, stat.S_IMODE(st.st_mode))\n except FileNotFoundError:\n os.chown(tmp, int(os.environ.get(\"PUID\", \"1000\")),\n int(os.environ.get(\"PGID\", \"1000\")))\n os.chmod(tmp, 0o664)\n os.replace(tmp, conf) # atomic: the app never reads a half-written conf\nexcept OSError as exc:\n sys.stderr.write(\"[qbt-webui-cred] cannot write %s: %s\\n\" % (conf, exc))\n raise SystemExit(1)\nraise SystemExit(10)\nPY\n rc=$?\n\n if [ \"$rc\" = 0 ]; then\n log \"conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)\"\n elif [ \"$rc\" = 10 ]; then\n log \"asserted WebUI Username + Password_PBKDF2 into $CONF\"\n # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY\n # have read the old, credential-less conf and be serving a random temporary password.\n # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.\n # Nothing to do when the service has not started yet - it will read the new file.\n # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the\n # 2026-09-13 failure spent the entire verification budget before the first login and\n # then fired attempts 2 and 3 with no wait left at all.\n sv=0\n while [ \"$sv\" -lt 60 ] && [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" != \"true\" ]; do\n sv=$((sv + 1)); sleep 1\n done\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"qbittorrent-nox was already running - restarting it to load the credential\"\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\"\n sleep 3\n fi\n else\n log \"FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep\" >&2\n log \"minting a random password\
\ every start and BOTH arrs will break. Fix and restart.\" >&2\n exit 0\n fi\n\n # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL\n # (shared ~120s budget across attempts), then spend exactly ONE login per attempt -\n # at most 3 rejected logins, which stays under qBittorrent's default ban threshold.\n #\n # \"THE API NEVER ANSWERED\" IS A DIFFERENT FAULT FROM \"THE PASSWORD WAS REJECTED\", and\n # conflating them is what made this hook lie once already (see the PORT comment). An\n # unreachable WebUI is reported as unreachable, and the hook stops rather than spending\n # its remaining login attempts to produce two more identical wrong diagnoses.\n api_ok=0\n while [ \"$waited\" -lt 120 ]; do\n code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' \"http://127.0.0.1:$PORT/\" 2>/dev/null)\n if [ -n \"$code\" ] && [ \"$code\" != \"000\" ]; then api_ok=1; break; fi\n waited=$((waited + 1)); sleep 1\n done\n if [ \"$api_ok\" != 1 ]; then\n log \"the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the\" >&2\n log \"credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was\" >&2\n log \"never submitted. Do not rotate the credential on the strength of this line.\" >&2\n exit 0\n fi\n body=$(curl -sS -m 10 \\\n --data-urlencode \"username=$QBT_WEBUI_USERNAME\" \\\n --data-urlencode \"password=$QBT_WEBUI_PASSWORD\" \\\n \"http://127.0.0.1:$PORT/api/v2/auth/login\" 2>/dev/null)\n if [ \"$body\" = \"Ok.\" ]; then\n log \"VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'\"\n exit 0\n fi\n log \"WebUI rejected the durable credential (login said '$body')\" >&2\n # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean\n # the running qbittorrent-nox is serving something it read from an EARLIER conf, so\n # rewriting the same bytes on the next attempt would change nothing. The remedy is to\n # make the process RE-READ the file, so restart the supervised service and let the next\n # attempt re-verify. Without this the retry loop could never change its own outcome.\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"restarting qbittorrent-nox so it re-reads $CONF, then re-verifying\" >&2\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\" >&2\n sleep 3\n fi\ndone\n\nlog \"GAVE UP after $attempt attempts: the WebUI does not accept the credential from git.\" >&2\n# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.\n# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the\n# app, and the arrs' own download-client health check surfaces a broken credential.\nexit 0\n"
expr:kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"} == 0 or absent(kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"})
for:5m
labels:
severity:critical
service:qbittorrent
annotations:
summary:qbittorrent is down
description:'qbittorrent has been unreachable for 5m (SLI:up).'
description:'qbittorrent:its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI:up, via kube_deployment_status_replicas_available).'
description:"plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time:10m
alertname:gitea-down
exp_alerts:[]
- eval_time:10m
alertname:jackett-down
exp_alerts:[]
- eval_time:10m
alertname:pihole-down
exp_alerts:[]
- eval_time:10m
alertname:qbittorrent-down
exp_alerts:[]
- eval_time:10m
alertname:vm-demo-down
exp_alerts:[]
# Before `for: 5m` has elapsed the rule must be PENDING, not firing — proof the 5m is real and
# a single scrape blip does not page anyone.
promql_expr_test:
- expr:'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"} == 0 or absent(kube_deployment_status_replicas_available{namespace="plex", deployment="plex"})'
description:"gitea: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time:10m
alertname:jackett-down
exp_alerts:
- exp_labels:
severity:critical
service:jackett
namespace:jackett
deployment:jackett
exp_annotations:
summary:"jackett is down"
description:"jackett: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time:10m
alertname:pihole-down
exp_alerts:
- exp_labels:
severity:critical
service:pihole
namespace:pihole
deployment:pihole
exp_annotations:
summary:"pihole is down"
description:"pihole: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time:10m
alertname:qbittorrent-down
exp_alerts:
- exp_labels:
severity:critical
service:qbittorrent
namespace:qbittorrent
deployment:qbittorrent
exp_annotations:
summary:"qbittorrent is down"
description:"qbittorrent: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
description:"plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): keycloak-allow-internet-egress
apiVersion:cilium.io/v2
kind:CiliumNetworkPolicy
metadata:
name:keycloak-allow-internet-egress
namespace:keycloak
labels:
app.kubernetes.io/name:keycloak
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name:keycloak
egress:
- toEntities:
- world
---
# HTTPRoute keycloak-8080 (R-TLS/R-DNS)
apiVersion:gateway.networking.k8s.io/v1
kind:HTTPRoute
metadata:
name:keycloak-8080
namespace:keycloak
labels:
app.kubernetes.io/name:keycloak
spec:
parentRefs:
- name:shared
namespace:gateway
sectionName:https
hostnames:
- keycloak.${BASE_DOMAIN}
rules:
- matches:
- path:
type:PathPrefix
value:/
backendRefs:
- name:keycloak
port:8080
---
# PrometheusRule SLI alerts (R-OBS-01)
apiVersion:monitoring.coreos.com/v1
kind:PrometheusRule
metadata:
name:keycloak-slis
namespace:keycloak
labels:
release:kube-prometheus-stack
app.kubernetes.io/name:keycloak
spec:
groups:
- name:keycloak.slis
rules:
- alert:keycloak-down
expr:kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"} == 0 or absent(kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"})
for:5m
labels:
severity:critical
service:keycloak
annotations:
summary:keycloak is down
description:'keycloak:its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI:up, via kube_statefulset_status_replicas_ready).'
- "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n# maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n# stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n i=$((i + 1))\n [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\nif [ -z \"${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)\"\n exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1; break\n fi\n d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n# HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n# single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n# form fed the value back through the wrapper's $*, which SPLIT it into extra\
\ positionals ->\n# occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n# (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n --clientid=nextcloud \\\n --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n --scope='openid email profile' \\\n --unique-uid=0 \\\n --mapping-uid=preferred_username \\\n --mapping-email=email \\\n --mapping-display-name=name \\\n --group-provisioning=1 \\\n --mapping-groups=groups\" >/dev/null 2>&1 || true\n if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n# NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n# login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n# built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n# start: occ group:adduser is a no-op for an existing member. A user is promotable only\n# AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n# rights on their next login + the following pod start (or run `occ group:adduser admin\n# <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n# passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n current_admins=$(parse_group admin)\n missing=0\n for uid in $(parse_group \"$ADMIN_SRC_GROUP\"); do\n if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n continue\n fi\n missing=1\n if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n fi\n done\n [ \"$missing\" = 0 ] && break\n p=$((p + 1)); sleep 2\ndone\n[ \"$p\" -ge 15 ] && nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\n"
- "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n# maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n# stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n i=$((i + 1))\n [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is\n# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS\n# TRUE \u2014 so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never\n# installing/enabling user_oidc nor registering the provider. SSO kept working only because\n# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with\n# NO OIDC at all. Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator\n# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion\n# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why\n# `--clientsecret-env` and the bare uses below always worked and only the guard broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[nextcloud-oidc]\
\ client secret unset; skipping OIDC (break-glass admin only)\"\n exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1; break\n fi\n d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n# HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n# single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n# form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n# occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n# (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n --clientid=nextcloud \\\n --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n --scope='openid email profile' \\\n --unique-uid=0 \\\n --mapping-uid=preferred_username \\\n --mapping-email=email \\\n --mapping-display-name=name \\\n --group-provisioning=1 \\\n --mapping-groups=groups\" >/dev/null 2>&1 || true\n if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n# NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n# login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n# built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n# start: occ group:adduser is a no-op for an existing member. A user is promotable only\n# AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n# rights on their next login + the following pod start (or run `occ group:adduser admin\n# <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n# passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n current_admins=$(parse_group admin)\n missing=0\n for uid in $(parse_group \"$ADMIN_SRC_GROUP\"\
); do\n if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n continue\n fi\n missing=1\n if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n fi\n done\n [ \"$missing\" = 0 ] && break\n p=$((p + 1)); sleep 2\ndone\nif [ \"$p\" -ge 15 ]; then\n nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\nfi\n# A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:\n# the final statement used to be `[ \"$p\" -ge 15 ] && nextcloud_oidc_fail ...`, so on the\n# SUCCESS path the test returned 1 and became the script's exit status. That was latent for\n# as long as the OIDC guard above was dead code and exited 0 first; fixing the guard\n# exposed it immediately. Every failure path here logs loudly and writes a marker file and\n# deliberately does NOT crash the pod \u2014 the break-glass local admin must stay usable.\nexit 0\n"
volumes:
- name:data
persistentVolumeClaim:
claimName:nextcloud-data
- name:legacy-archive
persistentVolumeClaim:
claimName:nextcloud-legacy-archive
---
# Service
apiVersion:v1
@@ -227,7 +280,7 @@ spec:
namespace:gateway
sectionName:https
hostnames:
- nextcloud.home.keithjackson.dev
- nextcloud.${BASE_DOMAIN}
rules:
- matches:
- path:
@@ -237,26 +290,6 @@ spec:
- name:nextcloud
port:80
---
# ServiceMonitor (R-OBS-01)
apiVersion:monitoring.coreos.com/v1
kind:ServiceMonitor
metadata:
name:nextcloud
namespace:nextcloud
labels:
release:kube-prometheus-stack
app.kubernetes.io/name:nextcloud
spec:
selector:
matchLabels:
app.kubernetes.io/name:nextcloud
namespaceSelector:
matchNames:
- nextcloud
endpoints:
- port:metrics
interval:30s
---
# PrometheusRule SLI alerts (R-OBS-01)
apiVersion:monitoring.coreos.com/v1
kind:PrometheusRule
@@ -271,11 +304,11 @@ spec:
- name:nextcloud.slis
rules:
- alert:nextcloud-down
expr:up{job="nextcloud"} == 0
expr:kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"} == 0 or absent(kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"})
for:5m
labels:
severity:critical
service:nextcloud
annotations:
summary:nextcloud is down
description:'nextcloud has been unreachable for 5m (SLI:up).'
description:'nextcloud:its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI:up, via kube_deployment_status_replicas_available).'
expr:kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"})
for:5m
labels:
severity:critical
service:radarr
annotations:
summary:radarr is down
description:'radarr has been unreachable for 5m (SLI:up).'
description:'radarr:its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI:up, via kube_deployment_status_replicas_available).'
expr:kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"} == 0 or absent(kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"})
for:5m
labels:
severity:critical
service:semaphore
annotations:
summary:semaphore is down
description:'semaphore has been unreachable for 5m (SLI:up).'
description:'semaphore:its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI:up, via kube_deployment_status_replicas_available).'
expr:kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"})
for:5m
labels:
severity:critical
service:sonarr
annotations:
summary:sonarr is down
description:'sonarr has been unreachable for 5m (SLI:up).'
description:'sonarr:its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI:up, via kube_deployment_status_replicas_available).'
description:"keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
description:"nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time:10m
alertname:radarr-down
exp_alerts:
- exp_labels:
severity:critical
service:radarr
namespace:radarr
deployment:radarr
exp_annotations:
summary:"radarr is down"
description:"radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time:10m
alertname:semaphore-down
exp_alerts:
- exp_labels:
severity:critical
service:semaphore
namespace:semaphore
deployment:semaphore
exp_annotations:
summary:"semaphore is down"
description:"semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time:10m
alertname:sonarr-down
exp_alerts:
- exp_labels:
severity:critical
service:sonarr
namespace:sonarr
deployment:sonarr
exp_annotations:
summary:"sonarr is down"
description:"sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
name:wave-1 all liveness series ABSENT (kube-state-metrics gone) - every rule fires
input_series:
# One unrelated series so the test has an input at all; no wave-1 liveness series exists.
- series:'up{job="kube-state-metrics"}'
values:'0+0x20'
alert_rule_test:
- eval_time:10m
alertname:keycloak-down
exp_alerts:
- exp_labels:
severity:critical
service:keycloak
namespace:keycloak
statefulset:keycloak
exp_annotations:
summary:"keycloak is down"
description:"keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
- eval_time:10m
alertname:nextcloud-down
exp_alerts:
- exp_labels:
severity:critical
service:nextcloud
namespace:nextcloud
deployment:nextcloud
exp_annotations:
summary:"nextcloud is down"
description:"nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time:10m
alertname:radarr-down
exp_alerts:
- exp_labels:
severity:critical
service:radarr
namespace:radarr
deployment:radarr
exp_annotations:
summary:"radarr is down"
description:"radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time:10m
alertname:semaphore-down
exp_alerts:
- exp_labels:
severity:critical
service:semaphore
namespace:semaphore
deployment:semaphore
exp_annotations:
summary:"semaphore is down"
description:"semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time:10m
alertname:sonarr-down
exp_alerts:
- exp_labels:
severity:critical
service:sonarr
namespace:sonarr
deployment:sonarr
exp_annotations:
summary:"sonarr is down"
description:"sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
expr:kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"})
for:5m
labels:
severity:critical
service:seerr
annotations:
summary:seerr is down
description:'seerr has been unreachable for 5m (SLI:up).'
description:'seerr:its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI:up, via kube_deployment_status_replicas_available).'
description:"seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
# Under `for: 5m` a 2m-old condition must NOT yet be a firing alert, but the expression must
# already be selecting — the distinction an inert rule cannot make.
promql_expr_test:
- expr:'kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"})'
name:wave-2 seerr liveness series ABSENT - seerr-down still fires
input_series:
- series:'up{job="kube-state-metrics"}'
values:'1+0x20'
alert_rule_test:
- eval_time:10m
alertname:seerr-down
exp_alerts:
- exp_labels:
severity:critical
service:seerr
namespace:seerr
deployment:seerr
exp_annotations:
summary:"seerr is down"
description:"seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. `flux install` → verify `sops-age` Secret → `kubectl apply -k clusters/dev`. |
| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. render+apply the Flux controllers (`components/`) → verify `sops-age` Secret → `kubectl apply -k clusters/dev`. |
| `components/kustomization.yaml` | **Customization of the Flux CONTROLLERS themselves.** Overlays `flux install --export` output; currently enables `StrictPostBuildSubstitutions` on kustomize-controller. See "Customizing the controllers" below. |
| `patches/platform-sops-decryption.yaml` | Strategic-merge patch enabling `decryption.provider: sops` + `secretRef: sops-age` on the `platform` Kustomization (realises the base TODO). |
| `patches/apps-sops-decryption.yaml` | Same, for the app-wave Kustomizations. There is **no single `apps` Kustomization** any more — the apps tier is now `apps-wave-0/1/2` + `apps-vms`, and SOPS decryption is layered on **per wave** (each wave Kustomization is patched by name in `clusters/dev`). |
| `patches/flux-system-source-url.yaml` | Overrides the `flux-system` GitRepository `url` placeholder with the real bootstrap source (template — fill at bootstrap). |
## Customizing the controllers (`components/`) — and how it reaches a RUNNING cluster
Because this repo does not use `flux bootstrap`, the Flux controller Deployments are **not in git
and not reconciled by Flux**. Verified on the live dev cluster (2026-09-01):
```
$ kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.metadata.labels}'
with no cluster, or an unwired seam / missing tool). A `3` is **never** counted as green — that is how
the gate stays honest in Phase 0 (no fake green).
### The gate's OWN verdict (changed 2026-09-13 — read this if you consume the exit code)
**`MERGE GATE: PASS` + exit 0 means every stage that was supposed to run RAN and was GREEN.** It
used to mean less than that: a FAILED stage set the block flag but a SKIPPED one only printed a
warning, so the gate printed `l0: skipped (unwired seam / missing tool) — not green` and then
`MERGE GATE: PASS`, exit 0. That happened for real twice — a missing `promtool` (the alert-rule
unit tests) and a missing pinned `kyverno` CLI — and exit 0 is what automation reads as
merge-approved. Fail-open in the one component whose whole job is to fail closed.
| verdict | exit | meaning |
|---|---|---|
| `MERGE GATE: PASS` | `0` | every stage that was supposed to run ran and was green; the R-TEST-03 subset was checked. Any capability the environment **declared** out of scope is **named in the headline** (see below), so a PASS never implies more than it verified. |
| `MERGE GATE: BLOCKED` | `1` | a stage FAILED, or a stage did not run **where it must**, or the decision input is unusable. |
| `MERGE GATE: UNDETERMINED` | `3` | nothing failed, but a stage did not run, so greenness was **not established**. Never a pass; non-zero so nothing can read it as approval. `3` is this repo's existing "not run" code, now spoken by the gate as a whole and not only by its stages. |
| `MERGE GATE: NOT APPLICABLE` | `0` | there is no branch under gate at all (`HEAD` is on base — a push-to-`main` run). Stages ran; no subset was checked, and the headline says so. |
Precedence: BLOCKED > UNDETERMINED > NOT APPLICABLE > PASS.
**`make` masks the difference.** GNU make reports any recipe failure as its own exit `2`, so
`make -C ci gate` collapses `1` and `3` into `2` (the headline still tells a human which it was). A
caller that must distinguish them invokes `./ci/gate.sh` directly.
### Which skips block, and why — a skip is not one thing
| stage | a skip there means | verdict |
|---|---|---|
| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is installed for us by `.gitea/workflows/scripts/install-tools.sh`, so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
| `spec-immutability` (R-TEST-06) | the sibling spec repo is not checked out, so the spec **could not be verified** | **UNDETERMINED** — unverified is unverified. Deliberately *not* upgraded to BLOCKED by strict skips: the live runner does not check the spec repo out at all, so blocking would leave the only enforced status check permanently red for a reason no PR could clear. It is therefore the **one** skip whose non-answer a caller may knowingly tolerate, and `.gitea/workflows/gate.yml` does so in one visible line. `REQUIRE_SPEC_VERIFICATION=true` makes it a hard block once that checkout is wired. Locally: check the spec repo out beside this one, or set `SPEC_REPO` (`dev.sh` mounts a sibling automatically). |
| `conformance` (L2/L3) | `CLUSTER_AVAILABLE!=true` — the environment **declared** the capability absent (documented Phase-0 behaviour) | **does not deny a PASS**: a declared-inapplicable stage was never *supposed* to run. It is NAMED in the PASS headline and step summary, the touched requirements it would have covered are reported UNVERIFIED, and `REQUIRE_CLUSTER_TESTS=true` (Phase 1+) turns that into a block. A skip **despite**`CLUSTER_AVAILABLE=true` is a defect and BLOCKS. |
That asymmetry is the whole rule: **declared-not-applicable may pass; could-not-run may not.**
`GATE_STRICT_SKIPS=true|false` decides strictness and is **declared, never sniffed** — `gate.yml`
sets it on the gate step, because the workflow is the thing that knows it is CI. Unset, it falls
back to `GITHUB_ACTIONS` / `CI` (set by the runner itself). A wrong answer in either direction
cannot fail open: both branches are non-zero and neither is a PASS. Set it locally to rehearse CI
strictness.
Locked down by `ci/tests/gate-decision.test.sh` (47 assertions over the decision table: stubbed
stage outcomes against the real `gate.sh`; 32 of them fail against the pre-2026-09-13 gate).
### The touched-requirement subset can also be *undeterminable* (exit 3)
`touched-requirements.sh` is the input to the R-TEST-03 decision. It resolves a diff base in this
volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)
namespace "chainsaw-cunning-hen-dr-src" deleted
source namespace destroyed
original PersistentVolume pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745 is GONE -- no in-cluster copy of the data remains
(longhorn Volume CR pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745 not present -- either deleted or not readable by this identity)
restore.velero.io/chainsaw-cunning-hen-dr-restore created
poll 1: restore phase=InProgress
poll 2: restore phase=InProgress
poll 3: restore phase=InProgress
poll 4: restore phase=Completed
restore chainsaw-cunning-hen-dr-restore Completed into chainsaw-cunning-hen-dr-dst
pod/dr-workload condition met
restored PVC bound to PersistentVolume: pvc-63c47973-9c7f-4423-a27e-caa008026e49
restored onto a NEW volume (pvc-63c47973-9c7f-4423-a27e-caa008026e49 != pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745) -- the data came from outside the cluster
Loaded 100 of 380 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
jakman85andClaude Opus 5
gitea-admin