home-network-k3s-impl

0 Fork 0

Author	SHA1	Message	Date
jakman85andClaude Opus 5	a79693d3bd	feat(monitoring): scrape the Flux controllers — five gotk_* alerts were inert gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details kube-prometheus-stack does not scrape flux-system: the controllers are bootstrapped, expose no Service, and no chart value covers them. So every gotk_* alert rule in platform/monitoring/alerts.yaml could never fire — verified on prod, count({__name__=~"gotk.*"}) returned nothing. A repo whose entire delivery model is Flux had no alerting on reconciliation failing, a Kustomization erroring, or a resource left suspended. Proven live before landing: with this PodMonitor applied, gotk_suspend_status returns 39 series and gotk_reconcile_condition 117, and FluxResourceSuspended actually fired on a deliberate test suspension — evidence the rule is live, not merely present. Extracted from the wider inert-alert audit on fix/scrape-targets-for-inert-alerts, which stalled unfinished. This is the self-contained, verified half; Longhorn and cert-manager metrics are still ABSENT and their alerts remain inert. It also adopts the object into git: it existed on prod only as kubectl-client-side-apply, so a rebuild would have lost it. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 08:53:21 -05:00
jakman85andClaude Opus 5	16b7fcd677	fix(cnpg): the grant reconciler died on every Flux-managed run gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details The SQL asserted convergence in a dollar-quoted PL/pgSQL DO block. That cannot survive this delivery path: the ConfigMap is rendered through Flux's postBuild substitution, and envsubst collapses a doubled dollar into one, so Postgres received a half-quoted DO and failed with a syntax error. Three scheduled runs had left nine Error pods in nextcloud-db. It passed verification because it was hand-applied with kubectl, which runs no envsubst — the defect only existed once Flux owned the object. The GRANT on the first line still succeeded, so the privilege held and the blind spot stayed closed; what was broken was the assertion meant to prove it, which is the part that makes a silent regression visible. Rewritten in plain SQL with no dollar sign anywhere in the file, comments included. Escaping by doubling again would also work and would be the next person's trap. has_database_privilege()::int is 1 or 0, so the division raises division_by_zero and ON_ERROR_STOP=1 fails the Job; the divisor is not constant so it cannot be folded away at plan time. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 00:19:49 -05:00
jakman85	5fa893095b	merge: the placeholder guard test was hiding a RED, not a pass gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details It had been permanently SKIPPED because it expected bootstrap/standup.sh, which never landed on main — planned, never merged, still living only on the unmerged prod-standup branches along with the 2026-09-05 incident it came from. Dropping the real script in and running the old check produced FAIL 2/34: it would have false-blocked the very merge that landed its own input. Now split into the half that needs no guard (no overlay ships a placeholder-shaped effective cluster-vars value; an empty value must be STATED by the overlay, not inherited in silence; a loud base sentinel must be explicitly overridden) and the half that self-activates. The guard is discovered by CONTENT — any tracked *.sh defining <name>_is_placeholder() — so landing one anywhere activates the second half with no edit here, and two definitions is a failure. The check never exits 3 again. Unscoped L0 goes 13 PASS / 1 SKIP -> 14 PASS / 0 SKIP.	2026-09-13 00:14:17 -05:00
jakman85	e49b86eb41	merge: 35 runbooks, served, with a guard that they resolve gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details All 46 runbook_url annotations pointed at https://<apex>/runbooks/<name> and every one returned 404 — there was no runbooks/ directory and no route. Every alert that reached a human handed them a dead link at the moment they needed it. Runbooks are now served from their own non-gating tier at the apex prefix, PUBLIC on purpose: gating them behind Keycloak would make keycloak-down unreadable exactly when Keycloak is down, which is a loop with no exit. The new L0 guard checks both directions — a runbook_url with no file, and a file not wired into the generator — and it caught a live one on contact with main: CNPGPrimariesCoLocated had shipped hours earlier pointing at a runbook that did not exist.	2026-09-13 00:11:03 -05:00
jakman85	d616fd4025	merge: promtool is in the toolchain, so the alert unit tests actually run 324 lines of mutation-checked promtool tests had landed with nothing to execute them: promtool was not in the pinned toolchain, so the coverage was present and enforced by nothing — the failure mode this repo keeps hitting. promtool 3.14.0 is now pinned in both versions.env files (matching prod's Prometheus, verified against the running image) and baked into the container. The two competing harnesses are consolidated into one, and the two competing wirings into one: the bespoke ci/prom-rules.sh stage is deleted in favour of a scope-declaring conformance test, because a hand-written stage running one hardcoded script is exactly the pattern `b9f8807` just removed. Missing promtool is exit 3 — UNWIRED, never green. 51 of 58 alert rules still have no unit test; that gap is a shrink-only ratchet that fails if it grows, rather than a blanket FAIL that would get disabled or a report nobody reads.	2026-09-13 00:11:03 -05:00
jakman85andClaude Opus 5	87c3d83dff	fix(conformance): placeholder-guard-coupling actually RUNS — it was skipping forever `conformance/tests/l0-static/placeholder-guard-coupling` (R-ENV-02, R-STATE-03) exited 3 SKIPPED on every gate run since it landed, because its check keyed its whole existence on `bootstrap/standup.sh`. Measured, not assumed: that file has NEVER been on `main` — there is no deletion commit for it anywhere; it landed only on unmerged prod-standup branches (`work/prod-standup-campaign`, `prod/`, `feat/prod-hardening`). It was planned and never merged, not removed. SKIPPED is not green, and a permanently-skipped incident-derived test is the quiet version of the failure it was written to prevent. The skip was also hiding a RED. Run against the real `standup.sh` from `work/prod-standup-campaign`, the old check FAILED 2 of 34 assertions — false positives from treating the empty string as placeholder-shaped for knobs cluster-vars authors empty on purpose (`BACKUP_CREDENTIALS_SUFFIX`, `VELERO_BACKUP_PREFIX`, both `""` on dev and both documented there as load-bearing). It would have false-blocked the merge landing its own input. NOT deleted — the hazard is not gone. Split into two halves, one of which needs no guard: HALF A (always runs): no environment overlay ships a placeholder-shaped EFFECTIVE cluster-vars value; an empty effective value is STATED by the overlay rather than inherited in silence; a LOUD base sentinel is explicitly overridden by every overlay. This is the 2026-09-05 outcome asserted at the source instead of at the guard — if prod's S3_BACKUP_ENDPOINT is never a placeholder, velero cannot be pointed at a hostname that does not exist. Input surface is `clusters/`, all of it on `main`. HALF B (self-activating): the real predicate extracted and executed against those same values, the consumer trace, and the mid-string anchoring property. The check NEVER exits 3 again, and the guard is DISCOVERED by content rather than hardcoded: any tracked `.sh` defining `<name>_is_placeholder()` is found, so a guard landing at `bootstrap/standup.sh` — or moved into a sourceable library, the remedy the check already recommends — activates HALF B with no edit to the test. Two files defining it is a FAIL. When no guard exists the gap is printed loudly every run instead of swallowed as a skip. LOUD vs EMPTY is the correction that makes it honest in both modes: a loud placeholder is always a defect in an overlay, while `""` is held to the explicitness rule instead — and still counts as a placeholder for a knob a guard covers, because a `""` case-arm is the guard author's declaration that empty is not a usable target. Proof, all reverted: 84 assertions and exit 0 on `main`; prod S3_BACKUP_ENDPOINT set to the literal `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` FAILS (A1); deleting dev's explicit VELERO_BACKUP_PREFIX op so it inherits `""` FAILS (A2); a LOUD base BULK_NFS_SERVER with prod's override deleted FAILS 2/86 (A1+A3); and with the real `standup.sh` present, reverting `s3_is_placeholder` to the `TBD-` PREFIX glob FAILS 12/110 (B3 anchoring). With the pristine `standup.sh` the check PASSES at 110 assertions where the old one failed 2 of 34. L0 layer: 14 checks, 14 PASS, 0 SKIP, 0 FAIL (was 13 PASS / 1 SKIP). Coverage unchanged — R-ENV-02 and R-STATE-03 both still list this test, 49/51 with the same 2 gaps. Also: `ci/touched-requirements.sh` — `bootstrap/` -> R-STATE-03 R-GITOPS-01 is KEPT, its stale comment corrected. It claimed standup.sh "carries the placeholder guards", describing a file that is not here; R-STATE-03 there is ANTICIPATORY. Deleting the entry would leave the change that lands a placeholder guard selected by nothing. `ci/tests/touched-requirements.test.sh` — new path-map section (12 assertions, was 8) locking `bootstrap/` and `clusters/`, so removing either entry is a test failure rather than a silent loss of selection. All three ci/tests pass (12 + 31 + 15). * `docs/BULK-NFS-STORAGE.md` §7 — its "nothing in the gate currently refuses a placeholder-shaped overlay value" is no longer true; corrected. * Scope re-audited: still deliberately requirement-scoped, not whole-repo. Every path that can break it attributes a requirement it declares. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 00:10:37 -05:00
jakman85andClaude Opus 5	bce8778096	docs(runbooks): forward-auth IS enforced — correct the posture note The serving route's comment claimed that because no Gateway ext-authz attachment exists, every forward-auth route is 'in practice reachable unauthenticated today'. That is wrong, and wrong in a dangerous direction for a document an operator reads during an incident. Enforcement here is PER-POD, not gateway-level: each of the five services declaring auth: forward-auth (pihole, jackett, qbittorrent, radarr, sonarr) publishes both its app port and 4180 on its Service, and its HTTPRoute targets :4180 — the oauth2-proxy sidecar. Verified on prod. The app port remains reachable in-cluster on purpose, which is how sonarr/radarr drive qBittorrent's API, and Cilium default-deny governs that path. The public-by-design decision for this route is unchanged and still correct; what changes is that it is genuinely the exception rather than one open route among many. Headlamp's exclusion is likewise deliberate (native OIDC with per-user id_token forwarding and no ServiceAccount cluster rights). Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 00:10:24 -05:00
jakman85andClaude Opus 5	78d0f1b0f6	feat(ci): promtool in the pinned toolchain, one alert-rule harness, and the gaps declared The CNPG alert unit tests (324 lines, mutation-checked) were run by NOTHING. promtool was not in the baked toolchain, no gate stage evaluated any .promtool-test.yaml, and the only runner `curl \| tar -xz`'d a promtool release from GitHub at TEST-RUN time if one was missing. Hours later a second, independent harness landed for the Velero rules (platform/monitoring/tests/) with its own fallback — a `podman run` of quay.io/prometheus/prometheus. So the repo had two harnesses, two acquisition paths, and coverage that looked present and was enforced nowhere: the exact failure mode that had just produced a permanently-firing CNPG alert and a structurally inert Velero alert. PROMTOOL IS NOW PINNED LIKE EVERY OTHER TOOL. PROMTOOL_VERSION=3.14.0 in infra/devcontainer/versions.env and ci/versions.env (the two stay in lockstep, per the repo's rule), baked by the Containerfile from the prometheus release tarball (promtool ships inside it), recorded in /etc/dev-toolchain-versions, passed by dev.sh, and installed for the l0 stage by .gitea/workflows/scripts/install-tools.sh so the CI runner is not permanently SKIPPED. The pin is not free-floating: promtool IS the rule engine, so it tracks the Prometheus prod actually runs — verified against the live cluster, not assumed: kubectl --context k3s-prod get prometheus -n monitoring -o jsonpath='{.items[].spec.version}' -> v3.14.0-distroless (image quay.io/prometheus/prometheus:v3.14.0-distroless) ONE HARNESS. platform/cnpg/run-alert-tests.sh is folded into platform/monitoring/tests/run.sh, which now covers every PrometheusRule in the repo that has tests, via two conventions it discovers automatically: `<manifest>.promtool-test.yaml` beside its manifest (cnpg), and `<component>/tests/.test.yaml` over that component's manifests (monitoring). It writes the flattened rules under whatever name each test file's own `rule_files:` declares, so no test file needs to know the harness exists, and it `promtool check rules` first — a rules file that does not parse would make every unit test pass vacuously by finding no rules at all. The cnpg script's reasoning is preserved: why a CR must be flattened (`.spec.groups`, never re-typed), and why generated files go to a temp dir. Its `${BASE_DOMAIN}` substitution is dropped in favour of extract_rules.py's verbatim placeholders, with the trade-off recorded there — the hazard it guarded against is addressed by the temp dir, and verbatim leaves one transformation fewer between the shipped rule and the rule under test. NO ENGINE IS ACQUIRED AT RUN TIME any more — not the GitHub release, not the container image. Not because a pinned download is unusual here (the Containerfile downloads every tool it bakes) but because those two decided the version, the network and the machine SILENTLY, at test time, on whatever host ran the tests; and each quietly turned "the toolchain has no promtool" into a green run, which is how the toolchain stayed without one. Missing promtool is exit 3 (UNWIRED, never green); a PROMTOOL= that does not resolve is exit 2. ONE WIRING: conformance/tests/l0-static/alert-rules-unit-tested, NOT a gate stage. The `prom-rules` stage (ci/prom-rules.sh + its Makefile target + ci/gate.sh entry) is deleted: it is the bespoke-per-check pattern `b9f8807` had just removed from ci/l0.sh, and the declarative replacement gives what a hand-written stage cannot — selection by `conformance.home.k3s/scope: "whole-repo"` (asserted: the test is selected even under a requirement filter matching nothing), requirement-attributed JUnit/coverage/ledger evidence, and run_l1_offline.py's rule that a SKIP of a scope-declaring check is the runner's rc 3. The scope declaration is load-bearing here: touched-requirements.sh keys apps/ to nothing and platform/monitoring/* to R-OBS-01/02, never R-OBS-04, so a new untested PrometheusRule would otherwise be gated by nothing. The check asserts the pins and their lockstep, that the image bakes and records promtool, that the promtool IN USE reports exactly the pin (a FAILURE, not a skip: the wrong engine still prints SUCCESS), that exactly ONE harness exists and none fetches an engine, that every rule-unit test file passes, and — on every run — that the harness CAN fail, by putting a synthetic rule through it twice with one expectation mutated. MISSING TESTS ARE A RATCHET, deliberately, and the reasoning is in the check's header. 51 of 58 alert rules have no unit test. Failing on all of them makes the gate red until ~51 tests exist, so it gets disabled and protects nothing; merely reporting is exactly how the CNPG tests came to be enforced by nothing. So the gap is DECLARED in the check as a shrink-only register, printed with counts every run (13 alert files nothing tests at all; 35 named alerts inside covered files that nothing asserts on), and it cannot grow: a new PrometheusRule, or a new alert in an already-covered file, FAILS without a test, and a stale entry FAILS too. A missing test is never an exit-3 skip — SKIPPED is reserved for the one genuine environment gap, promtool being absent. PROVEN, in the rebuilt image (promtool 3.14.0): * both suites pass through the single harness: cnpg 6 rules, monitoring 36 rules, SUCCESS. * the check passes: 95 assertions, 7/58 alert rules unit-tested, 13 declared gap files. * NOT VACUOUS, twice over. Shortening CNPGPrimariesCoLocated's `for: 30m` to 5m turns it red (pending expected at 25m/29m, got firing), reverted. Adding an untested PrometheusRule (platform/kubevirt/probe-alerts.yaml) turns it red on the ratchet, removed. The harness's own mutation self-test runs on every invocation. * the scope mechanism selects it: `run.sh --layer l0 --requirements r-nosuchreq-99 --include-scope whole-repo --print-selection` lists alert-rules-unit-tested. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 00:02:27 -05:00
jakman85	55b90849d1	merge: the arrs had no download client at all, and now agree on paths Investigating the path disagreement found the larger defect: sonarr and radarr each returned [] from /api/v3/downloadclient — Completed Download Handling had never been wired up, not merely broken. Both now have qBittorrent configured plus a remote path mapping /downloads/ -> /media/downloads/, proven with a real completed torrent and a negative control: same torrent, same client, mapping absent gave an unresolvable /downloads/... and mapping present resolved the real file at inode 36144. This state lives in each arr's database, NOT in git — a rebuild from git does not restore it. Recorded in DECISIONS-LOG with the recommendation to close it properly via path agreement instead.	2026-09-13 00:01:55 -05:00
jakman85andClaude Opus 5	b7341b329b	fix(runbooks): declare whole-repo scope, drop the bespoke l0 stage, +cnpg-primary-spread Three corrections after rebasing onto main. 1. THE GUARD CAUGHT A REAL ONE ON ITS FIRST CONTACT WITH main. main had gained `CNPGPrimariesCoLocated` (platform/cnpg/alerts.yaml) pointing at /runbooks/cnpg-primary-spread, with no runbook. The check failed the rebase exactly as designed — 46 annotations, 35 names, 34 present — naming both the missing file AND the missing configMapGenerator entry. runbooks/cnpg-primary-spread.md now exists: what co-located primaries actually cost here (blast radius, not an outage, because instances: 3 + required anti-affinity), why the drift is uncorrectable-by-declaration (CNPG has no primary-placement field), the readiness gate that keeps the rule quiet during a real node outage, and the ~9s-per-database switchover rebalance, one cluster at a time. The 2026-09-07 outage in which all three primaries sat on one node is the reason the alert exists at all. 2. WHOLE-REPO SCOPE, NOT A HAND-WRITTEN STAGE. main's refactor replaced ci/l0.sh's per-check "UNCONDITIONAL" stages with a declaration: `conformance.home.k3s/scope: "whole-repo"`, unioned into every gate run by Stage 3b's `--include-scope`. My +38-line Stage 3f is removed (ci/l0.sh is now byte-identical to main) and the annotation is declared instead. Assertion (7) of the check was rewritten to match: it no longer asserts its own path appears in ci/l0.sh — it asserts the test DECLARES the scope and that ci/l0.sh still passes --include-scope, since neither half alone makes the check load-bearing. That matters here because ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02", not R-OBS-04, and maps platform/kubevirt/* and runbooks/* to nothing at all. 3. A FACTUAL CORRECTION IN ALL 35 RUNBOOKS. Every page claimed `dev.sh` defaults its current-context to k3s-dev. main's `9b4744d fix(devcontainer): dev.sh defaults to prod` made that false, and a runbook that misstates which cluster a command talks to is worse than one that says nothing. The pages now say what is durably true: the default has already changed once, so naming `--context k3s-prod` is what makes a command mean the same thing next month, and the per-invocation banner is there to be read. The check's own failure message was corrected the same way. Re-verified after all three: check green at 374 assertions (46 annotations, 35/35 runbooks present, 35/35 projected), re-applied to prod, and re-probed in-cluster through the Gateway — node-down, keycloak-down, keycloak-client-secrets, cnpg-archiving, longhorn-faulted, vpn-down, node-io and cnpg-primary-spread all HTTP 200 with text/plain; charset=utf-8 and 8.9-12.1 KB of real body; /runbooks/ index 200; an unknown name 404s with a pointer rather than a blank page; and homepage still answers the apex `/`, confirming Gateway API longest-path precedence. Full merge gate: PASS (spec-immutability, lint, l0, render-golden, l1 all green; prom-rules SKIPPED for want of promtool in the image; L2/L3 SKIPPED, no cluster). Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 00:00:24 -05:00
jakman85andClaude Opus 5	2f38b8156b	fix(arr): wire qBittorrent as a download client + Remote Path Mappings qBittorrent reports finished releases at /downloads/<name>; sonarr and radarr see the same inodes at /media/downloads/<name>, so Completed Download Handling could never resolve what the client handed it. Configured, on prod, via each app's own HTTP API: * download client qBittorrent -> qbittorrent.qbittorrent.svc.cluster.local:8080 (it did not exist in EITHER arr: both /api/v3/downloadclient returned []) * Remote Path Mapping /downloads/ -> /media/downloads/ for that host Both built from each app's /downloadclient/schema, matched on identity before writing, and re-running creates nothing. downloadclient/testall reports isValid=true in both. Proven with a negative control: Sonarr's queue showed outputPath=/downloads/... with the mapping deleted and /media/downloads/... with it present, for the same completed torrent. This state lives in sonarr.db / radarr.db on the config PVCs, NOT in git — a rebuild from git alone does not restore it. Recorded as a wart, with path agreement (repointing SavePath to /media/downloads) recommended as the durable fix and put to the operator rather than taken unilaterally. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 23:57:56 -05:00
jakman85andClaude Opus 5	44983095e1	feat(runbooks): make the 42 alert runbook_url annotations resolve Every platform-health alert rule carries `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations over 34 distinct names, across platform/monitoring/alerts.yaml, platform/monitoring/pvc-capacity-alerts.yaml, platform/cnpg/alerts.yaml and platform/kubevirt/alerts.yaml. There was no runbooks/ directory and no HTTPRoute serving that path: an in-cluster probe of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404. Every alert that reached a human handed them a dead link at the exact moment the documentation is read, and the annotation made it look solved — the same green-surface-over-absent-mechanism class as the import Job that reported success while storing placeholder secrets. No alerts.yaml is touched and no runbook_url value changes. The existing URL scheme is made to resolve instead. WHAT LANDS - runbooks/<34>.md — one per name, each grounded in this cluster's real topology and, where one exists, its real incident history: the 2026-09-07 k3s-prod-1 etcd I/O starvation (measured PSI 11.36 vs 0.75/1.29, the exact journal lines, the four mechanisms proven inert), the Longhorn reboot tax and the freeze/thaw of concurrent-replica-rebuild-per-node-limit, Cilium's once-resolved egress interface cache (Ifindex 3 against a live 1780) and the absence of OSS failover, the CNPG 38-hour ContinuousArchiving=False with its measured 6.8 MiB/h disk-full clock and the 2026-09-12 aggregation-scope fix, the 2026-09-06 Keycloak lockout in which all 12 clients held literal placeholder secrets behind an entirely green cluster, the Velero R2 x-amz-tagging NotImplemented incompatibility and its v1.14.3-rc.2 pin, and swap re-enabling itself across reboots via systemd-gpt-auto-generator. Each page states what the alert means HERE, the first diagnostic commands, likely causes ordered by probability with a remediation each, and when to escalate or stop. - Every page uses FULLY-QUALIFIED resource names and says why: three CRDs register the kind `Backup` here, the bare short name resolves to Longhorn's (zero objects), so `kubectl get backup -A` prints "No resources found" on a cluster holding 6 Velero and 27 CNPG backups. Acting on that false negative is how a CLI quirk becomes data loss. - Every page states that kubectl only works via `infra/devcontainer/dev.sh run bash -c '...'` with an explicit `--context k3s-prod`, because dev.sh defaults its current-context to k3s-dev. - Pages also record which alert groups are INERT today (measured 2026-09-13: no Longhorn, cert-manager, Flux-controller or Hubble scrape job exists, so 13 rules evaluate against nothing) so nobody reads their silence as health. SERVING — runbooks/serving/ plus clusters/base/runbooks.yaml A two-replica nginx-unprivileged serving a ConfigMap generated from runbooks/*.md, at the apex `/runbooks` prefix, in its own non-gating post-platform Flux tier (dependsOn platform-config, wait:false) — the platform/homepage pattern, file for file. - The service catalog cannot express this route: ingressRoutes[].host is pattern-bound to a single DNS label plus the apex, so the bare apex is unrepresentable, and there is no path field. - Rejected: redirecting to the Gitea web UI. A runbook is read when the platform is broken, and Gitea is an app-tier workload whose own dependencies several of these runbooks document; Gateway API's RequestRedirect also has no "append .md" modifier, so it would need 34 hand-maintained ReplaceFullPath rules. - configMapGenerator maps each file to an EXTENSIONLESS key (node-down=node-down.md), so /runbooks/node-down is a plain static lookup with no rewrite rule. Served as text/plain: the namespace has DNS-only egress, so a CDN markdown renderer could not load anyway. - The tier declares NO postBuild.substituteFrom, deliberately: envsubst would run over the prose and silently empty every ${...} shell example (it fails open — the 2026-09-01 emptied Cilium k8sServiceHost). The HTTPRoute omits `hostnames:` and inherits the https-apex listener's, so nothing needs substituting. AUTH — PUBLIC, deliberately auth-mode=none. Gating these pages behind oauth2-proxy/Keycloak would make runbooks/keycloak-down unreadable when Keycloak is down and runbooks/keycloak-client-secrets unreadable during the exact 2026-09-06 failure mode. An operator who cannot log in is this route's primary reader. Content is operational procedure only, no secrets. The rationale is recorded in an annotation on the HTTPRoute so the route is left OUT of the Gateway ext-authz attachment when forward-auth enforcement lands. PROVEN — in-cluster probe through the Gateway on prod /runbooks/node-down HTTP 200 8613 bytes text/plain; charset=utf-8 /runbooks/keycloak-down HTTP 200 9378 bytes text/plain; charset=utf-8 /runbooks/cnpg-archiving HTTP 200 11729 bytes text/plain; charset=utf-8 /runbooks/longhorn-faulted HTTP 200 9135 bytes text/plain; charset=utf-8 /runbooks/vpn-down HTTP 200 11092 bytes text/plain; charset=utf-8 /runbooks/backup-failure HTTP 200 9604 bytes text/plain; charset=utf-8 HTTPRoute Accepted by io.cilium/gateway-controller on sectionName https-apex; the `/runbooks` prefix wins over homepage's apex `/` by Gateway API longest-path precedence. GUARD — conformance/tests/l0-static/alert-runbooks-resolvable (R-OBS-04) 363 offline assertions. It checks BOTH halves, because the regression is silent in both directions: an alert with a new runbook_url and no runbook links to a 404, AND a runbook committed without its configMapGenerator entry is in git, reviewed, merged, and still 404s in the cluster. Also asserts a quality floor so a stub cannot make the link look fixed, and that route/listener/nginx prefix agree with the annotations. Proven to fail, then reverted: a bogus runbook_url (2 FAIL), a runbook removed from the generator (1 FAIL), and a 31-byte stub (7 FAIL). Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 23:49:58 -05:00
jakman85	4865734c5f	merge: alert when a backup schedule has NEVER run, and when one is paused gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details VeleroNoRecentBackup could not see the failure that actually happened: velero-daily sat paused having never produced a successful backup, so velero_backup_last_successful_timestamp{schedule=...} did not exist and time() - <absent series> yields no series. Prod had no volume backups and nothing alerted, further masked by a resources-only schedule succeeding daily. No velero_* metric describes a Schedule OBJECT, proven by scraping a throwaway paused Schedule and finding zero lines, so this exposes the CRs via kube-state-metrics customresource_state and keys on defaultVolumesToFsBackup — the property, not a name list. Adds the repo's first prometheus rule-unit-test harness, mutation-checked both ways.	2026-09-12 23:49:26 -05:00
jakman85	b9f8807aa6	merge: whole-repo L0 invariants are DECLARED, not hand-wired per test The gate had been accreting one bespoke UNCONDITIONAL stage per whole-repo L0 test (3c/3d/3e), each repeating the same ~12-line rationale and each one something the next author had to remember to write. Forget it and the test runs only when some unrelated requirement happens to be touched. Tests now declare conformance.home.k3s/scope: whole-repo and run.sh unions them in via --include-scope; the three stages are deleted. 10 of 13 tests are whole-repo, 3 stay path-scoped with the path map widened instead (infra/* -> R-ENV-01, ci/* -> R-TEST-06). A new meta-test makes the bespoke pattern fail the gate if it creeps back, and a change under conformance/tests/<test>/ now attributes that test's own requirements, so weakening a test is gated by what it certifies. rc-3 (SKIPPED is UNWIRED, not green) is preserved and demonstrated; the incident narratives moved into the tests they describe rather than being deleted. Gate cost for a scoped change rises 20.7s -> 41.8s, a ceiling any empty-touched-set change already paid.	2026-09-12 23:49:26 -05:00
jakman85andClaude Opus 5	7057bbb711	fix(monitoring): alert when a Velero schedule has NEVER backed up, or is paused VeleroNoRecentBackup could not fire for the failure mode that actually happened. `(time() - velero_backup_last_successful_timestamp{schedule!=""}) > 243600` is the right SHAPE — per-schedule, un-aggregated — but Velero only creates that series once a backup for the schedule SUCCEEDS, and `time() - <absent series>` yields no series. So it detects "backed up once, then stopped" and is structurally INERT on "never backed up". Prod ran with `velero-daily` (the schedule carrying VOLUME DATA) paused and never-successful: no volume backups at all, nothing alerted, and it was masked because `velero-daily-resources-only` completed cleanly every 03:00, so every other backup signal read green while only Kubernetes objects were protected. A PromQL-only fix is impossible, verified rather than assumed: velero v1.18.1 exports 35 velero_ metric names and every one carrying a `schedule` label is created by a backup ATTEMPT, not by the Schedule object. A throwaway `paused: true` Schedule on prod produced ZERO velero_* series, and nothing exposes `spec.paused`. The fact is not in the TSDB, so the Schedule CRs are exported via kube-state-metrics' custom-resource-state collector — the path this repo already uses for kube_node_labels — as kube_velero_schedule_{created,paused,protects_volume_data,info}. Proven by running KSM v2.20.0 with this exact config against prod's API: the CRD list/watch rule the chart adds is MANDATORY, not cosmetic — without it KSM emits the CRS metrics as an empty set, silently. Three rules, all keyed on PROPERTIES rather than schedule names: * VeleroScheduleNeverBackedUp (critical) — a Schedule older than 36h with no successful backup ever. The 36h grace comes from creationTimestamp so a newly added schedule is not paged before its first run is even due. * VeleroSchedulePaused (warning) — the direct root-cause signal; a paused backup schedule must never be silently indefinite. * VeleroNoVolumeDataBackup (critical) — THE ANTI-MASKING RULE. Fires unless some schedule with spec.template.defaultVolumesToFsBackup=true succeeded in 36h, so a healthy resources-only schedule can never make the platform look protected. Reading the property that DEFINES volume protection (not a name list) matches platform/velero/volume-policies.yaml and survives renames. `or vector(0)` is load-bearing: `count(<empty>) == 0` returns no series and would be inert — absence of the exporter is deliberately the alarm. Adds the repo's first Prometheus rule-unit-test harness (platform/monitoring/tests/, wired as the `prom-rules` gate stage). It flattens the COMMITTED PrometheusRule CRs — never a copy — and pins every rule in the group both ways: quiet on today's healthy prod state, firing on the reconstructed bad state, with `for:` durations asserted. Two mutation checks confirm the tests are not vacuous: dropping `or vector(0)` and breaking the `unless on (schedule)` join each turn the suite red. This is the tool that makes both of this repo's recurring alert bugs — permanently-firing and structurally inert — provable offline, which neither yamllint, Kyverno, nor a healthy cluster can do. Proof of the fix is by construction; the live metrics cannot show it any more, because `velero-daily` was unpaused on 2026-09-12 and has since completed 2492/2492 items. The throwaway probe Schedule and the probe KSM deployment were deleted and the deletion confirmed; `velero-daily` remains unpaused. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 23:45:15 -05:00
jakman85andClaude Opus 5	5fa278f61c	fix(ci): declare whole-repo scope instead of hand-writing a gate stage per invariant R-TEST-03 runs the conformance subset a change TOUCHES. That has one blind spot, and it is the worst-shaped kind: an invariant that quantifies over the whole tracked tree — or over subtrees ci/touched-requirements.sh's coarse path map keys to NOTHING (apps/, infra/, most of platform/) — cannot be selected BY the requirements a change touches, because the change most likely to break it attributes no requirement at all. Requirement-scoped selection then filters the check out of exactly that change. The repo's answer had been a hand-written "UNCONDITIONAL" stage per such check in ci/l0.sh: 3c flux-postbuild-substitution, 3d keycloak-client-secret-reconciler, 3e bulk-tier-distinct-from-replicated. Three near-identical 12-line rationales, three hardcoded check.sh paths, three copies of the same rc 0/3/* case block — and worst, a mechanism that failed SILENTLY, because whoever added the NEXT whole-repo invariant had to REMEMBER to write a stage, and if they forgot the check ran only when some unrelated requirement happened to be touched. A test now DECLARES the scope of its invariant and the runner enforces it: metadata: annotations: conformance.home.k3s/scope: "whole-repo" * conformance/lib/common.sh conf_dirs_for_scope(s) resolves the annotation. Carries the general rationale, once, instead of per stage. * conformance/run.sh --include-scope SCOPE unions declaring tests into select_dirs, still bounded by --layer. New --print-selection resolves the selection and exits, running nothing. * lib/run_l1_offline.py --unwired-scope: a SKIP of a declaring test exits 3, not 0. --summary-out: one step-summary line per executed check. * ci/l0.sh Stage 3b gains --include-scope whole-repo and an rc-3 arm. Stages 3c/3d/3e DELETED (-150 lines). * ci/l1.sh, ci/conformance.sh pass --include-scope too, so the next whole-repo invariant at ANY layer needs only the annotation. rc 3 SEMANTICS PRESERVED EXACTLY. A whole-repo check reporting "nothing to audit" is reporting that the tree it audits has VANISHED. The deleted stages called that "not a pass" and set the gate's rc=3; the runner now returns 3 and ci/l0.sh maps it to rc=3. A missing check file still degrades to not-green, never to silence. THE REASONING WAS RELOCATED, NOT DELETED. Each stage's incident narrative moved into the header of the test it describes, beside its scope annotation: the Cilium k8sServiceHost empty-substitution CNI outage (2026-09-01), 12 unusable Keycloak clients behind a green cluster (2026-09-06), and a multi-TB bulk volume landing on the 3-replica tier. AUDITED ALL 13 EXISTING l0-static TESTS, individually. 10 declare whole-repo scope; 3 are DELIBERATELY left requirement-scoped and say so in their own headers: * placeholder-guard-coupling — inputs are bootstrap/ + clusters/, both already mapped to a requirement it declares. Scoping cannot hide a breaking change. * dev-nodes-provisioned-as-code — inputs are infra/ and nothing else. * spec-immutability-enforced — inputs are ci/ and the read-only spec repo. The latter two were nonetheless under-gated because the PATH MAP was incomplete, which is the right place to fix a path-scoped invariant: infra/* -> R-ENV-01 added, and R-TEST-06 added to .gitea/\|ci/ (it is no longer an uncovered ID — spec-immutability-enforced carries the label, so attributing it no longer false-blocks the coverage gate). NEW conformance/tests/l0-static/whole-repo-invariants-wired (R-TEST-03) keeps the declarative mechanism honest for ALL declaring tests at once, replacing the three tests' individual "ci/l0.sh greps my own check.sh path" self-assertions. It proves the union EMPIRICALLY: with a requirement filter matching no test the L0 selection is empty, and with --include-scope it is every declaring test; a scope-declared synthetic SKIP exits 3 while an undeclared one exits 0; and a ci/.sh that hardcodes a check.sh path must belong to a test that also declares the scope, so the bespoke pattern cannot creep back. TASK 2 — a changed test attributes the requirements it CERTIFIES. ci/touched-requirements.sh gains detection rule 3: a change anywhere under conformance/tests/<test>/ attributes that test's own conformance.home.k3s/R- labels, read from its chainsaw-test.yaml. The content scan could not close this — it reads only added/removed lines, so it sees a change to the LABEL BLOCK but not to the test BODY, which is exactly the case that matters. Labels are read from HEAD or, failing that, from the BASE, so DELETING a test still attributes what it used to certify; otherwise removing the only verifier of a requirement is the cheapest way to make the gate stop checking it. Whole-line `#` comments are dropped first and *.md is untouched, honouring the existing "reference, not declaration" rules. Both changes are fail-closed: every one of them can only ADD tests or requirements to a run, never remove one. TESTS: new ci/tests/touched-requirements-test-attribution.test.sh (15 assertions) covers rule 3 — body-only edit, nested-fixture walk-up, test deletion, header-comment R-IDs not attributed, non-test paths unaffected, and both new path-map entries. The two existing suites still pass (39 assertions). `make -C ci test` runs all three. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 23:45:06 -05:00
jakman85	78d76fa089	merge: qBittorrent can finally write where the arrs can see it gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details qBittorrent's SavePath was /downloads and no such directory existed in the container, so the download client had no working destination and nothing downstream could see its output. It now mounts the same NFS export the arrs use, narrowed to the downloads subtree via subPath — one mount, so the EXDEV hardlink trap the bulk-tier test guards stays closed. PUID/PGID are load-bearing: the export is owned 1000:1000 and the image's default 911 is denied.	2026-09-12 23:36:48 -05:00
jakman85	9e7d919d12	merge: CNPG primaries spread one per node, and the condition is now reported All three primaries had re-concentrated onto prod-1 (the node with known etcd/Longhorn I/O contention), so losing it meant three simultaneous failovers. Two switchovers rebalanced them; semaphore-db deliberately stays on prod-1 as the lightest writer, saving a third write interruption. CNPG offers no role-aware scheduling — targetPrimary exists only under status — so the placement is CURRENT, NOT DURABLE: the next failover or rolling update can re-concentrate it. What is durable is CNPGPrimariesCoLocated, which reports the condition instead of leaving it to be found by hand.	2026-09-12 23:36:48 -05:00
jakman85andClaude Opus 5	9b4744d450	fix(devcontainer): dev.sh defaults to prod — dev is decommissioned gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details Every unprefixed './dev.sh run kubectl ...' pointed at the dev cluster, which was decommissioned on 2026-09-12. The banner cheerfully named dev and the command then hung or errored against a context that no longer answers, with nothing saying the cluster is gone. I hit this repeatedly tonight. CLUSTER=dev is still accepted rather than removed, because the dev VM disks were deliberately kept so the environment is reversible — but it now warns. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 23:33:12 -05:00
jakman85andClaude Opus 5	78bc53afc7	feat(qbittorrent): attach the downloads area of the bulk export — the chain was broken end to end qBittorrent was the one member of the download chain with NO media volume. Its live config said Session\DefaultSavePath=/downloads/ Session\TempPath=/downloads/incomplete/ Downloads\SavePath=/downloads/ while `ls /downloads` in the container returned "No such file or directory". The catalog entry declared exactly one volume — config at /config on longhorn — and carried a comment deferring the downloads volume "until OQ-2 is resolved". OQ-2 WAS resolved (schema 1.5.0 static bulk NFS PVs); plex, sonarr, radarr and nextcloud all took their mounts and qBittorrent's was simply never added. So the download client was writing, or failing to write, a path that did not exist — and anything it did create would have been invisible to sonarr/radarr, lost on every pod recreation, and consuming the node's ephemeral disk until then. WHAT THIS ADDS to catalog/qbittorrent.yaml (rendered into apps/wave-0): * storage[downloads]: class bulk, export /mnt/Storage/plex, subPath downloads, mountPath /downloads, 9Ti advisory. ONE mount of the SAME export sonarr, radarr (rw) and plex (ro) already consume, narrowed to the handoff subtree. * env PUID=1000 / PGID=3000. LOAD-BEARING, not cosmetic: the container ran as uid 911 and /mnt/Storage/plex/downloads is 0755 owned 1000:1000, so the export would have mounted cleanly and then failed every write with EACCES. Measured both ways on prod: a write as uid 1000 succeeded, the same write as uid 911 was denied. * podSecurityContext.fsGroup 3000, for the LONGHORN config PVC only, exactly as sonarr/radarr set it — fsGroup is a no-op on an in-tree NFS volume. WHY subPath AND NOT THE EXPORT ROOT AT /media (sonarr/radarr's shape): a download client has no business writing the library. The arrs need the root mount because THEY hardlink downloads/ -> tv/\|movies/ and link() is a same-MOUNT operation; qBittorrent performs no such link, so narrowing to downloads/ removes movies/, tv/ and photos/ from its mount namespace without removing any operation it performs. It is still ONE mount of the export, so the L0 assertion that no workload mounts one export twice (the EXDEV trap) holds — and the arrs' own link() stays intra-mount. Proven live: nlink 1 -> 2, same inode, no EXDEV, every step executed as uid 1000. NETWORK: nothing to grant and nothing to steer. An NFS PersistentVolume is mounted by the NODE KERNEL in the host netns — the established socket to 192.168.2.102:2049 is on the node's own LAN address, the pod netns has no socket to 2049 at all, and no CiliumNetworkPolicy on this cluster names that host or port. `ip route get 192.168.2.102` picks eno1 even on the tunnel-owning node, and Cilium's own egress map lists 192.168.0.0/16 as an Excluded CIDR for every vpn-class pod while 0.0.0.0/0 goes to the gateway. Public egress still exits the PIA address; the VPN guarantee is untouched. NOT CLOSED BY THIS COMMIT, deliberately: absolute paths still differ between this pod (/downloads/x) and the arrs (/media/downloads/x), so Completed Download Handling needs one Remote Path Mapping per arr. That lives in the arrs' own databases, not in git. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 23:32:03 -05:00
jakman85andClaude Opus 5	505083fded	feat(cnpg): alert when >1 CNPG primary shares a node — the drift CNPG cannot prevent `instances: 3` + `podAntiAffinityType: required` fixed the INSTANCE spread but moved the PRIMARY concentration from k3s-prod-2 to k3s-prod-1: all three primaries (keycloak-db-2, nextcloud-db-2, semaphore-db-2) ended up on one node, so losing it meant three simultaneous failovers. There is NO declarative fix, established from the CRD installed on prod (CNPG 1.30.0, clusters.postgresql.cnpg.io/v1) rather than from memory. Every spec.affinity field and spec.topologySpreadConstraints applies to "all the cluster's pods" and is not role-aware; spec.replica.primary names a Cluster object in a distributed topology, not a node; primaryUpdateMethod/switchoverDelay/primaryLease govern how a promotion happens, never where. `targetPrimary` exists only under STATUS, never spec — so `cnpg promote` is an imperative status patch by construction and there is nothing to put in git. The obvious workaround is worse than nothing: pod anti-affinity is only ever ...IgnoredDuringExecution while CNPG promotes by relabelling a RUNNING pod, and with 3 instances on 3 nodes it would leave the third instance unschedulable. So this commits the part that IS durable — visibility — and rebalances live. CNPGPrimariesCoLocated joins CNPG's own answer to "who is primary" (the <cluster>-rw Service selects only the primary) against the node: kube_endpointslice_endpoints carries endpoint_nodename + targetref_name, which two label_replaces lift into node/pod. Neither family works alone — cnpg_* metrics have no node label, and kube_pod_labels is EMPTY here (no kube-state-metrics label allowlist), so cnpg.io/instanceRole is not in Prometheus at all. `* on (namespace, pod) group_left () cnpg_collector_up` keeps an unrelated `-rw` Service from triggering it, and uses the one metric present for EVERY instance rather than the cnpg_pg_stat_ family that is missing for nextcloud-db. A Ready-node gate stops this becoming another permanently-firing alert: with three clusters on three nodes a node loss NECESSARILY co-locates two primaries, so the rule only speaks when the spread is actually achievable, and re-arms when the node returns. Proven both ways. Live: the expr returned {node="k3s-prod-1"} => 3 before, empty after, with the inner join still resolving all three primaries (quiet because false, not blind); loaded into prod's Prometheus as a throwaway `for: 1m` twin it went pending at +70s and firing at +130s. alerts.promtool-test.yaml pins four cases under promtool 3.14.0 (the exact version prod runs) and is mutation-checked — dropping the join, dropping the gate or shortening `for:` each makes a test FAIL. Rebalanced live, one cluster at a time: keycloak-db -> k3s-prod-2, nextcloud-db -> k3s-prod-3, semaphore-db deliberately LEFT on k3s-prod-1 (lightest writer, and that node has the known etcd/Longhorn I/O contention) — two switchovers, not three. Write outage measured with a `select txid_current()` probe, not estimated: 9.4s and 9.43s, one contiguous burst each. All three back to readyInstances: 3 with synchronous_standby_names = ANY 1 (...) and both standbys streaming/quorum. The placement is CURRENT, NOT DURABLE, and the alert is the point: the next failover or rolling update can re-concentrate the primaries, and now that gets reported instead of discovered by hand. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 23:32:03 -05:00
jakman85	df77affc23	merge: the nextcloud database has never been monitored gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details cnpg_metrics_exporter lacked CONNECT on the nextcloud database: an out-of-band REVOKE materialized its datacl and dropped PUBLIC's implicit CONNECT, which keycloak and semaphore still inherit. CNPG's API cannot express a database-level grant, so this follows the repo's existing level-triggered CronJob reconciler pattern rather than a one-shot Job that a rebuild would silently skip. CONNECT only; pg_monitor already carries the read access.	2026-09-12 23:28:23 -05:00
jakman85	501d587541	merge: CNPG DR alerts scoped to the cluster, and the hint that lied CNPGNoRecoverabilityPoint fired permanently on all six CNPG replicas because only the primary owns the backup catalogue; CNPGNoRecentBaseBackup had the same latent defect, held back only by a > 0 guard. Both now aggregate pod away and key on the cluster. The remediation text also told the on-call to run 'kubectl get backup', which reports no resources here.	2026-09-12 23:28:23 -05:00
jakman85andClaude Opus 5	bacb4c61ee	fix(cnpg): the alert told the on-call to run the command that lies CNPGNoRecentBaseBackup's own remediation hint said `kubectl get backup`. On this cluster longhorn.io, postgresql.cnpg.io and velero.io each register kind Backup, so the bare short name resolves to Longhorn's empty set and prints 'No resources found' while CNPG backups exist. An on-call following the alert that fires BECAUSE backups look stale would have been told, with authority, that no backups exist at all — and might re-bootstrap a database over good ones. Fully qualify it. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 23:28:12 -05:00
jakman85andClaude Opus 5	7880d84bce	fix(cnpg): grant cnpg_metrics_exporter CONNECT on the nextcloud database The CNPG metrics exporter could not CONNECT to the `nextcloud` application database, so every user-defined collector query against that target failed on all three instances: FATAL: permission denied for database "nextcloud" (SQLSTATE 42501) DETAIL: User does not have CONNECT privilege. The whole cnpg_pg_stat_* / pg_settings / pg_postmaster family was therefore MISSING for nextcloud-db while the scrape target still reported up=1 — the blind spot platform/cnpg/alerts.yaml's CNPGMetricsCollectorErroring was written to catch. It had been firing on all three pods; measured before this change, cnpg_pg_settings_setting had 765 series for keycloak, 765 for semaphore-db and ZERO for nextcloud-db. Nextcloud's database had never been monitored. Root cause is the live ACL, not the spec — all three Cluster specs bootstrap identically: nextcloud : datacl = {nextcloud=CTc/nextcloud} <- explicit, no PUBLIC entry keycloak : datacl = NULL semaphore : datacl = NULL A NULL datacl means PostgreSQL's defaults, which include CONNECT for PUBLIC, so keycloak and semaphore admit the exporter only by inheriting PUBLIC's implicit CONNECT. On `nextcloud` something ran the equivalent of REVOKE ALL ON DATABASE nextcloud FROM PUBLIC, which materialises datacl and drops that implicit entry. That REVOKE is in neither this repo nor DECISIONS-LOG.md — it was applied out of band, which is why the remedy has to be declarative rather than another hand-run psql. CNPG 1.30.0 cannot express a database-level GRANT (verified against the live CRDs): `Database` has no ACL field (`allowConnections` is the datallowconn flag, not a per-role grant), and managed.roles/DatabaseRole cover role attributes plus `inRoles` membership only — and membership would hand the exporter the owner's CREATE/TEMP and all table access to buy one CONNECT. postInitApplicationSQL runs once at bootstrap, so it cannot repair the already-bootstrapped instances that are actually broken. So this follows the pattern the repo already uses for state Flux cannot reach: the level-triggered CronJob reconciler alongside platform/keycloak/client-secret-reconciler.yaml. A one-shot Job is deliberately avoided — a completed Job never re-runs, so a rebuild or PITR restore would silently reopen the blind spot. Exactly one privilege, and nothing else: GRANT CONNECT ON DATABASE nextcloud TO cnpg_metrics_exporter; giving `cnpg_metrics_exporter=c/nextcloud` — CONNECT only, no CREATE, no TEMPORARY, no ownership, no superuser, no table or schema grants. Nothing more is needed because CNPG already creates the role as a member of the built-in pg_monitor role (verified on all three clusters), which carries the read access the collector's pg_catalog queries want. It runs as the database OWNER, not a superuser: enableSuperuserAccess stays off, and a database owner holds the grant option on its own database. Verified live on prod: * the grant issued under SET ROLE nextcloud, proving no superuser is needed; * permission-denied and "Error collecting user query" both went to 0 on nextcloud-db-1/-2/-3, i.e. every collector query now succeeds; * cnpg_pg_settings_setting now 765 series for nextcloud-db, matching the other two, and cnpg_pg_postmaster_start_time present on all three instances; * cnpg_last_error = 0 on all NINE instances, so keycloak-db and semaphore-db are unaffected and their datacl is still NULL; * convergence proven by revoking the grant again and letting the committed CronJob restore it, then re-running it against a converged database to show idempotence; * CNPGMetricsCollectorErroring, firing on all three pods beforehand, has cleared (`for: 6h` delays firing, not resolution). Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 23:22:27 -05:00
jakman85andClaude Opus 5	5e7ab65c65	docs: kubectl get backup lies on this cluster — ambiguous Backup short name Three API groups register kind Backup (longhorn, cnpg, velero). The bare short name resolves to Longhorn's empty set, so kubectl reports absence while 27 CNPG backups and live Velero backups exist. A false negative on 'do I have backups' is the worst possible time to be wrong, so record the rule: fully-qualified resource names only. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 23:15:06 -05:00
jakman85andClaude Opus 5	21491f901b	fix(cnpg): scope DR alerts to the CLUSTER, not the pod — stop a permanent false positive CNPGNoRecoverabilityPoint has been firing continuously on prod against six pods — keycloak-db-1/-3, nextcloud-db-1/-3, semaphore-db-1/-3 — i.e. exactly the REPLICAS of all three Clusters, while the databases themselves are fine: every Cluster reports firstRecoverabilityPoint=2026-09-07T07:0x, a lastSuccessfulBackup within the last few hours, ContinuousArchiving=True and LastBackupSucceeded=True. Root cause is aggregation scope, not archiving. `cnpg_collector_first_recoverability_point` reads the backup CATALOGUE, which only the primary owns; a replica reports a hard 0 for as long as it is a replica. The metric carries no cnpg.io/cluster label (its label set on prod is container/endpoint/instance/job/namespace/pod), so `max by (namespace, pod)` evaluated each replica as its own subject and an `== 0` comparison then held forever. The rule went from correct to permanently wrong the moment the Clusters went to `instances: 3`; nothing about the expression changed. WHY THIS IS WORSE THAN HAVING NO ALERT. An alert that is always firing has zero information content: its value is identical in the healthy state and in the failure state, so it cannot distinguish them. It is worse than silence in two specific ways. First, it trains the operator to delete the notification on sight, so the day a base backup genuinely stops the seventh identical email is discarded with the other six — the alert actively MASKS the failure it was written to catch, which for a DR alarm means discovering the loss at restore time. Second, it is indistinguishable from a broken alert, so nobody can tell whether the absence of new alerts means health or means the rule rotted. This file exists because a real 38-hour archiving failure told nobody; a rule that cries wolf about six healthy replicas rebuilds exactly the blindness it was supposed to remove. This repo has been here before — see the permanently-firing critical VPNTunnelDown post-mortem in platform/monitoring/alerts.yaml. Fix: group by (namespace, job) and aggregate `pod` away. The CNPG operator creates one PodMonitor per Cluster, named after the Cluster, in the Cluster's namespace, so prometheus-operator derives job="<namespace>/<cluster>" — keycloak/keycloak-db, nextcloud-db/nextcloud-db, semaphore-db/semaphore-db — which identifies the CNPG Cluster uniquely. `job` rather than bare `namespace` deliberately: namespace alone works today (one Cluster per namespace, verified live) but would let two Clusters in one namespace mask each other, since the healthy one's non-zero value would satisfy `max` for both. The alert now states what its summary always claimed — this CLUSTER has no recoverability point — and fires only when no instance of that Cluster reports one. Proven live against prod Prometheus, read-only (promtool query instant): OLD max by (namespace, pod) (...) == 0 -> 6 series, and 6 ALERTS firing NEW max by (namespace, job) (...) == 0 -> empty The new expression is not vacuous: forcing the metric to 0 for one cluster's primary only yields exactly one series, {job="keycloak/keycloak-db"}, so a genuine single-cluster loss of DR still alerts, per cluster. `promtool check rules` passes on all 5 rules. CNPGNoRecentBaseBackup carried the same defect and is fixed the same way. `cnpg_collector_last_available_backup_timestamp` is also primary-only: on prod the primaries read 3.1h/2.1h/0.1h old while all six replicas read 0, i.e. time()-0 = ~56.7 YEARS. It was not firing only because the `> 0` guard was load-bearing — a single clause was all that stood between this rule and six more permanent false positives, too thin a margin for a DR alarm. Aggregating `pod` away removes the dependence: `max` takes the primary's timestamp as the cluster's answer, and the guard now means "this cluster has had at least one backup". Verdict unchanged live (old empty, new empty). The other three rules were audited and deliberately left on `by (namespace, pod)`: * CNPGWALArchiveBacklog — every instance reports archive_status and the shape is `> 0`, so a replica at 0 never fires and no pod can mask another; `pod` names the instance whose archiver is stuck. * CNPGWALVolumeNotRecycling — per-pod is REQUIRED, not merely safe: aggregating it away would compare one instance's WAL count against another's max_wal_size. * CNPGMetricsCollectorErroring — unchanged in meaning AND grouping. Its grouping is already correct: cnpg_last_error is genuinely per-instance, and prod proves it discriminates rather than tracking role — all three nextcloud-db pods report 1 while all six keycloak/semaphore pods report 0. It is a true positive (the nextcloud 42501 CONNECT grant) owned by separate work; nothing here silences or rescopes it. The general rule, recorded in the file: `== 0`/staleness shapes must aggregate `pod` away or every structurally-silent pod fires forever; `> 0` shapes are safe per-pod and keep the instance name. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 23:09:53 -05:00
jakman85andClaude Opus 5	ebce6dfabb	docs: the Flux->Gitea allowance is rendered and Flux-adopted — debt closed gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 22:58:39 -05:00
jakman85	eda4a9f674	merge: the Flux->Gitea allowance is rendered, not hand-applied gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details	2026-09-12 22:51:35 -05:00
jakman85andClaude Opus 5	d617983e83	feat(gitea): render gitea-allow-flux-ingress — prod's GitOps source, in git at last The CNP that lets flux-system's source-controller clone from the in-cluster Gitea Service was APPLIED LIVE on prod on 2026-09-13 and existed nowhere in this repo. It carried a provenance annotation saying so. Until now it could not be authored: the gitea CNPs are GENERATED from the catalog entry, and the contract had no field for "also admit this in-cluster client" (fixed in the previous commit). Why it matters more than a missing allow normally would. prod's GitRepository points at the Service, not a public name (clusters/prod/kustomization.yaml: `url: http://gitea.gitea.svc.cluster.local:3000/Home-Network/home-network-k3s-impl.git`), deliberately, so prod's GitOps depends on neither public DNS, Cloudflare, prod's own Gateway nor ${BASE_DOMAIN}. But gitea-allow-gateway-ingress admits only `fromEntities: [ingress]`, so source-controller's clone was dropped — and the HTTPRoute that would have served the post-apex-flip public name instead lives in a DOWNSTREAM tier that cannot apply while the source is down. Source down -> routes never rename -> source stays down. Every `kubectl delete cnp` and every cluster rebuild re-armed that deadlock, and it would have been extremely hard to diagnose cold. The rendered object is EQUIVALENT to the live one — same apiVersion/kind, same name/namespace/labels, and a byte-identical spec (endpointSelector app.kubernetes.io/name=gitea; fromEndpoints matchLabels k8s:io.kubernetes.pod.namespace=flux-system; toPorts 3000/TCP), verified against `kubectl --context k3s-prod -n gitea get cnp gitea-allow-flux-ingress -o json`. The name matches, so Flux ADOPTS the live object in place: no spec change, so no policy recompute and no flap on prod's own source. The only difference is that the rendered object carries no `homelab.keithjackson.dev/provenance` annotation — the reasoning now lives where the object is authored, in catalog/gitea.yaml, next to the field that generates it. Golden re-rendered with the canonical command; the whole apps/ diff is the one new document. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 22:46:57 -05:00
jakman85andClaude Opus 5	1f20506829	feat(renderer): inClusterClients — authorable direct east-west ingress (schema 1.5.0) A platform controller that dials a service's ClusterIP directly had NO authorable form in the ServiceDefinition contract: * ingressRoutes renders the north-south allow only — gateway-ingress admits `fromEntities: [ingress]`, the Gateway's Envoy identity, and nothing else; * dependencies compiles edges only BETWEEN CATALOG SERVICES, from the source entry's own declaration. Under the default-deny baseline (R-NET-01) such a client is therefore DROPPED, and the only way to unblock it was a live `kubectl apply` of a hand-written CNP — a cluster object outside git, one `kubectl delete` or one rebuild from coming back. `inClusterClients: [{namespace, port, protocol?, name?}]` closes that. Each entry renders ONE least-privilege ingress CiliumNetworkPolicy: this service's endpointSelector, `fromEndpoints` matching ONLY `k8s:io.kubernetes.pod.namespace`, `toPorts` carrying ONLY the declared port/protocol — never `fromEntities: all`. Additive: the namespace default-deny keeps denying everything undeclared (R-NET-02), and the entry is NOT a dependency edge, so it creates no wave ordering. Shape notes: * selected by NAMESPACE, not pod labels: an upstream controller's labels belong to its chart, not to this platform's contract. The cost is explicit — every pod in that namespace is admitted on that port, so the field is for platform-controller namespaces (flux-system, monitoring), never a shared one. The schema description says so. * optional `name` (default: the namespace) only names the rendered policy, `<service>-allow-<name>-ingress`. That name is operator-facing (kubectl get cnp, a Hubble drop), and pinning it lets an allowance that already had to be applied live be authored under its EXISTING name, so Flux adopts the live object in place instead of creating a near-duplicate and leaving the original unmanaged. * one field, not a policy DSL: no arbitrary selectors, entities or CIDRs. Covered by render.py --selftest (4c): name/namespace/selector/port shape, no fromEntities widening, default-deny still `ingress: []`, the gateway allow not displaced, `name` defaulting to the namespace, and nothing rendered when the field is absent. No golden changes here — no catalog entry uses the field yet. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 22:46:57 -05:00
jakman85andClaude Opus 5	a863542bf7	docs: record the apex migration, the GitOps deadlock, and the live-only policy Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 22:37:58 -05:00
jakman85andClaude Opus 5	f252ecec25	fix(gitops): prod's source is the in-cluster Service, not a public name gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details The apex flip deadlocked prod's GitOps and this is the durable fix. The GitRepository pointed at a public hostname. The flip renamed prod's HTTPRoute hostnames, so that name stopped serving — but the HTTPRoute that would serve the new name lives in a DOWNSTREAM tier which cannot reconcile while the source is broken. Source down -> routes never rename -> source stays down. Observed live: srcReady=False, "unable to list remote for https://gitea.home.keithjackson.dev". Pointing at gitea.gitea.svc.cluster.local:3000 removes the cycle permanently: prod's GitOps no longer depends on public DNS, Cloudflare, its own Gateway, or ${BASE_DOMAIN}. No future domain change can sever prod from its own source. Plain http is deliberate: cluster-internal traffic to a ClusterIP, TLS terminates at the Gateway for external clients, the GitRepository has no secretRef (anonymous clone), and all secrets are SOPS-encrypted at rest regardless of transport. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 22:14:27 -05:00
jakman85andClaude Opus 5	ae37e27f14	fix(gitops): move prod's own Git source with the apex flip gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details The GitRepository url is a hardcoded literal, not templated on ${BASE_DOMAIN}, so the apex flip did not carry it. But the flip renames prod's HTTPRoute hostnames, so external-dns (policy=sync) deletes gitea.prod.home.keithjackson.dev and CoreDNS's split-horizon zone moves from prod.home. to home. — which would have severed prod from its OWN GitOps source moments after the flip reconciled. Landed in the same push as the flip deliberately: prod has not yet fetched `9161cab` (still on `b9c0970`, all hostnames still prod.home.), so both commits arrive in one fetch and the url changes in the same apply that renames the routes. A live patch would not survive — the root Kustomization renders this very file and reverts it. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 22:10:57 -05:00
jakman85andClaude Opus 5	9161cab9cd	feat(dns): prod takes the apex — home.keithjackson.dev gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details Dev is decommissioned and has released the name. Prerequisites completed and verified BEFORE this flip: 1. dev's 18 A records released by its own external-dns (policy=sync; sources deleted after suspending dev's Flux so they could not be recreated). 2. dev's 18 LEGACY UNPREFIXED TXT registry entries deleted from Cloudflare by hand. external-dns v0.21.0 deleted only the `a-` prefixed form and then logged "All records are already up to date" — it would NEVER have cleared the older unprefixed format it no longer writes but still READS for compatibility. They carried `owner=default`; prod runs `owner=prod`, so prod could not have claimed these names while they existed. This was the real blocker, and it was invisible until measured. 3. dev's `.home.keithjackson.dev` + apex Certificate deleted so it stops renewing. Issued ~18d ago, so prod's request is #1 this week against LE's duplicate limit (5 per identical name set per week). Verified at the AUTHORITY (betty.ns.cloudflare.com), not via a resolver: 1.1.1.1 served stale TXT for minutes after deletion and would have shown a false failure. UNCHANGED, deliberately: EXTERNALDNS_TXT_OWNER_ID stays `prod` — giving prod `default` would let it adopt and sync-delete records it does not own, which is the 2026-09-04 incident. EXTERNALDNS_DOMAIN_FILTER stays the ZONE `keithjackson.dev`: finding F-DNS1 proved a strict subdomain filter drops the hosted zone and external-dns then publishes NOTHING. The zone has wildcards `.keithjackson.dev` and `.home.keithjackson.dev` pointing at the public IP (legacy non-k3s prod). Dev's specific records were shadowing the latter; prod's new specific records shadow it again. The legacy stack lives on `.keithjackson.dev` and is untouched. k3s_base_domain moves with it to hold the [parity] contract, but that value renders the apiserver's install-time oidc-issuer-url: it does NOT take effect until the k3s_server role re-renders config.yaml and k3s is restarted serial: 1. Until then the apiserver trusts the OLD issuer while Keycloak serves the NEW one, so kubectl OIDC fails in that window. Cert-based kubeconfigs are unaffected. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 22:07:20 -05:00
jakman85	b9c0970232	merge: Keycloak client URLs reconcile, not just secrets gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details	2026-09-12 21:35:15 -05:00
jakman85andClaude Opus 5	f7d4648851	docs(keycloak): record the URL-reconciliation decisions Records K1-K9: extend-vs-sibling and why, the generated+gated desired state, verbatim token pass-through, per-pass fail-closed, order-insensitivity, unmanaged-means-untouched, public clients' URLs managed, one alert pair not two, and the mutation harness. Also records a process note: my first pass reported three false findings (reconciler absent from main, import not templated, BASE_DOMAIN dead) because I diffed against origin/main -- origin is dev's Gitea, stale at `4809199`, while the live remote is prod-gitea at `fa4deac`. All three were wrong. The rule for this repo is written down so the next agent does not repeat it: origin is not the source of truth. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 21:31:26 -05:00
jakman85andClaude Opus 5	6202d92c27	test(conformance): gate the URL desired state, and prove the selftest can fail L0 (212 assertions, was 175): * runs ci/render-keycloak-client-urls.py --check, so client-urls.yaml must be exactly what the realm import renders. A hand edit that makes the two disagree is unmergeable -- this is what keeps the realm import the single source of truth for hostnames rather than just the place they were first typed. * asserts client-urls.yaml and the realm import are rendered by the SAME Flux Kustomization. postBuild.substituteFrom is per-Kustomization, so a split would let one ${BASE_DOMAIN} resolve to two different values and the realm would converge on a hostname the HTTPRoutes do not serve. * asserts the desired state uses no substitution token the realm definition lacks, names only clients the import declares, and declares no field outside the five-field write surface; that the CronJob mounts it readOnly at the path the script defaults to; and that the script, renderer and check agree on which fields are writable. * NEW mutations.py: injects 12 plausible regressions into the EXTRACTED script and requires --selftest to catch every one. A passing suite proves nothing if it cannot fail, and "reported success while the realm was broken" is precisely what this component exists to end -- so the same standard applies to its own test. ~20s. L2: the audit Job now mounts keycloak-client-urls and asserts KEYCLOAK_CLIENT_URLS_AUDIT_OK. Without that it would pass on a realm still holding pre-migration redirect URIs -- the exact state prod was measured in. Alerts: the existing Failing/Stale pair is EXTENDED to name redirect-URI drift, not duplicated. One CronJob gets one pair; a second pair on the same job_name regex would fire twice for one cause. No new critical, no new permanently-firing rule. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 21:26:12 -05:00
jakman85andClaude Opus 5	23ea72d01e	feat(keycloak): reconcile client redirect URIs, not just client secrets The realm import is a one-shot that never re-runs, so a redirect URI changed in git reaches the realm never. Prod's stored URIs were measured still on the pre-migration hostname while git declared the new one -- after a BASE_DOMAIN flip every OIDC login fails with a redirect_uri mismatch, with the CR still reading Done=True and nothing alerting. Same failure class as the literal-placeholder secrets, same cure. EXTENDS the existing reconciler rather than adding a sibling. Decisive reason: a URL write needs NO new endpoint. It is the same minimal client PUT, and the current URLs already come back in the allow-listed client lookup -- so the allow-list is widened by nothing and the users/groups/roles/authentication guarantee is preserved unchanged rather than re-declared in a second copy. The ConfigMap also carries exactly one script key by L0 contract, so a sibling would have had to fork ~300 lines of safety-critical code that the header explicitly forbids forking. Reconciles redirectUris, webOrigins, rootUrl, baseUrl, adminUrl -- only the fields a client actually declares, and on a write only the fields that actually drifted. A field the desired state omits is UNMANAGED and left untouched; absence never means force-to-empty. THE TWO PASSES ARE INDEPENDENT. Each reads its own desired state and fails closed for its own fields only. A malformed URL ConfigMap cannot stop client-SECRET repair, which would otherwise make credential healing depend on a hostname file. List fields compare order-insensitively (sorted, not set -- duplicates are still drift), because Keycloak does not preserve the authored order of redirectUris and an order-sensitive compare would rewrite every client forever while reporting success. --selftest grows to 84 assertions covering the domain flip, a client gaining a URI, a client losing a stale one, a public client's URLs being managed while its secret is not, per-pass fail-closed, order-insensitivity, and a server that accepts a PUT without applying it. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 21:25:34 -05:00
jakman85andClaude Opus 5	51d8676f90	feat(keycloak): render the client-URL desired state from the realm import The realm import is a one-shot: once Done=True it never re-runs, so a redirect URI changed in git never reaches the realm. Healing that needs a DESIRED STATE the reconciler can read -- and the reconciler pod has no Kubernetes API access at all (automountServiceAccountToken: false), so it cannot read the KeycloakRealmImport CR. Its desired state must be mounted, hence a ConfigMap. A hand-authored ConfigMap would be a second source of truth for every public hostname in the realm, free to disagree with the import -- the same shape as the bug being closed. So it is GENERATED from the import and the render is gated, the same render==golden idiom catalog/render.py already uses for apps/. URL strings are copied VERBATIM: a ${BASE_DOMAIN} token in the source appears unchanged here, and both files are rendered by the SAME Flux Kustomization (./platform-config), so both are substituted in one pass with one value. The renderer never parses a hostname and never reads cluster-vars, so there is no second place the domain can be decided. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 21:24:52 -05:00
jakman85andClaude Opus 5	fa4deac877	fix(conformance): the dirty-tree assertion must judge HONOURED evidence gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details One historical `-dirty` row reddened this gate permanently, even after a clean re-run of the same test. status.py already resolves duplicates latest-wins (latest_by_test), so a superseded row is not evidence of anything — but this check iterated every row, so the poisoned row from the git_sha bug (fixed in the parent commit) would have failed the audit forever. The dirty-tree assertion is now scoped to rows status.py would actually honour, by importing latest_by_test rather than reimplementing the rule, so the two cannot disagree about which rows are live claims. Structural assertions — provenance keys, sentinel, test-still-exists — still apply to EVERY row. Nothing is erased: the row stays in the ledger and runs/20260913T014437Z-19fc8bc/ run.log stays committed. It simply stops being read as a live claim. Verified the guard is intact afterwards — the check still REFUSES both forgeries it is built to catch (a self-skip presented as a pass, and a proven row from the root commit). A dirty-then-clean sequence is also the normal shape of noticing a dirty tree, fixing it, and re-running; that should not permanently fail an audit. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 21:12:55 -05:00
jakman85	a317493bca	merge: R-NET-06 re-proven on prod (recovery in <=3s)	2026-09-12 21:10:57 -05:00
jakman85andClaude Opus 5	c66a7e29e9	fix(conformance): the harness contradicted itself in two places 1. auditability-proofs-producible rejected any proven ledger row without a sentinel, with no layer exemption — while record_run.py DELIBERATELY records L0/L1 rows sentinel-less via its own SENTINEL_EXEMPT_LAYERS, because run_l1_offline.py renders skips as JUnit <skipped/> and those layers have no pass token to attribute. So ANY `--record` run touching L0 or L1 wrote a row this gate then rejected. Consequence: a requirement whose verifier set includes an L1 check could never be both PROVEN and L0-green — R-NET-06 is exactly such a requirement, and hit it the moment it was re-proven on prod. The check now honours the same constant instead of restating the policy, so the two halves cannot drift apart again. 2. git_sha() marked the tree dirty from tracked files, but conformance/runs/ ledger.jsonl IS tracked — so the FIRST `--record` run of a session dirtied the tree and every LATER run was stamped `-dirty`, which check.sh then refuses to honour. Recording one test poisoned the next, and that cost a second live fault-injection run against prod to work around. Its docstring already stated the intent for the untracked runs/<id>/ half of the same artifact; the ledger is the tracked half. Now excluded via pathspec, mirroring status.py's IMPL_EXCLUDE_PREFIXES which already lists this path for the same reason. Neither change weakens an assertion: (1) applies an exemption that already existed and was already documented, (2) stops the recorder invalidating its own output. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 21:10:38 -05:00
jakman85andClaude Opus 5	155df8ee3f	test(conformance): R-NET-06 PROVEN on prod — vpn-killswitch re-run from a clean tree Supersedes the -dirty row from the previous commit by the ledger's own append-only / latest-wins convention (lib/status.py latest_by_test): nothing is rewritten, a newer row simply outranks the older one. runs/20260913T014923Z-9d0814a vpn-killswitch (L3) proven env=prod git_sha 9d0814ae... (CLEAN — resolvable, so the freshness check passes) Fault injection against LIVE prod, gateway k3s-prod-3. Timeline (UTC): 01:49:27 tunnel-stop Job applied 01:49:31 Job succeeded — host vpn-egress-tunnel STOPPED, tun0 gone 01:49:31..01:51:01 vpn-class blackholed for 8 consecutive polls, NO response of any kind; internet-class unaffected on the WAN 01:51:02 self-healing finally: tunnel-restart Job 01:51:05 kubectl wait confirms the host brought the unit back 01:51:04 egress-uplink label AND the BPF egress map both move 134 -> 139 (measured independently, out of band, during the run) 01:51:08 vpn-class egress back on PIA 158.173.241.11 RECOVERY WAS UNAIDED and took ~3s from tunnel-up to a re-steered datapath — far inside the ~2-minute convergence timer, so the EVENT-DRIVEN up-script path did the work, not the timer backstop. This is the half that was previously unprovable: before the egress-uplink label, Cilium held the stale ifindex and vpn-class egress never came back. WHAT THE NO-LEAK ASSERTION DOES AND DOES NOT PROVE: it proves vpn-class fails CLOSED (zero packets out, never the WAN IP). It does NOT credit the host nftables backstop, which provably cannot fire for pod traffic — bpf_redirect() skips netfilter. Stopping the tunnel removes tun0 and Cilium drops on its own because the cached egress ifindex is unusable. Fail-closed is real; the mechanism is Cilium's drop, as the test header now says. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 20:52:47 -05:00
jakman85andClaude Opus 5	9d0814ae61	test(conformance): R-NET-06 run against prod — L1 proven, L3 pass recorded -dirty Two recorded runs against PROD (CONF_ENV=prod, context k3s-prod): 20260913T014321Z-19fc8bc egress-class-render (L1) proven 20260913T014437Z-19fc8bc vpn-killswitch (L3) proven, but git_sha -dirty The L3 run PASSED for real on prod — every declared sentinel present (NS_PRIVILEGED_OK, TUNNEL_HEALTHY_OK, NO_LEAK_OK, INTERNET_UNAFFECTED_OK, TUNNEL_SELF_HEAL_OK, TUNNEL_RECOVERED_OK), tunnel stopped 01:44:48Z and recovered unaided by 01:46:25Z. Its row is nevertheless NOT usable as proof: git_sha carries `-dirty`, so status.py's ancestry check cannot resolve it and the row degrades to stale-proof. Cause is a harness flaw, not the cluster: lib/record_run.py's git_sha() ignores UNTRACKED files (so a run's own runs/<id>/ does not mark it dirty) but conformance/runs/ledger.jsonl is TRACKED — so the FIRST --record run of a session appends a row, dirties the tree, and every SUBSEQUENT --record run in that same session is stamped -dirty. Recording one test poisons the proof of the next. Committed as-is rather than rewritten: the ledger is append-only and this is honest evidence of a real run. A clean re-run follows and supersedes it by latest-wins (lib/status.py latest_by_test). Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 20:48:55 -05:00
jakman85	19fc8bcb27	merge: VPN egress gateway on prod, plus three datapath fixes gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details	2026-09-12 18:15:02 -05:00
jakman85andClaude Opus 5	53e09a2610	docs(vpn): dev recovered the minute prod left the shared PIA server Closes the loop on the collision entry with the recovery evidence rather than leaving it as an open question for the operator: dev's last Inactivity timeout is 23:05:06 UTC — the minute prod switched to .135 — and at 23:12:03 that same session was still the live one, seven minutes unbroken against a two-minute cycle, 4/4 ping replies at 146 ms, 0% loss, and vpn-class pod egress back to 158.173.241.36 on three consecutive probes. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 18:12:46 -05:00
jakman85	9b97e073e2	merge: main (velero R2 + arr config PVC) into feat/vpn-egress-prod	2026-09-12 18:11:18 -05:00
jakman85andClaude Opus 5	38e4bf1459	docs(vpn): record the PIA server collision, and the accidental proof it handed us The stand-up broke dev, so the "one gateway per PIA server" rule and the measurement behind it are now written down where the next person will look: platform/vpn-egress/README.md and the session's DECISIONS-LOG entry. Both carry the verbatim journal lines — same server, same pushed `ifconfig 10.72.192.7` on both gateways — and the flap timestamps, because "same account, inside the connection limit" is true and is exactly what makes this non-obvious. Also corrected: the dev/prod table now shows the two environments on different servers with different exit IPs (dev 158.173.241.36, prod 158.173.241.11), and the open item about them sharing one public IP is replaced with the real remaining decision — whether prod gets its own PIA account, which would remove the hand-maintained per-environment server value entirely. The unplanned bonus, recorded in both places: moving prod to .135 gave it a genuinely NEW tunnel address on a real reconnect (10.72.192.7 ifindex 128 -> 10.150.192.86 ifindex 129) — the precise case Cilium does not follow and the one defect 1 was about. The egress-uplink label moved on its own, Cilium re-resolved, and all four vpn-class workloads kept egressing with no manual step. Better evidence than the synthetic address swap used to characterise it, and it cost nothing to capture. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 18:09:31 -05:00
jakman85andClaude Opus 5	37a6c94168	fix(vpn): two gateways on one pinned PIA server evict each other every 2 minutes Standing up prod broke dev, and the mechanism is worth spelling out because "same PIA account, well inside its connection limit" is true and still not enough. The vaulted .ovpn pins ONE server. With both gateways using it, dev and prod landed on the SAME server AND were handed the SAME tunnel address — both journals read `[Server-10779-2a] Peer Connection Initiated with [AF_INET]158.173.241.112:1198` and `ifconfig 10.72.192.7 255.255.192.0`. Each reconnect steals that address on that server; the loser then receives nothing, trips `--ping-restart` after 60s, reconnects and steals it back. Measured cycle: dev restarting at 22:57:05, 22:59:03, 23:01:05 UTC with prod interleaved about a minute off, each ~2 minutes apart, indefinitely. Neither environment had reliable vpn-class egress — prod's probes looked green only because they happened to land inside an up-window. Fix: `vpn_egress_gateway_pia_remote` ("<host> <port>", empty = honour the .ovpn) rewrites just the `remote` line of the deployed .ovpn, and prod's group_vars sets it to 158.173.241.135. That is another member of the same PIA Stockholm LB pool — `sweden.privacy.network` resolved from k3s-prod-3 to 158.173.241.{34,55,112,135,239} — and every server in the pool takes the same credentials and the same inline CA, so only the address differs. Dev keeps the .ovpn's .112 untouched, which is why the override is per-group rather than in the role or a second vaulted .ovpn. Two details that matter for whoever touches this next: * The regex is anchored to a line STARTING with `remote` plus whitespace, so `remote-cert-tls` and `remote-random` are not collateral damage. The task keeps `no_log: true` — the transformation never widens what is logged. * A PIA server address is NOT a credential, so it lives in group_vars/vpn_egress_gateway_prod/vars.yml in the clear, next to the vault rather than inside it. Keeping it out of the vault is what makes it reviewable. If prod ever gets its own PIA account, this override can be deleted again. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 18:05:49 -05:00
jakman85andClaude Opus 5	04606d48b2	docs(vpn): stop crediting the nftables kill-switch for a drop it never makes Four places in the tree told the reader that the host nftables table is the authoritative fail-closed backstop for egress-class=vpn pods: platform/vpn-egress README, egress-steering.yaml, the role README, and the vpn-killswitch test's own GIVEN/THEN prose. It is not, and cannot be — Cilium's eBPF egress-gateway datapath bpf_redirect()s those skbs to the egress device without invoking netfilter POSTROUTING. Corrected with the measurement in each place rather than softened, because a guard the repo trusts and that does not fire is worse than no guard. What replaces the claim, everywhere: vpn-class fails closed because CILIUM drops the traffic when it cannot use its cached egress interface, verified by tcpdump on both the WAN and tun0 during a tunnel stop and a tunnel restart (0 packets out either). The nft table keeps a narrower, now-proven job: node-originated traffic sourced from a tunnel address (killswitch_drop moved 0 -> 3 packets on k3s-prod-3 when such a packet was routed at the WAN). Also recorded: * egress-steering.yaml — `interface: tun0` is resolved ONCE and cached, and the gateway node is what tells Cilium it changed (the egress-uplink node label). A future reader's instinct will be to have the node write to that object; the comment says not to, because it is Flux-managed, and points at the label and `cilium-dbg bpf egress list` as the debugging entry points instead. * egress-steering.yaml — why the new NFS media mounts are not affected by pod-scoped steering: an NFS PV is mounted by the node kernel in the host netns, so podSelector never matches it. The 192.168.0.0/16 exclusion covers the server too, but that is the second reason, not the first. * vpn-killswitch test — its recovery step depends on the re-steer path, and could not have passed before it existed. Its no-leak assertion was always right; only the explanation was wrong. Also states the prerequisite plainly: the test cannot run at all without a labelled gateway node. * platform README — prod is live on k3s-prod-3 with the before/after egress IPs, and the note that prod's vpn-class pods had NO internet beforehand (CEGP with no matching node -> `Gateway IP: Not Found` -> dropped), so arming the gateway restored egress rather than moving it off the WAN. * role README — the standing "add the shared fwmark to the interface contract" escalation is closed as not implementable on Cilium OSS, rather than left open for someone to attempt. DECISIONS-LOG carries the full session entry, including the drafted ADR-0015 / docs-03 amendment for the architect (specs are immutable to agents) and the five things left for the operator — chief among them that a silent gateway outage still has no alert, which is what let the dev gateway sit dead for 80 minutes. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 17:57:52 -05:00
jakman85andClaude Opus 5	60e3f09c67	feat(vpn): prod egress-gateway group — k3s-prod-3, singleton, opt-in only `[vpn_egress_gateway_prod]` is a SEPARATE inventory group with its own vaulted credentials, not a second member of `[vpn_egress_gateway]`. That group is the default target of playbooks/vpn.egress.gateway.yml, so adding a prod host to it would mean a flagless `ansible-playbook playbooks/vpn.egress.gateway.yml` silently reconfigures a prod node's kill-switch datapath. Prod is reached only by asking for it, with `--list-hosts` first: -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3 Node 3 because node 1's SSD was replaced yesterday (least operational history) and node 2 carries the §0b memattr history. Exactly one node, forever: Cilium OSS picks "the first node in lexical ordering" among nodeSelector matches, so a labelled k3s-prod-1 would silently steal the gateway role — and the public egress IP — from k3s-prod-3, with no event and no alert. vault.yml is a verbatim copy of the dev group's CIPHERTEXT: same PIA account per the operator, and copying the encrypted blob means the plaintext was never decrypted to disk, through a shell, or into a transcript. The two files are now independent — rotating means re-encrypting both, which is the right behaviour for two environments. Also corrects a stale inventory note that claimed k3s-prod-3 has wlp1s0 up on 192.168.2.195. Re-verified before choosing it as the gateway, because a second live uplink in a kill-switch datapath is not a footnote: there is no wl* netdev on the node at all, no phy80211 device, the /etc/network/interfaces wlp1s0 stanza is commented out, 192.168.2.195 is assigned nowhere, and the only IPv4 addresses are lo and eno1 192.168.2.12. Nothing needed bringing down. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 17:38:46 -05:00
jakman85andClaude Opus 5	6f698f59af	fix(vpn): Cilium never re-resolves tun0 — the gateway dies on every reconnect Measured on k3s-dev-3, not reasoned about. Three findings, one of which had the dev gateway silently dead for ~80 minutes before this change: 1. Cilium caches (egress IP, egress ifindex) when it first resolves the CEGP's `interface: tun0` and does NOT re-resolve when tun0 changes. Every tunnel restart re-creates tun0 with a NEW ifindex, so vpn-class egress is blackholed from then on. Observed: node had tun0 at ifindex 1987 while `cilium-dbg bpf egress list` still said 1986; pod curl timed out. Forcing a re-apply moved the map to 1987 and egress recovered instantly. `egress-gateway-reconciliation- trigger-interval: 1s` does NOT cover netlink link events — dev has that same 1s setting and still did not self-heal, 20s+ after recovery. Fix: vpn-egress-resteer.sh publishes "<tun0 addr>-<ifindex>" as a node label. A node update makes the gateway re-process the policy (verified: cached ifindex 1986 -> 1987 within 10s). A node label, not an annotation on the policy, because the policy is Flux-managed GitOps and the node's labels are already this role's territory. Fired by the OpenVPN up-script on every connect, plus a 2-minute convergence timer for the boot case where the kube-API is not up yet. Proof: `systemctl stop` then `start` of the tunnel now recovers on its own (map 1989 -> 1990, PIA IP returns); before this commit the identical sequence stayed broken indefinitely. 2. The nft kill-switch CANNOT see the traffic it claims to guard. A counters-only diagnostic table on the gateway node recorded, while a vpn-class pod egressed: 0 packets in forward and 0 in postrouting for both the pod CIDR and the PIA range, against 45166 packets on the same postrouting hook for WAN traffic and 16 packets for a node-originated curl out tun0. Cilium's eBPF datapath bpf_redirect()s these skbs to the egress device without invoking netfilter. So `ip saddr @set oifname != tun0 drop` can never fire for vpn-class pods. What actually fails closed is Cilium's own drop when the cached egress interface is unusable — tcpdump during a tunnel stop AND during a restart showed 0 packets leaving either the WAN or tun0. Every rule now carries a `counter` so the claim stays falsifiable, and every comment that called this table the authoritative vpn-class backstop is corrected rather than left to mislead the next reader. Also removed the `meta mark 0x1c8 oifname != tun0 drop` rule and its variable: nothing stamps that mark (the only Cilium mark on the node is 0x200/0xf00, the L7-proxy rule to table 2004), the traffic never reaches netfilter anyway, and a live rule matching an arbitrary mark is a hazard to whatever picks it next. 3. The up-script flushed the nft source set on every connect, so a packet still sourced from the previous tunnel address matched neither the ip-rule nor the drop rule. The set is now additive (`add element` is idempotent; the down-script already retained it): an element can only ever tighten the guard. Finding 2 means that window could not have leaked POD traffic, but node-originated tunnel traffic does traverse netfilter, and additive is free. Kill-switch reloads no longer disarm the guard either: the .nft file uses `table` + `flush table`, which clears rules but leaves the named set's elements intact, so an Ansible run cannot empty the set mid-flight. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 17:38:27 -05:00
jakman85	839f7fed4a	merge: velero works against R2 — plugin pin, NFS volume policy, metrics gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details	2026-09-12 17:20:55 -05:00
jakman85andClaude Opus 5	2d5171e064	feat(velero): scrape the backup controller — three critical alerts could never fire platform/monitoring/alerts.yaml has shipped VeleroBackupFailure (critical), VeleroBackupPartialFailure and VeleroNoRecentBackup since the observability wave. None of them could ever have fired. The velero chart defaults `metrics.serviceMonitor.enabled: false`, and prod confirms it live: `kubectl -n velero get servicemonitor` returns no resources, so velero_backup_failure_total and velero_backup_last_successful_timestamp were never scraped. A nightly backup could fail — or silently stop running — for weeks with nothing to say so. That is exactly what R-STATE-03 is for, which makes the scrape part of the backup path rather than an observability extra. Both flags verified against chart 12.1.0: `helm template` emits neither object without the monitoring.coreos.com/v1 capability, and emits both PodMonitor and ServiceMonitor with it — so the chart gates on capability detection AND `enabled`, and helm-controller supplies real capabilities from the API server. No additionalLabels are needed: platform/monitoring already sets serviceMonitorSelectorNilUsesHelmValues and podMonitorSelectorNilUsesHelmValues to false, so Prometheus selects every monitor regardless of labels. Independent of the R2/bulk-tier fix in the previous commit; droppable on its own if the PM wants to sequence it separately. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 17:18:26 -05:00
jakman85andClaude Opus 5	2e259a808f	fix(velero): prod backups could not write metadata to R2, and opt-out FSB would have shipped 8.6 TiB Velero was resumed on prod (live `flux resume`; suspension was never in git) and its BackupStorageLocation validated Available against Cloudflare R2 on the first try — bucket k3s-prod-backup, prefix `velero`, region auto, checksumAlgorithm "". Backups still failed, and a second, unrelated hazard was armed. Both are fixed here; both were measured on prod, not reasoned about. 1. velero-plugin-for-aws v1.14.2 cannot write to R2 at all. It sets PutObject's `Tagging` field unconditionally, so the SDK emits an empty `x-amz-tagging` header on every PUT. R2 does not implement object tagging in any form and answers HTTP 501 NotImplemented. Two separate Backups reproduced it on the write of velero/backups/<name>/velero-backup.json: kopia uploaded the volume data fine (it uses its own S3 client), then the backup went phase=Failed with no metadata object — i.e. unrestorable. No config avoids it; `config.tagging` with a real value sends a tag set R2 also rejects. Upstream PR #309 makes the field conditional and landed on release-1.14 after v1.14.2 was cut, so v1.14.3-rc.2 is the earliest build with the fix. Pinned to it, with the exit condition (v1.14.3 GA) stated at the pin. 2. The opt-out FSB schedule would have uploaded the NFS bulk tier. Schedule (1) runs defaultVolumesToFsBackup:true and excludes only monitoring and unifi. Since 2026-09-11 prod also carries four static NFS PVs — plex-media, sonarr-media and radarr-media all bound to the SAME export /mnt/Storage/plex (8.32 TiB measured via zfs list), plus nextcloud-legacy-archive at 270 GiB. The 03:00 run would have read that library three times over NFS and pushed ~8.6 TiB to R2 nightly, retained 720h. Namespace exclusion cannot express the fix: those four namespaces also hold real Longhorn PVCs (plex/sonarr/radarr config, nextcloud-data) whose loss is unrecoverable. So the opt-out is now per VOLUME, by SOURCE TYPE: a velero volume policy (ConfigMap velero-volume-policies) skips any volume whose source is NFS, and schedule (1) references it. That tests the property that makes a volume bulk-tier (R-STATE-04: the bulk tier IS the NFS export) instead of a name list the next PV would slip past. The per-pod backup-volumes-excludes annotation was rejected for failing OPEN — a new NFS mount would be backed up in full until someone remembered. PROVEN on prod (velero 1.18.1 / chart 12.1.0): a Backup of ns nextcloud with defaultVolumesToFsBackup:true plus this policy produced exactly ONE PodVolumeBackup — `data` (nextcloud-data, Longhorn, 875,417,102 bytes, Completed). The NFS `legacy-archive` volume produced none. Residual gap, stated rather than hidden: the rule matches NFS only. A future bulk volume on another out-of-cluster source needs its own condition. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 17:18:26 -05:00
jakman85	3ec67bafdd	merge: platform-admins may reach the media app UIs gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details	2026-09-12 17:15:14 -05:00
jakman85andClaude Opus 5	5da116b34d	fix(oauth2-proxy): platform-admins may reach the media UIs — the immediate 403 The four forward-auth media UIs (jackett, qbittorrent, radarr, sonarr) gate on `--allowed-group=media-users` alone. On prod the `homelab` realm has exactly one user — the operator's `jakman85` — and it is a member of `platform-admins` ONLY. `media-users` has zero members, so those four hosts were reachable by nobody: oauth2-proxy completed OIDC discovery, redirected to Keycloak, got a token whose `groups` claim is `["platform-admins"]`, found no allowed group in it, and answered 403. No login form appeared because the browser already held a Keycloak SSO session from a platform-admins login — hence "403 immediately". pihole and crafty were unaffected: they already gate on platform-admins. `--allowed-group` is repeatable and OR-ed, so listing both groups means "media consumers OR platform operators". This grants no real privilege — platform-admins already maps to cluster-admin via k8s-oidc-rbac.yaml (oidc:platform-admins), so a member can already exec into these pods — and it keeps media-users meaningful for non-admin household accounts. Ruled out while diagnosing: client-secret drift (all 7 sidecar secrets hash-match keycloak-realm-clients), stale apex URLs (every live sidecar carries keycloak.prod.home.keithjackson.dev), missing group mappers (all 7 clients emit claim=groups, full.path=false), and the central oauth2-proxy pod restarts — those were Keycloak returning 503 "no healthy upstream" to OIDC discovery at startup, a consequence of the Keycloak roll, not a cause. Goldens re-rendered with the canonical command: python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation R-IDP-06 Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 17:12:05 -05:00
jakman85andClaude Opus 5	6b953cc56b	fix(arrs): the config PVC was mounted where the app never looks gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details sonarr, radarr, jackett and qbittorrent all mounted their Longhorn config PVC at the renderer's default /data/config, but every one of these images runs with `-data=/config`. So the PVC held nothing but lost+found while the app wrote its entire configuration to the container's EPHEMERAL writable layer — destroyed on every pod recreation: restart, reschedule, image bump. Measured in-cluster before fixing, per app, rather than inferred: /data/config dev=2080 [lost+found] <- the Longhorn PVC /config dev=2053 [Sentry asp config.xml logs ...] <- the node's own disk and a marker file written to /config did not appear in /data/config, so they are demonstrably different storage. qbittorrent's /config sat on dev=66309 (/dev/nvme0n1p5), i.e. whichever node it happened to land on. This is the SAME failure catalog/plex.yaml:54 already records having fixed for Plex; plex was the only one of the five that was right. The live /config contents were copied onto each PVC BEFORE this landed, so jackett's API key and qbittorrent's credentials — which other apps reference — survive the move rather than being regenerated. catalog/sonarr.yaml:61 documented this as a latent bug and deliberately left it because it changes live on-disk state. It is being fixed now because all four pods had just been recreated, so their configs were freshly generated and nothing had accumulated to lose. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 16:35:18 -05:00
jakman85	f5e62d9169	merge: bulk NFS volumes for plex, sonarr, radarr and nextcloud gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details	2026-09-12 16:26:31 -05:00
jakman85andClaude Opus 5	daac05e7e8	feat(bulk): prod mounts the NFS export through pfSense's WAN address The operator added the pfSense port-forward, so BULK_NFS_SERVER is no longer a placeholder. Prod gets 192.168.2.102 (pfSense WAN, DNAT tcp/2049 -> 192.168.1.9); dev stays on the base 192.168.1.9 because it is already on pfSense's LAN. This value could not be left as a placeholder safely: spec.nfs.server is unvalidated by Kubernetes, so the PV would be created, the PVC would BIND, and the failure would only surface as four pods stuck in ContainerCreating. Verified from k3s-prod-1 before committing, deliberately as uid 1000 rather than root (the export carries no_root_squash, so a root write proves nothing, and LSIO images default to PUID 911 which would go Ready and then fail every import EACCES): mount negotiated nfs4 vers=4.1 proto=tcp, write succeeded, and ln() across downloads/ -> tv/ gave link count 2 — real hardlinks, so no silent copy-then-delete. The Nextcloud export was additionally tightened to server-side ro= for prod's three nodes; a write as root is refused. Its content is config/, config_old/ and config_backup/ with no data/ directory, so it is an archive, not a live data store. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 16:25:20 -05:00
jakman85andClaude Opus 5	da8a0eac04	test(l0): R-STATE-04 gets a real check — bulk storage provably is not the replicated tier The gate BLOCKED this branch, correctly, and the block was the interesting part. Attaching the media library selects R-STATE-04 as a touched requirement ("bulk media storage has a capacity strategy distinct from the replicated tier"), and R-STATE-04 had NO scaffolded test, so the coverage REQUIRE stage failed both l0 and l1. There were two ways out. One was to keep avoiding the token — the existing discipline in catalog/plex.yaml and catalog/qbittorrent.yaml, which say in as many words that naming the requirement would block the merge. That made sense while the tier was a stub. It does not make sense in the change that DELIVERS the tier: it would mean shipping the requirement and claiming nothing. So: the check. WHAT IT ASSERTS, and why each one is a real failure mode rather than a shape The `bulk` StorageClass sat as `server: REPLACE-ME` for six roadmap phases, and the only thing keeping a multi-terabyte library off the 3-replica `longhorn` class was four hand-written comments. That is the class of failure being closed: a bulk volume on the replicated tier does not fail loudly — Longhorn accepts the PVC and starts replicating 3x of data three mini-PCs cannot hold. (1) DISTINCTNESS — no bulk PV/PVC on any class the repo declares with the Longhorn provisioner (derived from platform/longhorn/, so adding a Longhorn class cannot create a blind spot), and the class is explicitly EMPTY. (2) CAPACITY STRATEGY as arithmetic — a bulk volume at least an order of magnitude above the same service's replicated volume. The requirement is about two magnitudes of data; a "bulk" volume sized like a config PVC means the tier was named, not applied. The 10x floor is CALIBRATED against the real corpus, not guessed: the media library is ~460x its config volume and the Nextcloud archive ~13x its 20Gi, so a higher floor would fire on a legitimate volume and teach the next reader to weaken the check. My first draft used 100x and did exactly that — the corpus caught it. (3) PRE-BOUND FROM BOTH ENDS (claimRef + volumeName, agreeing). Guards the most deceptive failure available in this tier: a dynamic bulk class hands out a FRESH EMPTY directory, the pod mounts, the PVC Binds, everything is green, and the 8 TiB was never attached. (4) Retain, always. (5) mountOptions carry `hard` and exactly one nfsvers=4.x, and never `soft` — a soft mount hands a server hiccup to the app as a short read, which for a media library is silent corruption; and v4 is what keeps the firewall surface at one port. (6) ONE EXPORT, AT MOST ONE MOUNT PER WORKLOAD. This is the assertion I most wanted. Splitting one export into two subPath mounts looks TIGHTER and breaks link() with EXDEV; Sonarr and Radarr then degrade to copy-then-delete, rewriting every import over NFS and destroying the seeding copy, surfacing weeks later as "imports are slow and my ratio is gone". No reviewer sees that in a diff. (7) READ-ONLY COUPLING — nfs.readOnly, the volumeMount flag and ReadOnlyMany must agree, so dropping one still leaves the others; and every bulk mode is a `...Many`. (8) The server is a BARE cluster-vars token, that knob has a base default, and EVERY overlay supplies it — the seam that lets one authored definition serve two clusters that reach this host by different addresses. (9) LOAD-BEARING and NON-VACUOUS. Wired unconditionally into ci/l0.sh as Stage 3e for the same reason as 3c/3d: Stage 3b scopes to touched requirements, and the path map keys NEITHER apps/* NOR platform/longhorn/* to this requirement — so the change most likely to break this could attribute nothing and be gated by nothing. An EMPTY bulk corpus FAILS rather than passing trivially, and a floor on the assertion count keeps a pass from degenerating. It reads the COMMITTED apps/ goldens rather than calling the renderer. Asking the renderer would only prove it agrees with itself; the goldens are what Flux reconciles, and render==golden is already its own gate stage. PERTURBED, because a check that has never failed is not evidence. Each of these was applied to the real tree and each fired with the intended message: bulk PVC moved onto `longhorn`; Retain -> Delete; `hard` -> `soft`; the token replaced by the literal 192.168.1.9; sonarr's single mount split into two subPath mounts (assertion 6); and the prod overlay's knob deleted. Also hardens render.py: `size` is structurally required of a bulk entry (a PV with no capacity is rejected by the API server) but the JSON Schema only enforces `export`, so omitting it raised a bare KeyError three frames down. It now raises a sentence naming the service, the volume and the remedy. Found by the same perturbation pass. Coverage: 48/51 -> 49/51 (96.1%), 2 gaps, 0 safety-critical. BASE_REF=main make -C ci gate -> MERGE GATE: PASS. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-08 00:03:34 -05:00
jakman85	cac413e87a	feat(cnpg): synchronous replication, quorum of 1, dataDurability preferred gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details Three instances replicating ASYNCHRONOUSLY still lose data on failover: COMMIT returns as soon as the primary has flushed locally, so a primary that dies before its standbys catch up takes the last few transactions with it and the promoted standby comes up quietly missing them. On keycloak-db that is a just-created user, a rotated credential or a consent vanishing at exactly the moment of a failover. `method: any` + `number: 1` makes COMMIT wait for ONE of the two standbys to confirm — the point of having them. dataDurability: preferred, NEVER required. `required` is a strict quorum: fewer than `number` healthy standbys and writes BLOCK. Postgres here shares three nodes with etcd and the CNI, so one degraded standby — a node reboot, a Longhorn replica rebuild, a Cilium restart — would turn a partial degradation into a total write outage, and on the SSO database that takes kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth with it. `preferred` keeps synchronous commit while a healthy standby is there and relaxes the required count dynamically when one is not. dataDurability is only honoured while standbyNamesPre/standbyNamesPost are unset — both are. NON-DISRUPTIVE: this compiles to `synchronous_standby_names`, a reload-only PostgreSQL parameter. No pod restart, no switchover, no client disconnect. It is the last of the three commits because it wants standbys that already exist and are already on their final nodes. Field shape verified against the live clusters.postgresql.cnpg.io CRD before relying on it (CNPG 1.30.0, byte-identical schema on k3s-prod and k3s-dev): spec.postgresql.synchronous requires `method` (enum any\|first) and `number` (x-kubernetes-validation self > 0); `dataDurability` is enum required\| preferred. None of the three Clusters previously had a `postgresql:` block, so this adds one rather than merging into existing parameters. DEV IMPACT: identical and non-disruptive, same reload-only path. Verified: yamllint clean; both tiers render the block. Co-Authored-By: Claude Opus 5 <[email protected]> (cherry picked from commit f1ce19461415e77eb2f1ca5109ebab49edf0860b)	2026-09-07 23:54:19 -05:00
jakman85andClaude Opus 5	82aff4d053	feat(apps): attach the media library and the legacy Nextcloud dataset over NFS plex, sonarr, radarr and nextcloud each gain one bulk volume. Authored in the catalog and re-rendered, never hand-edited into apps/ (render-golden: 10/10). Layout, the export change the operator must apply, the uid analysis and the validation probes are in docs/BULK-NFS-STORAGE.md; everything there was measured on the host over read-only SSH, not assumed. WHAT THE LAYOUT TURNED OUT TO BE, AND WHY IT DECIDED THE DESIGN /mnt/Storage/plex holds movies/ tv/ photos/ AND downloads/ as SIBLINGS in one dataset — so the arrs' import source and their libraries are one filesystem on the server, and the download client's completed folders are already named radarr/ and tv-sonarr/. The k3s arrs are joining a tree an arr stack has been driving for a long time. That sibling fact is the whole argument for the arrs mounting the export ROOT rather than a subPath per directory. Sonarr and Radarr import by hardlinking or atomically renaming, both same-filesystem operations, and to the kernel "same filesystem" means SAME MOUNT. Two subPath mounts of one export are two mounts: link() across them returns EXDEV regardless of what the server knows, and the arrs do not fail loudly — they degrade to copy-then-delete, rewriting every file over NFS and destroying the seeding copy. The smallest subtree containing both tv/ and downloads/ IS the export root, so the root mount is the MINIMAL mount that preserves hardlinks. Least privilege bounded below by a hard requirement. The cost is stated, not hidden: sonarr can also write /media/movies. plex /mnt/Storage/plex -> /media ro ReadOnlyMany sonarr /mnt/Storage/plex -> /media rw ReadWriteMany radarr /mnt/Storage/plex -> /media rw ReadWriteMany nextcloud /mnt/Storage/nextcloud -> /mnt/legacy-nextcloud ro ReadOnlyMany (subPath: nextcloud) Plex is read-only because it indexes and streams; every write on that tree belongs to the arrs, and saying so removes Plex from the set of things that can damage 8.33 T. All three media consumers use the SAME mountPath deliberately: when qbittorrent is eventually given this export, the arrs and the download client will already agree on absolute paths, which is the other precondition for hardlinked imports. * /media AND NOT /data — A COLLISION CAUGHT WHILE RENDERING. * The arrs' Longhorn config PVC renders at the renderer default /data/config. A media mount at /data is mounted OVER it and MASKS the config volume: the pod comes up with its app database shadowed by an 8 T media tree. That discovery also exposed a SEPARATE PRE-EXISTING BUG, reported and deliberately NOT fixed here: LinuxServer arr images run with -data=/config, so /data/config is very likely the wrong home for that PVC and the app has been writing its EPHEMERAL layer — the exact failure catalog/plex.yaml's mountPath comment records already having been bitten by. Plex was fixed; sonarr, radarr, qbittorrent and jackett were not. Fixing it starts those apps from an empty config volume, so it needs its own change with the operator present. This change does not make it worse. UID/GID — THE FAILURE THIS TIER ALWAYS HAS The export is sec=sys + no_root_squash, so the client's numeric uid passes through verbatim and the server's ownership is the entire access story. The tree is 0755 owned 1000:3000, so ONLY uid 1000 can write it. LinuxServer images default to 911. Without intervention sonarr and radarr would have mounted cleanly, gone Ready, and failed EVERY import with EACCES. And fsGroup cannot fix that — the in-tree NFS plugin does NO ownership management, so podSecurityContext.fsGroup is a no-op on an NFS volume. Reaching for it is how this gets misdiagnosed. So PUID=1000/PGID=3000 are added to both arrs (PGID 3000 keeps what they write group-readable to the live non-k3s consumers of the same export), and fsGroup: 3000 is added for the LONGHORN config PVC only, so changing PUID does not depend on the image re-chowning /config. plex already runs as 1000 and reads a 0755 tree; nextcloud reads as www-data. Neither needs work. NEXTCLOUD IS MOUNTED READ-ONLY, WHICH IS A DEVIATION FROM THE BRIEF — ON EVIDENCE The task said nextcloud writes. The dataset says otherwise: of 270 G, 267 G is config_backup/, the rest is config/ (103 M) and config_old/ (3.4 G). There is NO data/ directory. This is the OLD non-k3s instance's configuration plus a backup blob — an archive — and that instance is STILL LIVE on it (exported rw to 192.168.1.6 and .5, which the operator has said must stay active). Two Nextclouds writing one instance's config directory, each with its own database and therefore its own oc_filecache, is unrecoverable corruption. "Do not grant more than needed" lands on ro, and ro is the trivially reversible choice. If the intent is in fact a MIGRATION, that is a larger and more deliberate change than a mount; the one-line flip is documented. Read-only for nextcloud can additionally be enforced SERVER-side (only nextcloud consumes that dataset from k3s), and the sharenfs value in the doc uses ro= for the k3s clients. The media tree cannot: plex and the arrs may land on any node, so the server sees the same addresses for reader and writers. Plex's ro is therefore a client-side guarantee — stated as a limitation, not left implied. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-07 23:54:13 -05:00
jakman85	e8d8601ca6	feat(cnpg): required pod anti-affinity + switchover primary rolls gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details `instances: 3` alone does not keep the three instances apart. All three Clusters used CNPG's default `podAntiAffinityType: preferred`, which is a scheduler HINT — two instances may share a node whenever that scores better, and they WILL on the path that matters: when CNPG recreates an instance after a node loss, the surviving nodes are the only candidates and one of them already runs an instance. `preferred` is how a 3-instance cluster decays back into the single-node failure domain the previous commit exists to remove. `required` makes it a scheduling constraint. On a 3-node cluster with no spare, the cost is that an instance whose node is gone stays Pending until a node returns. That is correct: a replacement squeezed onto an already-occupied node dies with that node while making the cluster LOOK three-way replicated. 2 of 3 instances still serve and CNPG still promotes. ALSO SETS primaryUpdateMethod: switchover. This is not cosmetic. Changing the affinity changes the instance pod spec, so CNPG performs a rolling update, and the CRD default for primaryUpdateMethod is `restart` (verified against the live CNPG 1.30.0 CRD on both clusters) — a delete-and-recreate of the PRIMARY pod, i.e. tens of seconds of no Postgres. On keycloak-db that is an SSO outage, which also means kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth. With standbys now present, `switchover` promotes an already-updated streaming replica instead, reducing the roll to a connection reset. Known constraint documented in the manifest: under `switchover` the operator rejects an update that changes imageName and a postgresql parameter in the same reconcile. Verified satisfiable before merge (read-only): k3s-prod and k3s-dev each have 3 Ready, schedulable, untainted nodes, and every pgdata PVC already has a Longhorn replica on all three (replica-soft-anti-affinity=false), so any instance can attach from any node. DEV IMPACT: identical, and dev can absorb it. Longhorn dev nodes are 124.9 GiB each with 152.0 GiB already scheduled under a 200% over-provisioning ceiling (249.8 GiB); the six new 5Gi volumes add 30 GiB scheduled per node -> 182.0, still inside the ceiling, and only ~600 MiB per volume is really consumed. Prod is far looser: 239 -> 269 GiB scheduled against a 829 GiB ceiling. THIS IS THE DISRUPTIVE COMMIT of the three. Land it on its own reconcile, after the scale-out has produced two Ready standbys per Cluster, so the blast radius is one clean switchover per database rather than a scale-out and a roll at once. Expect a brief SSO interruption for keycloak-db. Verified: yamllint clean; both tiers render the new fields; field names, types and enum values checked against the live clusters.postgresql.cnpg.io CRD (CNPG 1.30.0, identical on prod and dev). Co-Authored-By: Claude Opus 5 <[email protected]> (cherry picked from commit e1d364238d73e6ee99c9a50108fe8b4e09481a7e)	2026-09-07 23:48:32 -05:00
jakman85andClaude Opus 5	2590bc4317	feat(env): BULK_NFS_SERVER — the one knob prod cannot answer yet, and why it is a placeholder The bulk tier's PersistentVolumes need a server address, and there is no single correct one: the two clusters sit on OPPOSITE SIDES of pfSense from the storage host, which has exactly one interface (vmbr0 192.168.1.9/24, on pfSense's LAN). dev 192.168.1.40/.41/.42 — the SAME segment. Reaches it directly, and already holds one NFS export from it (/mnt/Storage/k8s-volumes, in the host's /etc/exports). Proven path, so base's default is the dev value per this file's existing convention, and dev restates it. prod 192.168.2.10/.11/.12 — pfSense's WAN. tcp/2049 does not traverse that today; 22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. Opening the storage path is a PENDING operator decision whose two candidate outcomes give DIFFERENT values here: 192.168.2.102 (another port-forward) or 192.168.1.9 (a route). So prod holds TBD-PROD-BULK-NFS-SERVER, this repo's established loud-placeholder convention. Two findings recorded in the prod overlay because they change what the operator has to do: * THE EXPORT'S CLIENT LIST IS THE SAME UNDER BOTH OUTCOMES. Measured today from this workstation (192.168.2.132) through the existing port-forward: the storage host logged the connection as coming from 192.168.2.132 — a WAN-side address — not from a pfSense LAN address (`ss -tn` peer 192.168.2.132; the host's only route off-LAN is its default via 192.168.1.1). pfSense DNATs WITHOUT source NAT, so the NFS server sees prod's REAL node addresses either way. The `zfs set sharenfs=` work therefore does not have to wait for the routing decision. * ONE PORT, NOT FOUR. The PVs mount nfsvers=4.1, and NFSv4 carries mount and lock in-protocol — no rpcbind/111, mountd or statd. Whichever outcome is chosen, it opens tcp/2049 and nothing else. NFSv3 would have needed four ports, three of them on ephemeral numbers. * AND ONE HONEST WEAKNESS, stated in the overlay where an operator will hit it. * Unlike the LAN knobs above it, this placeholder does NOT fail at apply time. A PV's spec.nfs.server is an unvalidated string: the API server accepts it, the PV is created, and the PVC BINDS (binding is pure control-plane). The failure appears only when a pod mounts — MountVolume.SetUp "Failed to resolve server TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr and nextcloud would sit in ContainerCreating on prod. There is no CRD validation to lean on here, so the guard has to be procedural: set the value before prod reconciles a revision carrying the bulk volumes. Dev renders a working value and is where these manifests are meant to be proven. Related finding, NOT fixed here because the file does not exist on main: the static guard this convention is supposed to lean on, `bootstrap/standup.sh`'s `s3_is_placeholder` family, is absent from main (it lives on unmerged prod branches), so conformance/tests/l0-static/placeholder-guard-coupling exits 3 SKIPPED and nothing in the gate currently refuses a placeholder-shaped overlay value. When standup.sh lands, its guard needs an arm for this knob and a consumer that calls it — assertions (2) and (3) of that check then cover this value automatically, since the detector's SEED vocabulary already matches "TBD". Evidence: `kubectl kustomize clusters/dev` -> BULK_NFS_SERVER: 192.168.1.9; `kubectl kustomize clusters/prod` -> BULK_NFS_SERVER: TBD-PROD-BULK-NFS-SERVER. Both exit 0. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-07 23:47:39 -05:00
jakman85andClaude Opus 5	cec550045b	feat(storage): the bulk tier becomes real — static NFS PVs, authored once per environment The `bulk` class has existed since Phase 0 as a deliberate stub: the StorageClass named an NFS CSI driver that is not installed and a server/share of REPLACE-ME, so plex, sonarr, radarr and qbittorrent all carry a comment saying "attach the library once OQ-2 is resolved". This is that resolution — for the case that actually exists here, which is NOT the case the stub anticipated. The stub assumed DYNAMIC provisioning: ask a CSI driver for a bulk volume and get a fresh empty subdirectory. That is the wrong shape for this platform's bulk data. The media library is 8.33 TiB that already exists, on ZFS datasets on the Proxmox host, with live non-k3s consumers. Nothing needs provisioning; existing datasets need ATTACHING. So a bulk storage entry now renders a STATIC, pre-bound NFS PersistentVolume plus its PVC, and needs no CSI driver at all (the in-tree NFS volume source is not deprecated and has no CSI migration). Schema 1.4.0 adds, on a storage entry: export absolute NFS export path; REQUIRED when class is bulk, FORBIDDEN otherwise (an `if` in the schema, so a bulk entry cannot be authored without saying what it attaches). subPath mount a subtree instead of the volume root — any class. readOnly any class; on bulk it also sets the PV's nfs.readOnly, so the mount really is `ro`. The SERVER is deliberately NOT a schema field. It is the `${BULK_NFS_SERVER}` cluster-vars knob, because dev and prod reach the SAME storage host by different addresses and R-ENV-02 says the service definition is authored once. A ServiceDefinition says WHICH dataset; the overlay says where the host is. Three decisions worth the reader's time, argued in full in the r_bulk_pvs docstring: * PV/PVC, not an inline `nfs:` pod volume. Inline is fewer objects and would work. It cannot express `persistentVolumeReclaimPolicy: Retain`, which is the one property most worth having in git for 8.33 TiB of data with consumers outside this cluster; it puts read-only enforcement one Deployment edit away from silent removal; and it is invisible to `kubectl get pv,pvc`, which is where an operator looks for attached external storage. * storageClassName: "" on both ends, NOT the `bulk` StorageClass. Explicitly empty is the canonical marker for a statically provisioned volume. It also sidesteps a trap: a StorageClass's `provisioner` is IMMUTABLE, so re-pointing the deployed `bulk` stub would make Flux fail on that object forever. * Access mode is always `…Many` and follows the read/write posture — ReadOnlyMany for a reader, ReadWriteMany for a writer. One export is consumed by several namespaces AND several nodes at once, which is exactly what ReadWriteOnce cannot express; which `Many` it is then makes the intent legible in `kubectl get pv`. mountOptions are a platform decision held in the renderer rather than a per-service field: nfsvers=4.1,hard,noatime,timeo=600,retrans=2. NFSv4 carries mount and lock in-protocol, so the only port that must be reachable is tcp/2049 — no rpcbind/111, mountd or statd. That is a firewall decision as much as a storage one. `hard` over `soft` because a soft mount converts a server hiccup into a short read handed to the application. The contract doc records the trap that will otherwise be discovered in production: fsGroup is a NO-OP on an NFS volume (the in-tree plugin does no ownership management), so a writable bulk mount requires the container's own uid/gid to suit the export's ownership. A mount that succeeds and then cannot write is this tier's characteristic failure. Every pre-existing golden renders byte-identical: the new keys are emitted only when declared, and `readOnly` defaults to False. render-golden: 10/10. selftest: PASS. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-07 23:45:09 -05:00
jakman85	f338e91770	feat(cnpg): instances 3 on all three Clusters — a standby to fail over to gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details All three CNPG Clusters ran `instances: 1`. Longhorn's 3-way replication saves the DATA when a node dies; it does not save the SERVICE — a single instance has to be rescheduled, re-attach its volume and replay WAL before Postgres accepts a connection again. With nothing in the spec keeping them apart, all three primaries had drifted onto ONE node (k3s-prod-2 on prod, k3s-dev-3 on dev). On 2026-09-07 that node took SSO, Nextcloud and Semaphore down together, and with SSO went kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth. Three instances give each Cluster a streaming standby CNPG can promote in seconds, repointing the -rw Service with no volume re-attach and no restore. UNBLOCKED BY WORKING BACKUPS, NOT A SUBSTITUTE FOR THEM. Standbys replicate corruption and deletions faithfully; only the Barman archive is a backup, and adding standbys while archive_mode=on with a failing archive_command would have multiplied the disk-full clock, not hedged it. All three Clusters on both k3s-prod and k3s-dev now report ContinuousArchiving=True with a real firstRecoverabilityPoint (verified read-only, 2026-09-07). Scaling out is non-disruptive on its own: CNPG creates -2 then -3 sequentially, each seeded by pg_basebackup from the RUNNING primary. The primary is not restarted and clients are not disconnected. The anti-affinity change that DOES cause a switchover is deliberately the next commit. DEV IMPACT: identical — dev is on the same shape (three single-instance DBs, all on k3s-dev-3) and gets the same scale-out. Dev has three untainted nodes and Longhorn headroom for it (see the next commit's message for the numbers). Cost: 2 extra 5Gi PVCs per Cluster; ~30 GiB provisioned cluster-wide against live pgdata of ~600 MiB each, on Longhorn with 200% over-provisioning. Verified: yamllint clean; `kubectl kustomize --load-restrictor LoadRestrictionsNone platform-config` and `kubectl kustomize apps/wave-0` both render `instances: 3`. Co-Authored-By: Claude Opus 5 <[email protected]> (cherry picked from commit ed6fe6cb019b96f677a92448a0347f979d1a6d02)	2026-09-07 23:41:31 -05:00
jakman85	a624c638ef	merge: docs/node-1-drive-replaced gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details	2026-09-07 23:05:59 -05:00
jakman85	90b39de527	merge: fix/swap-off-persists-across-reboot	2026-09-07 23:05:59 -05:00
jakman85	0247d796a3	merge: fix/founder-moved-to-node-2	2026-09-07 23:05:59 -05:00
jakman85	c9e9dcf960	merge: fix/r-env-03-overlay-selected-credentials	2026-09-07 23:05:59 -05:00
jakman85	89cf59ffb7	docs(conformance): the GIVEN/WHEN/THEN header states the exception it now allows	2026-09-07 23:04:39 -05:00
jakman85	e319777849	docs(pm): draft the R-ENV-03 amendment, and name the gate hole that hid it The requirement's literal reading ("no prod-specific artifact in a shared path") forbids the environment-selected PAIR that R-ENV-02 prescribes for a value that must differ per environment while the manifests stay identical — and a Secret's difference cannot live in cluster-vars, because it is ciphertext in a SOPS file. Drafted wording that keeps the guard's teeth (an artifact only prod APPLIES still fails) and removes the false positive, for the System Architect: the spec repo is read-only here and CI-gated by R-TEST-06. Also recorded, checked rather than assumed: the merge that turned R-ENV-03 red was gated with a touched-requirement subset that never selected R-ENV-03, so ci/l0.sh Stage 3b filtered the already-red check out of the run that printed PASS. Replaying that delivery's twelve paths through fix/touched-requirements-fail-closed's requirement-paths.map still yields no R-ENV-03, so landing that branch would NOT have caught this. The two changes that would (unconditional whole-repo L0 stages; a changed conformance test attributing its own requirement labels) are ci/ work and are left to that branch, not smuggled in here.	2026-09-07 23:02:56 -05:00
jakman85	730d12ddc4	fix(conformance): R-ENV-03 shared-path rule tests the harm, not the filename The blanket regex around RENDERED_DIRS failed any tracked file whose name carries a delimited `prod` token. That caught the four per-environment backup credential halves (BACKUP_CREDENTIALS_SUFFIX) even though BOTH halves are applied to BOTH clusters as base resources and the overlay selects only which one the shared consumer REFERENCES — i.e. exactly the R-ENV-02 contract the regex exists to protect. Renaming the files to dodge the regex was rejected as gaming the guard, so the rule now encodes the property instead. The fix is not an allowlist of those four paths. A prod-token file in a shared path is excused only when the tree proves all of: (A) it declares nothing but Secrets whose names end in the environment suffix (1) its own directory's kustomization applies it, and EVERY environment value of the selector resolves to a Secret that some resource of that same kustomization declares, in the same namespace (2) nothing names the suffixed Secret — or the suffixed file — literally; every reference is `<base-name>${VAR}` (3) that VAR is defined for the base and for every overlay, and prod's value is the one that selects this file References are read from parsed YAML values, never raw text: the mechanism is spelled out by name in comments beside the resources, and a comment references nothing. No other assertion is weakened; the check still runs 96 assertions, and the pass output now notes each excused file and why.	2026-09-07 22:55:58 -05:00
jakman85andClaude Opus 5	df1d2ab44d	docs(prod): node 1's drive was replaced — and §2's exit condition still is not met Records the 2026-09-07 I/O-starvation incident and the drive swap: the C400 was replaced with a Samsung 850 EVO, and under the same load class (an active 18-replica Longhorn rebuild, with node 1 as the write target) I/O pressure fell from full avg300 11.36 to 3.79 — from ~4x worse than its peers into node 3's band. Updates the three §0 cells that had become false (disk model, drive health, kernel — all three nodes now run 7.1.8, not 6.12) and adds §0c rather than rewriting the 'NOT hardware-symmetric' analysis, which is still the correct reasoning about the state it described. Two things the rebuild taught, recorded so the next one is cheaper: Longhorn refuses a replaced disk with DiskFilesystemChanged and needs disable -> remove -> re-add in that order, and the inventory's [k3s_prod_primary] decides BOTH --cluster-init and k3s_api_host, so rebuilding the founder without moving that designation would have founded a second cluster. The replacement does NOT close §2: etcd and Longhorn still share one SATA spindle. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-07 22:50:45 -05:00
jakman85andClaude Opus 5	69a25cff82	fix(node_health): say 'active via <unit>', not a bare parenthetical The unit name in the swap-active detail read as an unlabelled aside. Naming it as the unit that activated the swap is what makes the line actionable. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-07 22:42:38 -05:00
jakman85andClaude Opus 5	c1e36d9353	docs(node_baseline): say why the Longhorn claim guard's swap list is now always empty The claim guard builds node_baseline_lh_protected_devices from ansible_mounts plus `swapon --show`. With swap.target masked, a baselined node has no active swap, so that half of the list contributes nothing — which reads like a guard that quietly stopped guarding. It is not: the claim candidate comes from whatever is mounted at node_baseline_longhorn_claim_mountpoint, and a swap partition is neither mounted nor ext4, so it could never be the candidate. Say so where the next reader will ask. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-07 22:39:39 -05:00
jakman85andClaude Opus 5	9d00a3fcd3	fix(node_health): memory.swap blamed the wrong thing, and could not see the defect The WARN read "roles/node_baseline runs swapoff and neutralises the fstab entry; this node has not been baselined yet". That was a MIS-DIAGNOSIS. All three prod nodes HAD been baselined, had their fstab swap line commented out by the role, and were still swapping 977 MiB each after a reboot. Anyone reading that WARN would have re-run the role that had already run, and watched swap come back next boot. The detail now names the actual mechanism — systemd-gpt-auto-generator re-creating a .swap unit from the partition TYPE GUID 0657fd6d-a4ab-43c4-84e5-0933c84b4f4f at every boot, without reading /etc/fstab — says in as many words that this is not "the node has not been baselined", and points at `node-baseline.yml --tags system`. The check was also blind in the other direction: SwapTotal == 0 is not evidence that swap is disabled, only that nothing has been swapped on yet this boot, so a node freshly swapoff-ed PASSed while still set to swap again on reboot — exactly the state the old role left behind. The collector now reports memory.swap_persistence (swap.target masked?, .swap units the generators emitted this boot, which of them are masked, leftover active fstab swap lines — all world-readable, no root, no systemctl parsing), and the check WARNs on "clean now, swapping after the next reboot". Rendered against every fact shape (swap active + unmasked, swap off + unmasked, after the fix, a dev VM with no swap partition at all, per-unit mask only, persistence unreadable, an older collector's output with the key absent, and /proc/meminfo unreadable) and verified live on k3s-prod-1 post-reboot: [PASS] memory.swap — no swap active and none can be activated: swap.target is masked Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-07 22:35:31 -05:00
jakman85andClaude Opus 5	0e92d258dd	fix(node_baseline): swap-off now survives a reboot — mask swap.target The role's `swapoff -a` + commented fstab line held only until the next boot, on every prod node. Measured 2026-09-07: all three came back from a reboot with 977 MiB of swap ACTIVE (k3s-prod-1 /dev/sda4, k3s-prod-3 /dev/sda4, k3s-prod-2 /dev/nvme0n1p4) while their fstab swap entry was commented out by this role. The role reported success and the setting silently expired. CAUSE. systemd-gpt-auto-generator claims any GPT partition typed 0657fd6d-a4ab-43c4-84e5-0933c84b4f4f ("Linux swap") from the PARTITION TABLE, synthesises a .swap unit into /run/systemd/generator.late/ and links it into swap.target.wants/. It never reads /etc/fstab, so neutralising fstab cannot stop it. Re-running the generator by hand on k3s-prod-1 says so verbatim: Probed fstype 'swap' on partition /dev/disk/by-diskseq/1-part4. Adding swap: /dev/disk/by-diskseq/1-part4 MECHANISM. Mask swap.target. `systemctl show` on the generated unit says its only inbound dependency is `WantedBy=swap.target` — Requires/RequiredBy/BoundBy empty, and the matching .device unit has an empty Wants=, so a device appearing does not pull its swap unit in by itself. swap.target is the single choke point for every activation path: gpt-auto, systemd-fstab-generator, systemd-cryptsetup-generator. It is device-shape independent (no device name appears in the change, so one task covers sda4, nvme0n1p4 and dev's vda VMs), boot-safe (swap.target is Wants=/After= from sysinit.target and After= from tmp.mount; RequiredBy= is EMPTY on all three prod and all three dev nodes, and a masked unit reached only via Wants= is skipped, not failed), and inert if the device is absent or renamed. The generated units are masked too, as a second independent barrier. Rejected: `systemd.gpt_auto=0` (disables GPT auto-discovery wholesale, needs a GRUB edit + reboot to become true, and on DEV would switch off root auto-discovery — dev's vda1 carries the discoverable GUID 4f68bce3-e8cd-4db1-96e7-fbcaf984b709); clearing the partition's GPT type GUID (a partition-table write to the live boot disk of a node serving etcd and Longhorn replicas, and an irreversible decision about that 977 MiB that belongs to the operator); `systemd.swap=0` (systemd-fstab-generator only — this swap does not come from fstab). `swapoff -a` is now conditional on `swapon --show` returning something, so the task reports honestly instead of hiding a command that ran anyway behind changed_when: false. PROVEN on k3s-prod-1 (all 18 Longhorn volumes healthy before the reboot): run 1 changed=2, run 2 changed=0 after `reboot`: /proc/swaps empty, `free` Swap 0B, swap.target masked, the generated unit still emitted by the generator and still masked, `systemctl --failed` empty, `systemctl is-system-running` = running. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-07 22:35:16 -05:00
jakman85andClaude Opus 5	08fbe85638	fix(ansible): move the founder designation to node 2 Node 1's SSD failed and was replaced; Debian was reinstalled, so node 1 is now a JOINING node with no cluster state. It was still listed in [k3s_prod_primary], and that group decides two things at once: 1. which play gets k3s_server_mode: primary -> --cluster-init, which FOUNDS A NEW CLUSTER, and 2. what k3s_api_host resolves to -- the --server join target. So re-provisioning node 1 unchanged would have pointed a rebuilt, empty node at ITSELF and founded a second cluster beside the live one. The role's `creates: /etc/rancher/k3s/k3s.yaml` guard does not help: on a fresh install that file is absent, so cluster-init WOULD have run. Node 2 is the senior surviving server (up since 2026-09-01) and takes the founder slot. Its own cluster-init task is inert precisely because k3s.yaml already exists there. Verified after the change: k3s_api_host renders 192.168.2.11, and groups[k3s_primary_group][0] is k3s-prod-2. This is a latent gap the role documented but did not handle -- its comment assumes the founder is always alive ('On a WARM single-node rejoin the founder is up and answering too'). Rebuilding the founder itself was not covered. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-07 21:41:54 -05:00
jakman85	863af01062	docs: the §2 shared-device measurement came back, and it does not hold gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details infra/PROD-PROVISIONING.md §2 recorded the etcd/Longhorn shared-partition risk as "accepted to be MEASURED, not assumed", with an exit condition. The measurement arrived on 2026-09-07 in production, 2 d 7 h after bring-up, as an outage. §2 now carries "THE MEASUREMENT CAME BACK": the journal evidence, the three-node IO PSI table (11.36 / 0.75 / 1.29), and the honest verdict that the section's own premise — "every prod node is SSD or NVMe, where rebuild I/O costs latency rather than collapsing it" — was right about node 2 and wrong about node 1. A 2011 C400 with 61,440 reallocated sectors and no power-loss protection is not the "SSD" that reasoning assumed, and §0 had already named the three-way drive asymmetry as the fault line. The exit condition is OPEN and the second bullet (dedicated device) applies; that remains the operator's. Also records what the three software mitigations in this branch do and do not buy, including the point that no cgroup or scheduler control arbitrates the ext4 journal both tenants share — and the gap that etcd's own fsync histograms, which the exit condition literally asks for, are not collected because kubeEtcd is disabled for k3s. PSI is the signal that exists. DECISIONS-LOG gains the full entry: the four I/O mechanisms found INERT on these nodes (IOSchedulingClass, IOSchedulingPriority, io.latency, bare IOWeight), the finding that concurrentReplicaRebuildPerNodeLimit was already set and already held during the outage, the third instance of config that looked applied and was not (orphanAutoDeletion), and the three things still requiring the operator. infra/ansible/README.md: CORRECTED a stale claim that k3s-prod-2/-3 "stay commented out" of the inventory. They have been live in [k3s_prod_secondary] since 2026-09-03. That sentence is exactly the kind used to reason about blast radius — `-e k3s_hosts=k3s_prod` targets THREE hosts, not one — so it is replaced with the --list-hosts command that settles it. The new etcd-io-priority play is added to the run order, the layer table and the prod command list. Co-Authored-By: Claude Opus 5 <[email protected]> (cherry picked from commit a00a27a2567ea6a805c8c196b75df2c92a7c63a4)	2026-09-07 02:46:26 -05:00
jakman85	f7a9229d81	fix(ansible): the NVMe node would have died on an undefined variable Self-review of the role I had just written. On k3s-prod-2 the scheduler path is deliberately skipped, so the `stat` on /sys/fs/cgroup/system.slice/k3s.service is skipped too — and a SKIPPED registration has no `.stat` key at all. Ansible templates `loop:` BEFORE it evaluates `when:`, so loop: "{{ ['system.slice'] + (['system.slice/k3s.service'] if _eio_k3s_cgroup.stat.exists else []) }}" raises on that host even though the task would never have run. The role would have failed on the one node it is designed to leave alone — and it would have failed at the very end, after doing nothing, which reads like a broken mitigation rather than an intentional skip. `\| default(false)` on both uses. Exercised offline against a simulated skipped registration; `--syntax-check` re-run. Co-Authored-By: Claude Opus 5 <[email protected]> (cherry picked from commit 5ccf5111a54857abb934a64c41332b7aa5046323)	2026-09-07 02:46:26 -05:00
jakman85	a819c50ffe	feat(ansible): arbitrate the shared etcd/Longhorn device — and prove it landed roles/etcd_io_priority + playbooks/etcd-io-priority.yml, enabled for [k3s_prod] only. Mitigation for the 2026-09-07 k3s-prod-1 outage: etcd's WAL fdatasync took 3.4-6.7 s (expects 1 s), the node went NotReady while the host stayed up, and /proc/pressure/io full avg300 read 11.36 there against 0.75 (node 2, NVMe) and 1.29 (node 3). WHAT IT DOES, on the SATA nodes only: 1. k3s-etcd-io-priority.service (oneshot, Before=k3s.service) switches the shared device to BFQ and drops nr_requests 64 -> 32. 2. IOWeight=10000 drop-ins on system.slice AND k3s.service, which BFQ turns into io.bfq.weight=1000 — a 10:1 share against the default-weighted kubepods.slice where Longhorn's instance-managers live. 3. Reads all of it back out of the kernel and ASSERTS. If the scheduler did not switch or the weight did not land, the play fails. THE THREE OBVIOUS ANSWERS ARE INERT ON THESE NODES, ALL CHECKED LIVE, ALL DELIBERATELY ABSENT: * IOSchedulingClass=realtime on k3s.service. mq-deadline DOES honour the priority class, so this looks right — but ioprio is inherited across fork and kept across exec (block/blk-ioc.c __copy_io copies ioc->ioprio) and CONTAINERD IS A CHILD OF THE K3S SERVER PROCESS. Verified on k3s-prod-1: containerd pid 84553 has ppid 84333 = k3s-server, and ionice -c2 -n0 -- sh -c 'ionice -p $$; sh -c "ionice -p \$\$"' prints best-effort prio 0 at both levels. Realtime for k3s is realtime for every container on the node, Longhorn included. No relative gain. * IOSchedulingPriority (the level inside a class): mq-deadline buckets by CLASS only, so BE-0 vs BE-4 is a no-op. * io.latency, which would be the ideal scheduler-independent protection: CONFIG_BLK_CGROUP_IOLATENCY is not built in Debian trixie's 7.1.8 kernel (only IOCOST is) and no io.latency file exists under /sys/fs/cgroup. AND ONE NON-OBVIOUS TRAP: bare IOWeight= is DECORATION here. systemd writes it to io.weight (blk-iocost — compiled in but disabled, io.cost.qos is empty) and io.bfq.weight (BFQ — not loaded). Neither reader exists on the stock nodes, so the scheduler change and the weights have to be one role with one verification. The value is 10000 rather than a "reasonable" 1000 for the same reason: systemd rescales 1..10000 onto io.bfq.weight's 1..1000, so 1000 maps to ~101 against BFQ's unweighted default of 100. The role asserts the result is >= 500. k3s-prod-2 IS SKIPPED ON PURPOSE. The role discovers the device under k3s_data_dir per node (verified: prod-1/3 -> /dev/sda5 -> sda, prod-2 -> /dev/nvme0n1p5 -> nvme0n1) and writes NOTHING on NVMe rather than leaving inert files behind — BFQ is a poor fit for NVMe and node 2 measured 0.75 IO PSI. DEV IS UNTOUCHED. Master switch defaults false and is set true only in group_vars/k3s_prod; dev has a dedicated etcd device (docs/plans/ssd-etcd- isolation.md) and nothing to arbitrate. Verified with --list-hosts: no extra vars -> the three dev hosts (where the role is a no-op); -e k3s_hosts=k3s_prod -> the three prod hosts; --limit k3s-prod-3 -> one. NOTHING IS RESTARTED. The oneshot writes two sysfs files; the weights go in through daemon-reload. If daemon-reload turns out not to re-realize a running k3s.service's cgroup, the verification FAILS and names the rolling restart as the fix rather than quietly leaving the weight unapplied. WHAT IT CANNOT DO, stated in the role README and worth repeating: it does not separate the two tenants. Both still write one ext4 filesystem, and no cgroup control arbitrates the shared jbd2 journal that etcd's fdatasync waits on. It is also UNPROVEN on this hardware — it fails loudly if the settings do not land, it cannot tell you they helped. Apply to k3s-prod-3 first and watch NodeIOPressureHigh. Replacing the C400 / giving etcd its own device remains the real fix and the operator's (PROD-PROVISIONING.md §2 exit condition). Syntax-checked, and the fact-derivation and assert expressions were exercised offline against all three node shapes plus negative controls (a node still on mq-deadline fails the gate; IOWeight=1000 -> 101 fails the floor). Co-Authored-By: Claude Opus 5 <[email protected]> (cherry picked from commit de8c1f832724c011b03466842e18296de07a6776)	2026-09-07 02:46:26 -05:00
jakman85	efc6db6c1f	feat(obs): alert on I/O pressure — the metric that missed the prod-1 outage On 2026-09-07 06:05:21Z k3s-prod-1 went NotReady ("Kubelet stopped posting node status") with the host up, pingable and SSH-able. The cause was disk I/O starvation of the embedded etcd, from its own journal: slow fdatasync took=3.56s / 6.71s / 4.95s / 3.40s (expects 1s) apply request took too long took=8.30s (expects 100ms) etcdserver: too many requests /proc/pressure/io FULL avg300 read 11.36 on k3s-prod-1 against 0.75 on k3s-prod-2 (NVMe) and 1.29 on k3s-prod-3 — all three nodes put /data/k3s (etcd's WAL) and /data/longhorn (every replica) on ONE partition, and node 1's is a 2011 consumer SATA SSD with 61,440 reallocated sectors. The stack alerted on CPU-PSI and memory and had NO I/O alert at all, so the first signal the operator got was a node already out of the cluster. Adds: NodeIOPressureHigh full IO PSI > 5% for 10m warning NodeIOPressureCritical full IO PSI > 10% for 5m critical Thresholds are measured, not guessed: rate() is a 0-1 fraction, so the outage's 11.36% is 0.1136 and the healthy steady state on the same node is ~0.01. Verified live that node-exporter already exports node_pressure_io_{stalled,waiting}_seconds_total for all three prod nodes, so these rules fire on arrival — no new exporter, no scrape change. (etcd's own fsync histograms are NOT available: kube-prometheus-stack's kubeEtcd scrape job is deliberately disabled for k3s, so PSI is the signal we actually have.) PSI full (every runnable task blocked on disk), not some: `some` is noisy on a storage node and did not separate the failing node from its peers. Dashboard gains the matching I/O row and is retitled accordingly. Not scoped to prod — dev shares this file and its k3s VMs are HDD-backed, where the same alert is if anything more likely to earn its keep. Co-Authored-By: Claude Opus 5 <[email protected]> (cherry picked from commit cb28fc80e7e2085d2539530d55f90fc92e3b1bba)	2026-09-07 02:46:26 -05:00
jakman85	09265e0381	merge: OIDC issuer/callback URLs become per-environment gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details	2026-09-07 02:12:15 -05:00
jakman85andClaude Opus 5	e65a76dc51	docs(log): record the prod OIDC-URL outage and the rule it establishes Non-secret, environment-specific values must not live inside an encrypted Secret: Flux's ${BASE_DOMAIN} substitution rewrites rendered manifest text and cannot reach inside ciphertext, so such a value silently freezes at whatever one environment happened to need. Dev was correct only by coincidence (dev IS the apex); prod was wrong from its first reconcile, for both Headlamp and Semaphore. StrictPostBuildSubstitutions cannot catch this — there is no unresolved ${...} to complain about. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-07 02:10:10 -05:00
jakman85andClaude Opus 5	bc9c93d2ab	fix(keycloak): the reconciler could not read its own Secret gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details Shipped in the previous delivery and failing on every run since: FATAL: cannot read /etc/keycloak-realm-clients (PermissionError) BackoffLimitExceeded The client-secrets volume used defaultMode 0400 — owner-only — but the owner of a projected Secret volume is root, while the pod runs runAsUser/runAsGroup 65532. The sibling script volume worked only because it is 0444. So the healer that exists to correct client-secret drift was itself inert. It failed CLOSED (no writes attempted, no drift introduced), and the two alerts shipped alongside it are exactly what surfaces this: KeycloakClientSecretReconciler Failing fires on the failed Jobs. The observability caught its own subject. Fixed with fsGroup 65532 + defaultMode 0440 rather than relaxing to world-readable 0444: fsGroup makes the projected volume group-owned by the running user, so the secret stays unreadable to any other uid in the container. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-07 02:07:28 -05:00
jakman85andClaude Opus 5	6548f9a89c	fix(semaphore): per-environment OIDC URLs — same latent break as Headlamp Semaphore's whole SEMAPHORE_OIDC_PROVIDERS JSON blob lived inside the SOPS-encrypted semaphore-bootstrap Secret (key `oidc-providers-json`) because it CONTAINS the confidential client secret. That buried two environment-specific, NON-SECRET URLs in ciphertext — provider_url (the realm issuer) and redirect_url — where Flux's ${BASE_DOMAIN} postBuild substitution cannot reach. Both were frozen at `home.keithjackson.dev`, so prod's Semaphore was pointed at dev's Keycloak and dev's own redirect URI: identical root cause to the Headlamp login timeout, not yet reported only because nobody had tried prod SSO. FIX: split the blob. The URLs are public and now live as a plaintext literal in catalog/semaphore.yaml (rendered to the apps/wave-1 golden), ${BASE_DOMAIN}-substituted like every other hostname. Only the client secret stays encrypted, in its own key `oidc-client-secret`, and the KUBELET splices it back via dependent-env-var expansion: $(SEMAPHORE_OIDC_CLIENT_SECRET) resolves against the secretKeyRef env var declared immediately above it (order is load-bearing — the kubelet resolves env in list order and leaves a forward reference as literal text). Verified: the new key holds the same value as keycloak-realm-clients/semaphore-client-secret (the mirror invariant), Semaphore v2.19.10's util.OidcProvider consumes client_secret as before, and `flux envsubst --strict` leaves $(...) untouched while resolving ${BASE_DOMAIN} — dev renders keycloak.home.keithjackson.dev, prod keycloak.prod.home.keithjackson.dev. apps/wave-1/semaphore.yaml is a render golden; regenerated with the canonical `catalog/render.py --all catalog -o apps --wave-layout --no-image-automation`. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-07 02:06:52 -05:00
jakman85andClaude Opus 5	bfa39bc5d0	fix(headlamp): per-environment OIDC URLs — prod was dialing DEV's Keycloak The operator could not log in to Headlamp on prod: Get "https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration": dial tcp 192.168.1.53:443: i/o timeout ROOT CAUSE: OIDC_ISSUER_URL and OIDC_CALLBACK_URL were keys INSIDE the SOPS-encrypted headlamp-oidc-credentials.sops.yaml. Flux's ${BASE_DOMAIN} postBuild substitution rewrites the RENDERED manifest text and can never reach inside ciphertext, so both values were frozen at the apex `home.keithjackson.dev` in BOTH environments. Dev IS the apex, so dev looked correct; prod (BASE_DOMAIN = prod.home.keithjackson.dev) therefore pointed at dev's Keycloak at 192.168.1.53 — on the dev LAN behind pfSense, unroutable from prod's 192.168.2.0/24. FIX: an issuer URL and a callback URL are PUBLIC. Move both out of the Secret into the HelmRelease's spec.values.env, written with ${BASE_DOMAIN} so substitution resolves them per environment. The Secret keeps only what is genuinely secret or environment-invariant (OIDC_CLIENT_SECRET, OIDC_CLIENT_ID, OIDC_SCOPES). Exactly ONE source per variable: the chart renders `env:` after `envFrom:`, and an explicit env entry silently WINS over the same name from envFrom — so the two sets must not overlap, and they no longer do. The five load-bearing names the chart's $(...) args require (OIDC_CLIENT_ID/CLIENT_SECRET/ISSUER_URL/SCOPES/CALLBACK_URL) are all still delivered; verified by rendering platform/headlamp per overlay, `flux envsubst --strict`, and `helm template` of chart 0.41.0 against the substituted values. Side effect, deliberate: naming OIDC_CALLBACK_URL in `env:` makes the chart finally EMIT `-oidc-callback-url=$(OIDC_CALLBACK_URL)` — it gates that arg on the value being set via `env:`/`config.oidc`, which a Secret key alone never tripped, so the Secret's callback URL had been dead weight. The value is byte-identical to the single redirectUri registered on the Keycloak `headlamp` client (also ${BASE_DOMAIN}-templated), the only URI Keycloak accepts. [parity] holds: prod's kube-apiserver oidc-issuer-url is an install-time k3s flag from k3s_base_domain=prod.home.keithjackson.dev, which is exactly what Headlamp now uses. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-07 02:06:38 -05:00
jakman85	3ca4ad6011	merge: R2 off-cluster backups + per-environment backup credentials gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details	2026-09-07 01:55:14 -05:00
jakman85andClaude Opus 5	708a05dfc1	fix(devcontainer): restore multi-cluster kubeconfig selection infra/devcontainer/kubeconfig.sh was never staged into the delivery-A commit, so the branch built on main carried main's single-cluster tooling and `--context k3s-prod` failed with 'context does not exist' mid-incident. Not Flux-reconciled, so it never affected the cluster — but it breaks prod diagnosis from a checkout of this branch, which is when you need it most. Carries the concurrency fix: the merged kubeconfig holds both contexts and NO current-context, with each invocation getting a per-invocation selector file prepended to KUBECONFIG. Two concurrent dev.sh runs with different CLUSTER values previously overwrote each other's current-context mid-command — reproduced live, a CLUSTER=dev reader returned k3s-prod nodes on 3 of 7 iterations. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-07 01:55:14 -05:00
jakman85andClaude Opus 5	161430d915	feat(backup): the overlay picks the credential — BACKUP_CREDENTIALS_SUFFIX The previous commit created the two halves. This one wires the choice, so dev renders MinIO credentials and prod renders R2 credentials from the SAME base manifests (R-ENV-02: the overlay patch is the entire difference between environments; no base manifest is forked). cluster-vars BACKUP_CREDENTIALS_SUFFIX base "" \| dev "" \| prod "-prod" velero HelmRelease credentials.existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX} 3x CNPG Cluster s3Credentials.*.name: <db>-backup-credentials${BACKUP_CREDENTIALS_SUFFIX} ONE SUFFIX, NOT FOUR SECRET-NAME KNOBS. Four independent knobs can be updated three-quarters of the way, leaving one consumer reading the OTHER environment's key with nothing rendering wrong — and StrictPostBuildSubstitutions cannot catch that: strict mode fails on an UNDEFINED variable, never on a defined-but-stale one. With one suffix the four move together or not at all. The cost is that the rendered Secret name is not greppable from the consumer, so both kustomizations spell the two names out in full beside the resources. THE BASE DEFAULT IS "" AND THAT IS LOAD-BEARING. It renders the UNSUFFIXED names — objects that exist in every render, and that dev is running today. The alternative failure mode is strictly worse than the shared-Secret state this replaces: velero's pod hangs in ContainerCreating forever when `existingSecret` names a missing Secret (the volume never mounts), and a CNPG Cluster whose s3Credentials Secret is absent stops archiving WAL — which under archive_mode=on is a filling pgdata PVC, not a missing backup. So an unresolvable default breaks the referencing cluster HARDER than a wrong key does. Both overlays state the value explicitly anyway, so neither inherits it silently. Prod's overlay change is a cluster-vars op only. The velero Flux Kustomization patch is untouched (it stays decryption-only, and velero stays suspended on prod). VERIFIED — `kubectl kustomize --load-restrictor LoadRestrictionsNone` on clusters/dev and clusters/prod for the cluster-vars values, then on each of the four affected tiers (platform, platform-config, platform/velero, apps/wave-0) piped through `flux envsubst --strict` with that environment's variables: strict envsubst PASSES for all 8 (env x tier) combinations dev velero existingSecret velero-cloud-credentials dev keycloak/nextcloud/semaphore-db <db>-backup-credentials (both key refs each) prod velero existingSecret velero-cloud-credentials-prod prod keycloak/nextcloud/semaphore-db <db>-backup-credentials-prod (both key refs each) both all EIGHT Secret objects present in the rendered output (dev and prod alike) no `${...}` survives in any rendered tier except gitea's three DELIBERATE `$${...}` shell escapes in apps/wave-0, which are pre-existing and unrelated. Read alongside the endpoints the same overlays already set, the two environments are now isolated on credential AND target: dev velero s3://velero (root) + cnpg s3://cnpg @ http://192.168.1.9:9000 region us-east-1 prod velero k3s-prod-backup/velero + barman prefix @ https://<acct>.r2.cloudflarestorage.com region auto Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-07 01:41:56 -05:00
jakman85andClaude Opus 5	e7c4a95c2d	feat(backup): the four backup credentials become TWO objects each, one per environment The four backup-credential SOPS Secrets are SHARED BASE manifests: platform/ and apps/ are rendered by the same Flux tiers on both clusters, and the ONLY per-environment difference in this repo is cluster-vars substitution (R-ENV-02). So writing the R2 token into all four (0a6a862) did not "give prod credentials" — it REPLACED DEV'S. Merging that as-is would have handed dev's Velero and all three dev CNPG Clusters an R2 key id to sign MinIO requests with: 403 on every WAL segment, and archive_mode=on turns that into a disk-full clock on the pgdata PVC, not a degraded backup. Dev's PROVEN DR restore (R-STATE-03) is the thing that would have broken. This commit splits the material. Each credential is now TWO Secret objects, both applied to BOTH clusters, differing only in name suffix: dev -> velero-cloud-credentials keycloak/nextcloud/semaphore-db-backup-credentials prod -> velero-cloud-credentials-prod ...-db-backup-credentials-prod The consumer picks one by name; that wiring is the NEXT commit, so this one changes no rendered reference and is a no-op for both live clusters. WHY DEV KEEPS THE UNSUFFIXED NAME. Dev is live. Renaming its Secret would change `s3Credentials.accessKeyId.name` on three running CNPG Clusters — which projects into the instance pods' environment and therefore ROLLS keycloak-db, nextcloud-db and semaphore-db — and would roll the velero Deployment, all to rename an object whose contents do not change. This repo already makes exactly this call for dev's live identifiers (EXTERNALDNS_TXT_OWNER_ID "default", VELERO_BACKUP_PREFIX ""): the live cluster keeps the name it is running, and the NEW environment takes the new one. The asymmetry is deliberate and documented at each kustomization. WHERE THE MATERIAL CAME FROM. Neither side is invented: - the four unsuffixed files are restored byte-for-byte from `main` (the MinIO material as it stood before 0a6a862), so dev's Secrets are IDENTICAL to what dev is running; - the four `-prod` files carry the R2 token from 0a6a862 (already on this branch as `9739e9b`), re-encrypted only because metadata.name changed. RE-ENCRYPTION, AND WHY A COPY+SED IS NOT ENOUGH. `.sops.yaml` sets `encrypted_regex: ^(data\|stringData)$`, so metadata.name stays cleartext — but SOPS MACs the WHOLE document (there is no `mac_only_encrypted` here), so editing that cleartext line invalidates the MAC and `sops -d` then refuses the file. The `-prod` files were therefore produced inside the dev toolbox (infra/devcontainer, which has sops 3.9.4 and the age key mounted) by decrypt -> rename -> re-encrypt, with the plaintext never leaving a container tmpfile. Note for the next person: `sops -e --output NEW OLD` matches creation rules against the INPUT path, so encrypting a tmpfile into a repo path needs `--filename-override NEW` or it dies with "no matching creation rules found". Verified: all eight files `sops -d` cleanly; the four dev files hash to dev's material and the four prod files to R2's, and the two sets differ. `.sops.yaml` needed no new rule — the existing `(platform\|apps\|catalog)/.(secret\|sops)\.ya?ml$` rule already matches `-prod.sops.yaml`, so the new files encrypt to the same age recipient. apps/wave-0/kustomization.yaml is GENERATED, so render.py learns the `-prod` sidecar rather than the golden being hand-edited: `<db>-backup-credentials-prod.sops.yaml` now places into the DB's wave alongside its unsuffixed twin (both must exist before the Cluster that names one), and the selftest asserts it. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-07 01:38:22 -05:00
jakman85andClaude Opus 5	0a0527e7db	feat(backup): prod's off-cluster DR lands on R2 — and stops the WAL clock Prod has been archiving WAL to `TBD-PROD-S3-BACKUP-ENDPOINT` since bootstrap. Verified live 2026-09-07 on all three CNPG Clusters: ContinuousArchiving=False, "unexpected failure invoking barman-cloud-wal-archive: exit status 4". Because archive_mode is ON, that is not a degraded backup — it is a DISK-FULL CLOCK. Postgres will not recycle a segment it has not archived, so keycloak-db's pgdata PVC is at 579 MiB used / 4360 MiB free and climbing monotonically. When it fills, the primary PANICs and takes SSO down, and with it kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth. Wiring the endpoint is what lets Postgres drain the whole accumulated backlog in one pass. THE KNOB LAYOUT (three new, one repurposed) and why it is shaped this way. The operator created exactly ONE R2 bucket, `k3s-prod-backup`, but the two backup tools want differently-SHAPED values: the Velero chart takes `bucket` and `prefix` as separate fields, while CNPG's barmanObjectStore.destinationPath is one full s3:// URL whose first path segment IS the bucket. So: S3_BACKUP_ENDPOINT dev http://192.168.1.9:9000 prod https://<acct>.r2... S3_BACKUP_REGION dev us-east-1 prod auto VELERO_BACKUP_BUCKET dev velero prod k3s-prod-backup VELERO_BACKUP_PREFIX dev "" prod velero CNPG_BACKUP_PATH dev s3://cnpg prod s3://k3s-prod-backup/barman There is deliberately NO separate `S3_BACKUP_BUCKET`: CNPG's bucket already lives inside CNPG_BACKUP_PATH, and a second knob holding the same bucket name twice is a rotation waiting to drift. Three knobs, each shaped for its own consumer, and no two of them can contradict each other. TENANCY. Inside k3s-prod-backup, velero and barman get disjoint TOP-LEVEL prefixes — s3://k3s-prod-backup/velero/ and s3://k3s-prod-backup/barman/<cluster>/ — with the bucket ROOT left empty so a misrendered prefix lands somewhere obviously wrong instead of silently colliding. This is not tidiness: Velero ADOPTS every backup under its location and TTL-DELETES it on retention, so an overlapping prefix would let it reap WAL segments; and interleaving two Postgres servers' WAL into one prefix corrupts the chain outright. Dev↔prod isolation is doubly independent — different endpoint AND different bucket names — so neither cluster can reach the other's objects even with a mis-substituted prefix. DEV IS UNCHANGED, BYTE-FOR-BYTE, and that is asserted rather than assumed. Base defaults are dev's LIVE values (verified against the running cluster: BSL bucket=velero with NO prefix, region us-east-1; all three CNPG Clusters ContinuousArchiving=True at s3://cnpg on the MinIO), and the dev overlay now restates all five explicitly for the same reason LONGHORN_DATA_PATH is restated — a silent drift in base would ORPHAN the backups dev's PROVEN restore depends on. The empty-prefix case was the one real risk, and it is settled empirically: with prefix "" the chart omits objectStorage.prefix from the rendered BackupStorageLocation ENTIRELY (helm template, chart 12.1.0), so dev's live BSL is unchanged, not merely equivalent. R2's TWO GOTCHAS, both wired up front: - region must be exactly `auto`. head-bucket answers {"BucketRegion":"WNAM"}, which is a PHYSICAL LOCATION HINT, not a signing region. This was hardcoded us-east-1 inside the Velero HelmRelease — correct-by-accident for MinIO, wrong for anything else — so it is a knob now. - checksumAlgorithm: "". velero-plugin-for-aws v1.14.2 bundles an AWS SDK with flexible checksums on by default and sends x-amz-sdk-checksum-algorithm / x-amz-checksum-crc32 on every PUT; R2 rejects those with HTTP 400 InvalidArgument, failing every backup and every kopia blob upload. It must be PRESENT-but-empty (the plugin branches on the key existing), and the chart does emit an empty value — verified by rendering it. Harmless against MinIO, so it is unconditional rather than a fourth knob. WHAT THIS DOES NOT DO. Velero stays SUSPENDED on prod. `spec.suspend: true` is owned by field manager `flux` (verified: kustomize-controller's managedFields do not include f:suspend), so server-side apply will not strip it, and nothing here declares suspension — which would also trip R-ENV-03, since a prod overlay may patch only cluster-vars and a Flux Kustomization's `decryption`. Unsuspending is the operator's, after `standup.sh --s3-endpoint <r2> s3check` proves a round trip. NO DNS KNOB IS TOUCHED. BASE_DOMAIN stays prod.home.keithjackson.dev and every EXTERNALDNS_* value is untouched. The apex inversion on work/prod-standup-campaign was deliberately NOT carried: this branch ports that branch's R2 reasoning knob by knob, and merges nothing. VERIFIED: ci/l0.sh green (flux-postbuild-substitution 272 assertions, 0 hazards, 33 knobs defined / 28 referenced, every new token resolved in BOTH overlays); render-golden, lint, schema, spec-immutability green; touched requirement R-STATE-03 covered. Refs R-STATE-03, R-IDP-05, R-ENV-02, R-ENV-03. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-07 01:14:29 -05:00
jakman85andClaude Opus 5	dc2d1ce17e	fix(conformance): the R-NET-04 scan descended into nested agent worktrees gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details TRACKED_YAML rglob'd from the repo root excluding only .git and _artifacts, so it also walked .claude/worktrees/agent-*/ — the isolated worktrees created for agent runs. The check counted 5 CiliumEgressGatewayPolicy objects where exactly 1 is authored, failing R-NET-04 for a reason unrelated to egress steering. Now skips dot-DIRECTORIES (covering .git, .claude, .cache) while still matching dot-FILES such as .sops.yaml. .claude/ is gitignored, so nothing beneath it is authored surface. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-07 01:01:53 -05:00
jakman85andClaude Opus 5	6a8953e288	feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing Minimal prod delivery, authored directly on main (= prod's live revision `d1a6628`) rather than by merging work/prod-standup-campaign. That branch carries the DNS apex inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun rewriting apex records dev owns while the non-k3s prod still serves 443 there. Operator sequencing is "dev migrates first". Nothing under clusters/ is touched here at all, which is what makes this safe to land now. Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of platform/velero/ and the CNPG destinationPath changes. Those land as feat/r2-backup-prod together with the endpoint and bucket knobs. WHAT THIS FIXES 1. Nothing was watching CNPG. All three prod databases have run since bootstrap with ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed anywhere. Because archive_mode=on and the archive command fails, Postgres refuses to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within the hour, and evaluates to 3 firing series on prod today. Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because nextcloud-db's exporter cannot log in to its own database (permission denied for database "nextcloud", both clusters) so the pg_stat_archiver family does not exist for it — the obvious metric would have been blind to one database in three. 2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to job="kubelet" (the apiserver copy of the same series reports namespace=default), and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report available=0 forever and would otherwise have fired critical permanently. 3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment a node carries the egress-gateway label, with dev's behaviour unchanged today. 4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored LITERALLY and the import still reports success, then sits at Done=True and never re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal $(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was green. The CronJob is the level-triggered correction; two alerts make its silence observable. Repaired live already — this stops it recurring. Also: the R-NET-04 conformance scan descended into nested agent worktrees under .claude/, counting 4 egress policies where 1 is authored. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-07 00:59:22 -05:00
jakman85andClaude Opus 5	9739e9be91	chore(backup): R2 credentials for the off-cluster DR target Written by platform/backup/set-r2-credentials.sh (operator, 2026-09-07T05:41:34Z) into all four consumers, in the two shapes they need: velero wants an AWS INI file, CNPG/barman wants discrete ACCESS_KEY_ID / ACCESS_SECRET_KEY. All four verified to carry the SAME 32-character key id, so no consumer was missed. Proven from a pod INSIDE prod against https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com: head-bucket on k3s-prod-backup returns BucketRegion WNAM, and PUT and DELETE both succeed. Byte-integrity of GET is NOT yet proven (the aws-cli image has no cmp); s3check is what will establish it. Credentials only — no endpoint or bucket wiring. That is feat/r2-backup-prod. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-07 00:58:45 -05:00
jakman85andClaude Opus 5	d1a6628a97	feat(ha): the CNI dials the control-plane VIP — kube-vip becomes a CNI prerequisite gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details OPERATOR DECISION 2026-09-06: "use the VIP as intended — a virtual address to reach the cluster and provide fault tolerant HA. Without a change it will not work in HA." THE DEFECT. Cilium's kube-proxy replacement dialled a SINGLE NODE. Re-confirmed live before changing anything: all three prod cilium-agent pods carried KUBERNETES_SERVICE_HOST=192.168.2.10 — k3s-prod-1's own address — while the kube-vip VIP 192.168.2.9 was Running on all three nodes with a held lease. Lose prod-1 and the CNI's control-plane path is dead on an "HA" cluster. clusters/prod's own comment said the value must be the VIP and NOT .10, then called it "Not yet decided" — I treated that stale comment as an open question when the evidence already settled it. SPLIT, NOT REPOINT, because k3s_api_host had two consumers with opposite needs: k3s_api_host unchanged — the FOUNDER's IP: the cold-bootstrap join target and a tls-san. infra/tofu/outputs.tf GENERATES this name, so renaming would ripple into tofu and the runbooks. k3s_apiserver_endpoint NEW = {{ kube_vip_address }} — the HA address; what Cilium consumes. Derived, not restated, so an environment overrides ONE line and the VIP, the SAN and the CNI endpoint move together. Prod needed no new override. THE SECOND HAZARD, WHICH MY BRIEF DID NOT NAME and the agent found: pointing Cilium at the VIP breaks a COLD bootstrap for the same reason the join would — site.yml ran node-baseline -> k3s -> cilium, so Cilium would install against an address nothing announced yet. That is the 2026-09-01 deadlock shape. Fixed by SEQUENCE, not value: site.yml now imports kube-vip.yml BETWEEN k3s and cilium. It cannot go earlier (kube-vip is a k3s auto-deploy manifest, so it needs a cluster) and it works there because the DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 — needing neither CNI nor kube-proxy. roles/kube_vip gained a read-only `wait_for` on the VIP so a cold run stops THERE with a named cause instead of surfacing as unreachable agents. Also fixed a pre-existing bug that only mattered once this play joined site.yml: the role's read-only interface probe ran without `check_mode: false`, so under --check it was skipped and the assert fired with a misleading "set kube_vip_interface explicitly". VERIFIED BY ME, not just reported: renders dev K3S_API_HOST=192.168.1.49 prod=192.168.2.9 (each environment's VIP) cilium template consumes k3s_apiserver_endpoint, not k3s_api_host cold join still `--server https://{{ k3s_api_host }}:6443` — the founder order site.yml: node-baseline -> k3s -> kube-vip -> cilium -> verify -> flux gate full CI gate PASS Agent-side, additionally: in-memory render of cilium-values.yaml.j2 per host showing k8sServiceHost=the VIP with the join still on the founder; k3s_tls_sans renders ['.10','.9','127.0.0.1','.11','.12'] so NO cert reissue and NO k3s restart is involved — .9 already answers under full CA verification; and --check --diff of kube-vip.yml against prod reported 6 ok / 0 changed, i.e. the role edits change nothing about the kube-vip prod already runs. ha-control-plane-as-code 92 -> 112 assertions, turning two escalations into machine checks: the Cilium template must use the endpoint var; the endpoint must derive from kube_vip_address and k3s_api_host must not; per environment K3S_API_HOST must equal that environment's VIP and be none of its node IPs (the [parity] contract, now machine-checked rather than a comment); site.yml must order k3s -> kube-vip -> cilium. Every new assertion mutation-tested and bites. One honest negative recorded: reverting only the BASE default is not caught, because both overlays patch the key — the check reads effective per-environment values, so the base default is unguarded. APPLY ORDER, and this is the operator's step — nothing was applied: PROD FIRST, and it is the low-risk direction (prod is the broken one). Push and let Flux reconcile platform/cilium: the value change rolls the cilium-agent DaemonSet ONE NODE AT A TIME (maxUnavailable: 1 — the knob that bounded the 2026-09-01 incident), it will not proceed past an unready pod, existing datapath survives an agent restart, and .9 is already live before the change lands. DEV SEPARATELY, operator present. Same mechanics but live dev agents dial 192.168.1.50, and dev is mid-DNS-migration on dev-live. Do not fold it into that. ANSIBLE: nothing to run for either live cluster; the Ansible half matters only for a cold build. Remove the now-redundant standalone kube-vip.yml step from any runbook. NEW NAMED GAP, unwritten: the L2 test cannot assert the running agents' live KUBERNETES_SERVICE_HOST, which is the drift the L0 check cannot see. Recorded in that test's gap list. Refs R-PLAT-01, R-NET-01, R-ENV-02. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-06 22:17:30 -05:00
jakman85andClaude Opus 5	f178d85c71	fix(kyverno): unblock CDI imports by SATISFYING the label requirement, not exempting it Prod's UniFi VM has no disk. `DataVolume unifi/unifi-rootdisk` stuck in ImportScheduled for hours, because Kyverno denied CDI's own generated pod: validate.kyverno.svc-fail denied Pod/unifi/importer-prime-<uid> require-auth-mode-label (R-IDP-02) require-egress-class-label (R-NET-05) Both policies match any Pod in an `app-namespace: "true"` namespace with no exclusion, and CDI generates `importer-` itself — it cannot carry repo labels. A COLD-START defect, not a prod quirk: dev's identical import Succeeded 9 days ago, before the policies enforced, so dev has never exercised it and would hit it on any rebuild. Fourth bug of this shape in two days. MECHANISM: a MUTATE ClusterPolicy that ADDS `egress-class: internet` + `auth-mode: none` (add-if-absent anchor) to Pods on CREATE in app namespaces WHEN the requesting subject is ServiceAccount `cdi/cdi-sa`. Both validate policies are untouched — no exclude, no exception, `background: true` preserved on them. Why mutate beats exempting: the requirement stays literally true ("every pod in an app namespace declares both labels") and the values are HONEST — the importer does pull over the default WAN path, and `internet` is the least-reach class (it does not grant the VPN path); a transient worker pod with no UI genuinely is `auth-mode: none`. THE MATCH IS IDENTITY, NOT A NAME. `subjects: ServiceAccount cdi/cdi-sa` is the AdmissionReview userInfo, taken from the real prod denial (`system:serviceaccount:cdi:cdi-sa`, name importer-prime-…, op CREATE). A name prefix is attacker-choosable; a requesting ServiceAccount is not — and the test proves it, with a fixture `importer-prime-forged-by-an-app` that copies both the name prefix AND CDI's labels and is still DENIED. Covers every CDI worker kind (importer/uploader/cloner) without enumerating names that change per release. Confirmed on prod that cdi-sa is the only cdi-namespace SA that can create pods. REJECTED, and the first one is a trap worth recording: PolicyException — the CRD EXISTS (v2/v2beta1, from the pre-applied bundle) but all controllers run `--enablePolicyException=false` (verified live on prod: admission, background and reports controllers all carry the flag). An exception would apply cleanly through Flux and be SILENTLY IGNORED. Enabling the gate is a cluster-wide capability change for no gain over the mutate. * `exclude` scoped to cdi-sa — sound identity, worse consequences: any userInfo match forces `background: false`, which would drop background scanning and policy-report coverage of R-NET-05/R-IDP-02 for ALL EXISTING PODS to fix one pod class. It also leaves the importer with no labels at all — the unlabelled fall-through that require-egress-class-label's own header calls a fail-open leak. * name-prefix / CDI-label matching — attacker-choosable, and proven walkable. TWO-WAY PROOF, non-vacuous by construction — same resource, different requester, opposite outcomes (verified by me, not just reported): userinfo cdi-sa -> mutate pass, require-egress-class pass, require-auth-mode pass userinfo ordinary -> mutate SKIP, require-egress-class FAIL, require-auth-mode FAIL Kyverno L0 went 17 -> 29 tests passed, 0 failed. The mutate policy is deliberately kept OUT of the existing tests/kyverno-test.yaml: that suite declares no userinfo, and without userinfo the offline engine does not discriminate on subjects — it mutates everything, which would have silently turned its negative rows into passes. platform-config/kustomization.yaml is the live apply path for these ClusterPolicies; without it the policy never reaches a cluster. RESIDUAL TRADE-OFF, reversible by deleting one file: anything created by cdi-sa in an app namespace now has these labels supplied rather than being rejected for not declaring them — including a pod that is not a CDI worker. That identity already holds `pods create/delete` cluster-wide via ClusterRole `cdi`, so it is not a new capability; what changes is that for that one identity the declaration is assumed rather than demanded. restrict-host-namespaces, the namespace default-deny baseline and the CiliumNetworkPolicies all still apply. FIRST resource-mutating policy on this cluster: kyverno-resource-mutating-webhook-cfg holds 0 webhooks today; with autoUpdateWebhooks=true Kyverno registers mutate.kyverno.svc-fail for Pods within seconds of this applying. Prod's forceFailurePolicyIgnore=false makes it failurePolicy: Fail — same posture as the validating webhook already in that path. CDI retries importer creation on a loop, so the stuck import should proceed once this lands. The detached prime-be3f9d1c-… 40 GiB scratch PVC is separate operator cleanup; untouched, as is the DataVolume. VERIFIED: full CI gate PASS. Nothing applied to any cluster. Refs R-IDP-02, R-NET-05, R-PLAT-01. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-06 22:17:30 -05:00