735 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
gitea-admin	acae2ad36c	Merge pull request 'PROVEN: 0 → 1 — R-STATE-02 proven, plus two ledger bugs that made proof impossible' (#11) from feat/ledger-section-parsing into main	2026-09-02 00:10:06 +00:00
jakman85andClaude Opus 5	fdc3265585	fix(conformance): the ledger counted as evidence FOR the requirements it proves ... Committing the first PROVEN row immediately degraded it to `stale-proof`. Cause: a self-defeating loop. `IMPL_EXCLUDE_PREFIXES` excluded `conformance/tests/` and `ci/tests/` but NOT `conformance/runs/`. The ledger and the preserved run logs CONTAIN the requirement IDs they are evidence FOR — `ledger.jsonl` literally holds `"requirements":["R-STATE-02"]` — so they were scanned as IMPLEMENTING evidence for R-STATE-02. That made `last_change` for the requirement move to the ledger commit itself, which the row's `git_sha` necessarily predates, so the freshness check failed. The consequence was total, not marginal: **recording a proof destroyed it, so NO ledger row could ever have been non-stale.** The ledger would have reported `PROVEN: 0 (stale N)` forever while looking like it was working — the same shape of silent-uselessness as the COMMAND-echo bug in the previous commit, and found the same way: by actually running the thing and disbelieving a green-looking result. Fix: exclude `conformance/runs/`. The ledger is the proof RECORD; it must not participate in the freshness computation about itself. Verified: `PROVEN: 1 (stale 0, self-skipped 0, failed 0, unknown 0) from 1 ledger row(s)`; R-STATE-02 reads **proven**. status.py --selftest PASS. The staleness machinery is otherwise untouched — a genuinely stale row (impl changed after the run) still degrades, which the selftest continues to assert. Serves: R-TEST-01..04. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-01 19:08:38 -05:00
jakman85	4cb5e7a8d4	feat(conformance): first PROVEN row — R-STATE-02 node-loss ... Records the first requirement this project has ever PROVEN, in the only sense its own definition of done accepts: a green, requirement-tagged, reproducible behaviour test, executed against a real cluster, with the evidence committed. test node-loss run 20260901T235117Z-e9a4b66 req R-STATE-02 (safety-critical) junit passed sentinel NODE_LOSS_VOLUME_AVAILABLE_OK proof proven sha 57d4df8 (clean tree) kube v1.36.4+k3s1 What actually ran: seed a 3-replica longhorn volume, cordon the node the writer landed on, evict only its own pod, assert the volume stays available and a reader on another node re-attaches and reads the data back. 131s. It deliberately does NOT drain, so kubelet and the etcd member stay up and quorum is never touched — which is what retired the old 'defer node-loss to a maintenance window' reason as unfounded rather than by fiat. Recorded against the clean tree of the preceding commit so git_sha resolves and the freshness check passes. The test was NOT re-run for this: the preserved run.log is the evidence, and re-running would have proved the parser rather than preserving the run. Serves: R-STATE-02, R-TEST-03, R-TEST-05.	2026-09-01 19:07:39 -05:00
jakman85andClaude Opus 5	57d4df8c4f	fix(conformance): ledger read tokens from Chainsaw's COMMAND echo, not real output ... The run ledger mis-classified the FIRST real L3 execution — a genuine PASS recorded as `skipped`. Diagnosis, from the preserved log, section-classified: line 51 COMMAND echo NODE_LOSS_SKIPPED_OK <- script SOURCE line 190 STDOUT NODE_LOSS_VOLUME_AVAILABLE_OK <- the only REAL emission Chainsaw echoes each script into a `=== COMMAND` block before running it, and every skip-gated test contains `echo <TEST>_SKIPPED_OK` as source text. record_run.py scanned the raw log without distinguishing sections, so it matched the skip token in the echo. NODE_LOSS_SKIPPED_OK never appeared in output at all. SEVERITY: since every skip-gated test embeds its own skip token as source, and skip outranks pass, THE LEDGER COULD NEVER HAVE RECORDED `proven` FOR ANY TEST WITH A SKIP GATE. It would have silently recorded nothing forever while appearing to work. It failed CLOSED — the right direction — but a ledger that always fails closed is an expensive way to write "unknown". FIX: classify_sections() splits the log on indented `=== NAME` markers (a section runs to the first non-blank line indented LESS than its marker). Verified against the real log's actual markers, including that Go's column-0 `=== RUN/PAUSE/CONT` lines carry a test name and must never open a section. Then: PASS tokens are read from STDOUT ONLY — exactly where Chainsaw's own contains($stdout,…) looks, so a source echo or stderr can never manufacture a proof. SKIP tokens are read from every section that is NOT COMMAND, so an unknown future section name fails closed in BOTH directions. Lines outside any section prove nothing — this also closes a second false-positive vector: a FAILING Chainsaw run prints the assertion expression, which contains both tokens. A log with no markers is scanned whole exactly as before (such logs have no COMMAND echo). SECOND BUG, same root: _LOG_HEADER_RE never matched a real Chainsaw log — the lines are ` l.go:52: <BS>| 23:51:18 | <ANSI>node-loss<ANSI> | …`, not `| … |`. EVERY genuine run was silently falling into DEGRADED whole-log mode. Now strips ANSI/backspace noise first and accepts the Go source prefix, so runs attribute per-testcase as designed. Safety properties re-asserted, not assumed: skip still beats pass WITHIN real output; unknown layers still require a sentinel; JUnit failed/error/skipped still outrank everything; missing JUnit or log still exits 3. Negative control: with scan_texts forced section-blind in a throwaway copy, exactly the new cases fail — they do catch this bug. Includes the preserved run directory 20260901T235117Z-e9a4b66 (report.junit.xml + run.log) for R-STATE-02 / node-loss — 131s, cordon -> evict -> volume stayed available -> reader re-attached -> clean teardown, the first L3 fault-injection test ever executed on this cluster. The ledger row is recorded in the FOLLOWING commit, deliberately: git_sha is sampled at record time and a dirty tree records `-dirty`, which is unresolvable and correctly degrades to stale-proof. Recording against this clean commit is the honest way to get a resolvable sha — NOT a --git-sha override, which would be a lever for recording a clean sha over a dirty tree and was deliberately not added. Serves: R-TEST-01..04, R-STATE-02. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-01 19:07:26 -05:00
gitea-admin	14494e1947	Merge pull request 'feat(conformance): committed run ledger — prerequisite for moving PROVEN off zero' (#10) from feat/conformance-run-ledger into main	2026-09-01 23:50:28 +00:00
jakman85andClaude Opus 5	e9a4b669d4	feat(conformance): committed run ledger — the only thing that can move PROVEN off zero ... status.py has always reported PROVEN: 0 for all 51 requirements, and named the fix in its own header: "a committed, machine-readable run ledger". There was none — _artifacts/ is gitignored, the root .gitignore kills *.junit.xml, and a stray JUnit was surfaced only as a quarantined mtime advisory. Built BEFORE the first real L3 execution so that run is captured rather than reconstructed from memory. THE TRAP THIS EXISTS TO AVOID. These tests exit 0 and Chainsaw records `passed` on a SELF-SKIP exactly as on a real run, and Chainsaw's JUnit carries no stdout — only properties. So committing the JUnit as proof would let DR_RESTORE_SKIPPED_OK be recorded as proof of R-STATE-03: authoritative-looking and wrong, which is worse than no ledger. The sentinel token is the ONLY thing distinguishing proven from skipped, and it lives solely in the run log. The ledger therefore records the LOG alongside the JUnit and derives proof from the sentinel, never from the JUnit result. Sentinel tokens are DERIVED, not hand-listed: record_run.py parses every contains($stdout,'TOKEN') out of each test's own chainsaw-test.yaml, so the list cannot drift from the tests. 23 of 35 tests declare no sentinel at all. decide() is the single place `proven` is emitted, ordered WORST-FIRST: JUnit failure/error -> failed JUnit <skipped/> -> junit-skipped ANY declared skip token in the log -> skipped (beats a green JUnit AND beats a pass token present in the same log) no sentinel declared -> proven ONLY for L0/L1 (whose JUnit is truthful on its own, since run_l1_offline.py maps exit 3 to <skipped/>); written as an explicit exempt-set so an unknown/blank layer fails CLOSED EVERY declared pass token present -> proven otherwise -> unknown-sentinel-absent Log attribution is a state machine over Chainsaw's per-test headers; a log with no recognisable header falls back to whole-log scanning restricted to GLOBALLY-UNIQUE tokens, so DNS_OK (owned by two tests) can never prove either. record_run.py exits 3 if the JUnit or the log is missing — nothing recorded, never green. STALENESS: a proven row is honoured only while `git merge-base --is-ancestor` holds between the requirement's last implementing/test change and the row's git_sha; otherwise it degrades to stale-proof. A -dirty sha, an unknown rev, or an uncommitted test path all fail ancestry the same way. A requirement reads proven only when EVERY verifying test does (worst-of), and the latest row per test wins — not best-ever. REFUSAL POSTURE PRESERVED: with an empty ledger the report is unchanged in intent — every requirement not-derivable, headline "PROVEN: 0 — NOT DERIVABLE". This replaces "not derivable" with real evidence only where evidence exists, never with optimism. --record is explicit, never implicit (a run that silently records itself is how a bad run becomes permanent false proof), is rejected for --status/--coverage-only/--lint, and re-execs under tee returning ${PIPESTATUS[0]} so recording can never soften a gate. It rm -f's a stale _artifacts/report.junit.xml first — a hazard hit live during development, where a stale JUnit got paired with a fresh log. Selftests: record_run.py 23 checks (incl. end-to-end Chainsaw-passed + NODE_LOSS_SKIPPED_OK -> proof "skipped"), status.py +16 against a throwaway git repo (self-skip -> skipped; proven-at-older-sha -> stale-proof; dirty/unresolvable -> stale-proof; worst-of; latest-row-wins; malformed lines dropped). No regressions: --coverage-only, --status, --lint, --strict-coverage and the merge-gate shape all rc=0 and unchanged; nothing in ci/ references status.py. Root .gitignore needed no change — conformance/runs/ is not itself excluded, so the deeper negations override the root's global *.junit.xml. Verified with git check-ignore. Serves: R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-01 18:49:01 -05:00
gitea-admin	aa0a50a143	Merge pull request 'chore(flux): restore platform prune:true — Cilium split merge 2 of 2' (#9) from chore/cilium-split-merge2 into main	2026-09-01 23:41:51 +00:00
jakman85andClaude Opus 5	22174186ff	chore(flux): restore platform prune:true — MERGE 2 of 2 (Cilium split cleanup) ... Closes the transition opened in merge 1 (63f98aa). `prune: true` is restored on the `platform` Kustomization now that the ownership transfer is VERIFIED on the live cluster, not merely expected: 13/13 Kustomizations Ready, `cilium` among them, at main@sha1:23aa2854 Namespace/kube-system label kustomize.toolkit.fluxcd.io/name = cilium HelmRelease/cilium label ... = cilium HelmRepository/cilium label ... = cilium platform inventory: 0 entries matching kube-system__Namespace | kube-system_cilium_* cilium inventory: exactly those three objects Cilium pods UNCHANGED from the pre-merge baseline — cilium-2f8h8 r=0 2026-09-01T05:55:27Z cilium-425m8 r=1 2026-08-30T13:25:10Z cilium-54cww r=0 2026-09-01T05:55:26Z HelmRelease still v11 So the relabel deterministically preceded any prune, exactly as the revision-gated dependsOn edge predicted, and the ownership move cost zero pod restarts. The in-file comment now records the lesson rather than the temporary state: before moving ANY platform/* component in future, check whether it is the sole declarer of a shared Namespace. platform/cilium/namespace.yaml was the only declaration of Namespace/kube-system under ./platform, which would have turned a component-scoped prune into `delete namespace kube-system` — taking coredns, metrics-server, local-path and Cilium. L0 substitution gate: 103 assertions, 0 hazards. Serves: R-PLAT-01, R-GITOPS-01. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-01 18:39:20 -05:00
gitea-admin	23aa2854f1	Merge pull request 'feat(flux): Cilium gets its own Kustomization — merge 1 of 2' (#8) from feat/cilium-own-kustomization into main	2026-09-01 23:35:04 +00:00
jakman85andClaude Opus 5	63f98aab26	feat(flux): give Cilium its own Kustomization — MERGE 1 of 2 (transition) ... Cilium was one of ~12 components inside a single `platform` Kustomization with wait:true, so the CNI's health was indistinguishable from eleven unrelated components in the tier's Ready condition, and the only emergency valve was freezing the whole platform tier. This repo already set the precedent in reverse (oauth2-proxy and headlamp were split OUT so unhealthy leaves would not gate a tier); Cilium is the mirror image — the one component that legitimately SHOULD gate everything. New clusters/base/cilium.yaml: path ./platform/cilium, wait:true, prune:true, interval 1h (a CNI does not need a 10m re-render cadence; kustomize-controller is source-triggered so a push still reconciles immediately — the interval is only the drift-correction floor), timeout 10m (a 3-node DaemonSet rollout with wait exceeds the 5m default), dependsOn:[flux-system], and postBuild.substituteFrom: cluster-vars — the last is LOAD-BEARING, not decorative: helmrelease.yaml references ${K3S_API_HOST} and kustomize-controller now runs --feature-gates=StrictPostBuildSubstitutions=true, so without it this Kustomization would fail immediately. *** THE HAZARD WAS FAR WORSE THAN "the CNI gets pruned" *** platform/cilium/namespace.yaml is the ONLY declaration of Namespace/kube-system anywhere under ./platform, and the live object carries kustomize.toolkit.fluxcd.io/name=platform. With prune:true, the moment ./platform/cilium leaves platform's path the stale set is not "the Cilium HelmRelease" — it is `kubectl delete namespace kube-system`, taking coredns, metrics-server, local-path AND Cilium. A full cluster kill, not a CNI kill. TWO INDEPENDENT PROTECTIONS, deliberately not trusting either alone: 1. prune:false on `platform` for this transition merge — Flux's documented Procedure 1 for moving resources between Kustomizations. Chosen over the per-resource `kustomize.toolkit.fluxcd.io/prune: disabled` annotation because that needs THREE merges and leaves the new Kustomization owning resources it can never prune. 2. The pruner's own owner-label filter. In v1.4.0, prune() passes Inclusions: GetOwnerLabels(obj.Name...), so platform only deletes objects whose LIVE labels still say name=platform. The new cilium Kustomization relabels all three to name=cilium on apply, and platform.dependsOn:[cilium] is REVISION-GATED, so platform provably cannot reconcile revision N until cilium has APPLIED revision N. The relabel deterministically precedes the prune. Protection 2 alone would probably have sufficed. Not betting kube-system on "probably". NOTED FOR THE FUTURE, in-file: anyone moving a platform/* component must first check whether it is the sole declarer of a shared Namespace — that turns a component-scoped prune into a cluster-scoped one. Verified: `kubectl kustomize platform` goes 88 -> 85 documents, 3 removed / 0 added / 0 CHANGED, and `kubectl kustomize platform/cilium` reproduces those 3 byte-identical (sha256-matched). A pure ownership move; no Cilium value touched. L0 substitution gate 103 assertions (was 102), new cilium row, 0 hazards. Server-side dry-run against the live cluster accepted both CRs. clusters/prod renders clean off the same base. MERGE 2 (restore prune:true) follows only after verifying: 13/13 Ready; kube-system, the HelmRelease and the HelmRepository all relabelled to name=cilium; platform's inventory no longer lists them; and Cilium pod ages/restarts UNCHANGED from cilium-2f8h8 r=0 2026-09-01T05:55:27Z / cilium-425m8 r=1 2026-08-30T13:25:10Z / cilium-54cww r=0 2026-09-01T05:55:26Z, HelmRelease v11. Serves: R-PLAT-01, R-GITOPS-01. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-01 18:33:20 -05:00
gitea-admin	62968c9ee9	Merge pull request 'docs: P0/P1/P2 landed + postStart crashloop incident' (#7) from docs/poststart-incident into main	2026-09-01 22:02:56 +00:00
jakman85	645994e3a3	docs: record P0/P1/P2 landing + the postStart crashloop incident ... Two bugs stacked and the outer one hid the inner. Fixing the dead OIDC guard made the block execute for the first time in these apps' deployed lifetime, which immediately exposed a latent defect: the hook's last statement was `[ cond ] && action`, so on the SUCCESS path it returned 1 and became the hook's exit status — and a non-zero postStart exit makes the kubelet kill the container. Records the pattern for the next PM: fixing dead code is not a safe no-op. It is the first execution of never-tested code and deserves the caution of new code, not of a one-character escape. Also records what is verified live (Cilium maxUnavailable applied with pod ages UNCHANGED, proving nothing rolled), and that strict substitution is committed but NOT ACTIVE — it needs bootstrap.sh step 1 because the controllers sit outside Flux's inventory.	2026-09-01 16:59:41 -05:00
gitea-admin	ac9f20034c	Merge pull request 'fix(sso): postStart hooks must exit 0 — nextcloud crashloop' (#6) from fix/poststart-exit-status into main	2026-09-01 21:55:38 +00:00
jakman85andClaude Opus 5	4c4f5355d9	fix(sso): postStart hooks must exit 0 — nextcloud crashlooped the moment OIDC ran ... LIVE REGRESSION, caught on the reconcile after merging #5. nextcloud went CrashLoopBackOff with FailedPostStartHook, 7 restarts, and apps-wave-1/2 stalled. ROOT CAUSE — a latent bug that could never fire until now. The final statement of nextcloud's postStart script was: [ "$p" -ge 15 ] && nextcloud_oidc_fail "could not promote ... after retries" On the SUCCESS path (p < 15) the test returns 1, and as the script's LAST command that becomes the hook's exit status. A non-zero postStart exit makes the kubelet KILL the container. This was harmless for as long as the OIDC guard above it was dead code that exited 0 first (see 3484955) — fixing that guard made the block execute for the first time in these apps' deployed lifetime, and the bug fired immediately. That is the honest sequence: the previous commit did not introduce this defect, it UNCOVERED it. Two bugs stacked, and the outer one hid the inner one. FIX: convert to a plain `if`, then `exit 0` explicitly, with a comment recording why. Also made gitea's hook explicit. Its last statement already returned 0 on both paths, so it was correct by accident of ordering; leaving one hook implicitly-safe and one explicitly-safe invites exactly this class back. Both now end with an unconditional `exit 0`, and both say why in-file. Every failure path in both hooks still logs loudly to stderr AND writes a marker file on the persistent volume, and deliberately does NOT crash the pod — the break-glass local admin must stay usable. That contract is now actually honoured; before, the success path was the one that crashed. Fixed in catalog/ (the ServiceDefinitions are the source of truth) and goldens regenerated; render==golden green, renderer selftest PASS. Serves: R-IDP-02, R-IDP-06, R-GITOPS-02. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-01 16:53:08 -05:00
gitea-admin	63002d19c8	Merge pull request 'Close the post-build substitution race: dependsOn + strict mode + CNI blast radius' (#5) from fix/strict-postbuild-substitutions into main	2026-09-01 21:30:53 +00:00
jakman85andClaude Opus 5	6b4da27f84	feat(prod): SOPS decryption (same age key as dev) + GitOps source on main ... Operator decisions, 2026-09-01: 1. Prod uses the SAME SOPS age key as dev. 2. Prod tracks the SAME BRANCH as dev (`main`). Dev is a temporary instance and will be decommissioned once prod is deemed stable. SOPS: eight `decryption: {provider: sops, secretRef: sops-age}` patches mirroring dev exactly — platform, apps-wave-0/1/2, oauth2-proxy, headlamp, crafty, velero. The sops-age Secret is still created OUT OF BAND on prod (bootstrap/sops/install-age-key.sh); the overlay references it by name only, never inlines key material. RECORDED CONSEQUENCE of the shared branch: once prod is live, EVERY MERGE TO main RECONCILES BOTH CLUSTERS. There is no promotion step and no soak window — dev and prod move together. That is a deliberate consequence of dev being temporary, not an oversight, but it removes the "prove it on dev first" safety net that the last few days have leaned on heavily. Worth revisiting if dev outlives its expected life. *** BOOTSTRAP BLOCKER recorded, NOT papered over *** — the source URL cannot work on a cold prod cluster, for two independent reasons found while wiring this: 1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER — the one slated for decommission. Prod syncing from it means prod's source of truth lives inside the cluster being retired. Gitea must move to prod (or off-cluster) BEFORE dev is torn down. This makes Gitea migration the FIRST task of the decommission plan, not the last. 2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP} via a template plugin. On prod that is 192.168.2.20 — PROD's own gateway, where no Gitea listens. Prod Flux would resolve gitea.home.keithjackson.dev to itself and never reach a repository. This is the same split-horizon mechanism that makes dev work, inverted against us. Bring-up therefore needs a TEMPORARY source (the bootstrap host-Gitea, or an off-cluster mirror) with a cutover to the in-cluster URL once Gitea actually runs on prod — precisely the sequence docs/05 describes for dev's own Phase-6 cutover. The branch is settled and correct; only the host is blocked. Documented at the patch site and in PROD-PROVISIONING.md §7. Gate: MERGE GATE: PASS. Serves: R-SEC-01, R-GITOPS-01, R-ENV-02, R-ENV-03. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-01 16:27:32 -05:00
jakman85andClaude Opus 5	666a60776a	fix(flux): close the substitution race — platform dependsOn root + strict substitution (P0/P1) ... P0 — clusters/base/platform.yaml gains `dependsOn: [flux-system]`. This is the actual fix for the incident, not the feature gate. Verified against kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies` (our version): dependsOn is REVISION- gated, not merely Ready-gated — when the dependency shares a GitRepository, the dependent cannot reconcile revision N until the dependency has APPLIED revision N. Applying revision N is exactly what writes a newly-added cluster-vars knob. Graph verified mechanically, not by eye: built clusters/dev and clusters/prod and walked the resulting objects. `platform` was the ONLY Kustomization without a dependsOn, and all 11 substituting Kustomizations chain transitively through it (platform-config -> platform; the waves, apps-vms, oauth2-proxy, headlamp, homepage, crafty, velero -> platform-config). The "not chaining through platform" set is EMPTY, so this one edge protects the whole graph. DFS over the full edge set after the edit: no cycles. The root (clusters/dev/ sync.yaml) runs wait: false, which is what makes the edge deadlock-free — it reaches Ready on apply success without health-gating its children. Accepted trade-off, documented in-file: the root becomes a gate. A malformed manifest anywhere under clusters/<env> — including clusters/base — now stalls platform and everything downstream, where today it would not. Fail-closed, per doctrine. P1 — StrictPostBuildSubstitutions. MY PREMISE WAS WRONG about where this belongs and it matters: this repo deliberately does NOT use `flux bootstrap` (see bootstrap/flux/README). There is no gotk-components.yaml in git, and the live kustomize-controller Deployment carries NO Flux ownership labels — it was installed imperatively by `flux install` from bootstrap.sh and is not in Flux's inventory. `flux-system` self-manages the CR graph, NOT the controllers. Nothing under clusters/** can reach a controller flag. So it uses Flux's documented mechanism for hand-installed clusters: a new bootstrap/flux/components/kustomization.yaml wrapping `gotk-components.yaml` with a JSON-6902 patch appending --feature-gates=StrictPostBuildSubstitutions=true to kustomize-controller. JSON-6902 `add /args/-` deliberately, NOT strategic merge — a container's args is atomic and a merge patch would replace all five install-generated args. bootstrap.sh step 1 now renders (`flux install --export` into a temp dir, apply the patch, ASSERT the gate string is present so a future Flux bump that breaks the patch target fails loudly) then applies. FLUX_INSTALL_PLAIN=1 is a loud escape hatch. Nothing is vendored, so components can never drift from the pinned FLUX_VERSION. Rejected: vendoring gotk-components.yaml under the root's path (the root's prune: true would put Flux's own controllers and CRDs in a prune inventory), and shipping a partial Deployment as a Flux-managed resource (a future prune would delete kustomize-controller). ROLLOUT ORDER MATTERS. P0 arrives automatically on the next reconcile. P1 does NOT arrive on a git push — the controller Deployment is outside Flux's inventory, so an operator must re-run bootstrap.sh step 1, which restarts kustomize-controller (~10-20s; Flux is level-triggered so an interrupted reconcile is simply retried). P0 must be reconciled BEFORE P1 is rolled out: with the gate on but the ordering edge absent, the same race turns a transient empty render into a transient hard Kustomization failure. ALSO FIXES A PHASE-7 BLOCKER THIS CHANGE WOULD OTHERWISE HAVE CREATED. clusters/prod had no root Kustomization — dev/sync.yaml even carried the note "a prod overlay needs its own sync.yaml -> ./clusters/prod". With the new dependsOn, a prod bootstrap would have left platform blocked forever on a flux-system Kustomization that does not exist, and nothing downstream would ever reconcile. clusters/prod/sync.yaml is now authored, mirroring dev. Two further prod-overlay gaps found and DOCUMENTED, not guessed (PROD-PROVISIONING §7): prod has no SOPS `decryption` patch (every encrypted Secret would fail to decrypt — needs the prod age-key decision, a security choice not a copy), and no GitRepository source URL patch (needs a decision on whether prod tracks the same branch as dev, which would make a dev-facing merge reconcile prod). Substitution gate re-run: check OK, 102 assertions, 0 hazards. Gate: MERGE GATE: PASS. Serves: R-ENV-02, R-GITOPS-01, R-PLAT-01, R-ENV-03. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-01 01:45:04 -05:00
jakman85andClaude Opus 5	fdc4a7d9f4	fix(cilium): maxUnavailable 2 -> 1 — bound CNI rollout blast radius (P2) ... During the substitution incident a bad render (k8sServiceHost: "") took down TWO of three Cilium agents simultaneously. The agents' `config` init container falls back to the API ClusterIP, which only the Cilium agent itself can translate — and that agent is the one being replaced — so it deadlocks on `dial tcp 10.43.0.1:443: i/o timeout`. Root cause of the 2-at-once: `updateStrategy.rollingUpdate.maxUnavailable: 2` is the UPSTREAM CHART DEFAULT, confirmed by downloading cilium-1.20.1.tgz and reading values.yaml:437-444 — not a local choice, and not overridden anywhere in this repo. maxUnavailable: 1 makes a rollout replace ONE agent at a time and stop at the first pod that fails to become Ready. The whole value of the setting is that the rollout halts on the first bad node instead of taking the majority of the dataplane with it, and that guarantee only exists at 1. It matters far more on a cold PROD bootstrap, where there is no healthy peer to recover from — dev only self-healed because k3s-dev-3 kept a working agent throughout. Set as a [parity] field in ALL THREE places, per this repo's documented contract that the Ansible bootstrap values and the Flux HelmRelease values must agree on settings that define cluster networking (Ansible installs Cilium at bootstrap; the HelmRelease adopts it; the two layers fight otherwise): platform/cilium/helmrelease.yaml (authoritative for Flux) infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 (authoritative for bootstrap) platform/cilium/values.yaml (reference mirror, kept in sync) Verified by rendering the .j2 with roles/cilium/defaults and diffing full value sets: every [parity] field is byte-equal across all three; only k8sServiceHost differs, as designed (${K3S_API_HOST} vs {{ k3s_api_host }}). MERGING THIS IS NON-DISRUPTIVE. updateStrategy lives at DaemonSet.spec.updateStrategy — a SIBLING of spec.template (chart daemonset.yaml:49-53). The DaemonSet controller decides a pod is out of date by hashing spec.template alone, so the pod-template hash does not move, no new ControllerRevision is created, and zero pods are recreated. Helm patches the field in place. (If it were disruptive it would be self-defeating: rolling agents under the old maxUnavailable: 2 in order to install maxUnavailable: 1.) Verify after reconcile with `kubectl -n kube-system get ds cilium -o jsonpath='{.spec.updateStrategy.rollingUpdate.maxUnavailable}'` plus unchanged pod ages — that is the cleanest proof nothing rolled. Trade-off recorded in the comments: upgrades serialise, roughly two waves to three at 3 nodes (a minute or two). On a large cluster this becomes painful and a percentage such as "10%" would be right — noted so a future larger cluster revisits it deliberately. SCOPE: this bounds blast radius, it does NOT close the substitution race. A bad render still ships; it now breaks one node and stalls rather than taking the majority. platform.dependsOn: [flux-system] (P0) is the fix for the cause. Serves: R-PLAT-01, R-NET-01. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-01 01:40:20 -05:00
jakman85	a344b046c9	docs: research findings — measured Flux substitution semantics + ranked remediation plan ... Records three parallel research passes and, importantly, TWO CORRECTIONS. Correction 1 (mine): StrictPostBuildSubstitutions alone is INSUFFICIENT — it fixes nothing about ordering. The actual fix is one line: platform.dependsOn: [flux-system]. Verified against kustomization_controller.go::checkDependencies at our exact version — dependsOn is REVISION-gated, not merely Ready-gated, so a dependent sharing a GitRepository cannot reconcile revision N until the dependency has APPLIED revision N. Correction 2 (a research agent's): the claim that Helm considered the bad release healthy, so remediation never fired, is WRONG — measured on the live cluster, the Helm release secrets show v9 = failed. The machinery did fire. maxUnavailable: 2 of 3 IS confirmed, so the real harm was blast radius, not missed detection. Adds a MEASURED substitution-semantics table, including the two anti-patterns nobody had written down: ${VAR:?msg} renders the MESSAGE as the value, and ${VAR:=default} silently ships the dev value to prod. Both are exempt from strict mode by design — which is exactly why strict would never have caught the gitea/nextcloud dead-guard bug. Also corrects repo doctrine: escaping is TOTAL ($${ARR[0]} is safe); only the unescaped indexed form is fatal. And qualifies an earlier overstatement in this log: kustomize strips structural YAML comments before envsubst, so the hazard is block-scalar bodies, not all comments. Notes Flux 2.4.0 is out of support and that strict became default-on in kustomize-controller v1.9.0 — so the audit is a prerequisite for the upgrade regardless. Ranked plan P0-P5 with what is done, what is ready, and what needs an operator call.	2026-09-01 01:33:16 -05:00
jakman85andClaude Opus 5	8a2be824f6	feat(ci): substitution-safety gate + escape 6 comment hazards; strict mode is now SAFE ... Adds conformance/tests/l0-static/flux-postbuild-substitution/ — an offline, stdlib-only L0 check tagged R-ENV-02 + R-GITOPS-01 (deliberately NOT R-TEST-01, which would overclaim). It reuses lib/status.py's `resources:` walk rather than forking it, so "what a path renders" has exactly one definition. It asserts: every ${...} on a substituted path is defined-or-escaped (comments included); no default-operator form on an undefined name; no unparseable expansion; the rendering path actually substitutes cluster-vars; every overlay-patched /data/KEY has a base default; and that the gate itself is load-bearing. WIRED AS AN UNCONDITIONAL STAGE (3c) in ci/l0.sh, and that matters: stage 3b scopes to touched requirements, and ci/touched-requirements.sh maps only clusters/* to R-ENV-02 — so an edit to a substituted manifest under apps/ or platform/ would have filtered this check out of exactly the change that introduces the bug. l0 fail -> BLOCK=1 in gate.sh. Mutation-tested all five assertion classes (undefined token, indexed expansion, :- form, missing substituteFrom, un-wired gate); each fails as intended, baseline green. STRICT MODE IS PROVEN SAFE TO ENABLE — measured, not inspected. For all 11 substituted paths, `kubectl kustomize --load-restrictor LoadRestrictionsNone <path> | flux envsubst --strict` exits 0 against cluster-vars base defaults. Negative control confirms the incident: with K3S_API_HOST unset, non-strict renders k8sServiceHost empty while strict exits 1 with `variable not set (strict mode)`. The feature gate itself is NOT flipped here — that is a live-cluster change and the operator's call. Escapes the six comment hazards that stood in the way (apps/vms/unifi x4, platform/longhorn x2). All were prose ABOUT the escaping convention; since the repo's own contract is that $${...} emits a literal ${...}, escaping them is both precise and self-consistent. kustomize strips these particular comments today, but comment survival is an implementation detail — a body comment carrying $ WAS observed surviving into built apps/wave-1 — and a survivor would kill the whole Kustomization. DOCTRINE CORRECTED (measured against flux 2.4.0): clusters/base/apps-vms.yaml claimed an indexed expansion is fatal "escaped or not". Escaping is TOTAL — $${ARR[0]} emits the literal text safely; only the UNESCAPED indexed form is fatal. Corrected in place. Also empties DOCUMENTED_GAPS in conformance/run.sh with rationale: R-ENV-02 and R-GITOPS-01 now have a real verifying test, which run.sh's own comment named as the trigger for removing them. Advisory output (not failures): 6 dead cluster-vars knobs — BASE_DOMAIN, CLUSTER_ENV, CLUSTER_POD_CIDR, PIA_REGION, PLATFORM_NODE_SELECTOR_{KEY,VALUE}. The last four are documented reference-only; BASE_DOMAIN and CLUSTER_ENV look like genuine rot (hostnames are hardcoded elsewhere) and are worth a follow-up. Gate: MERGE GATE: PASS. 102 assertions, 0 hazards. Serves: R-ENV-02, R-GITOPS-01. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-01 01:31:07 -05:00
jakman85andClaude Opus 5	3484955903	fix(sso): gitea + nextcloud OIDC bootstrap was DEAD CODE — envsubst ate the guard ... Both ServiceDefinitions guarded their Keycloak OIDC registration with an UNESCAPED braced expansion: if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then # gitea if [ -z "${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then # nextcloud, then exit 0 apps-wave-0/1 both declare postBuild.substituteFrom: cluster-vars, and neither name is a cluster-vars key — so Flux's envsubst substituted them away AT RECONCILE TIME, long before the container ever ran. Verified in the LIVE cluster: the deployed Deployments literally contain if [ -z "" ]; then which is always true. The entire OIDC registration block has been skipped on EVERY boot of both apps. WHY NOBODY NOTICED: both apps DO have the provider registered — `gitea admin auth list` shows keycloak/OAuth2/true and `occ user_oidc:provider` shows the keycloak provider. Those live in each app's own database, persisted from an earlier successful registration. SSO works today and DECISIONS-LOG is not wrong. But the idempotent re-registration is dead, so any PVC rebuild, DB restore, or fresh cluster build comes up with NO OIDC and only the break-glass local admin. A PROD BUILD FROM THIS REPO WOULD SILENTLY HAVE NO SSO on gitea or nextcloud. The irony: the surrounding code was hardened twice — see its own comments about a "silent first-boot RACE", the verify-then-retry loops, the marker files. All of that hardening was unreachable, because the guard above it always evaluated true. Not a regression from the recent R-ENV-02 tokenization — introduced 2026-08-27 (8738871). FIXED AT THE RIGHT SEAM: apps/wave-0/gitea.yaml and apps/wave-1/nextcloud.yaml are RENDER-GENERATED, so the fix is in catalog/*.yaml and the goldens are regenerated. Escaped to $${...}, the convention documented in clusters/base/apps-vms.yaml. Post-envsubst the container now receives the intended `${VAR:-}` instead of `""`. STRICT MODE WOULD NEVER HAVE CAUGHT THIS. Flux deliberately exempts default-providing operators (-, :-, +, :+, =, :=, ?, :?) from StrictPostBuildSubstitutions, because they "intentionally handle the unset case". So a targeted check was required. MADE UN-REPEATABLE IN THE RENDERER: catalog/render.py gains a substitution-safety section wired into emit(), so every render path refuses to emit a manifest containing a braced expansion that is not exactly ${KNOWN_KEY} for a key read LIVE from cluster-vars (no hardcoded copy to drift; unreadable => empty set => fail-closed). Dollar-runs are counted properly ($$ is a literal $ to envsubst, so odd runs are flagged and even runs are not), and indexed forms get their own hard-error message. Nine new --selftest assertions, including the exact regression. Proven to bite: rendering the pre-fix catalog files from HEAD now exits 1 with the precise diagnosis. Sweep: the complete substituted surface was enumerated from every Kustomization's path. catalog/ contains exactly TWO braced expansions and both were these bugs; every other braced expansion on a substituted path is a declared cluster-vars key (zero typos, zero undeclared names); existing $${UOS_URL} escapes verified intact. Verified: --selftest PASS; canonical render is byte-identical to the committed apps/ for all 10 onboarded services; the three L1 render checks pass. NEXT RECONCILE: both pod templates change, so gitea (wave 0) and nextcloud (wave 1) roll — strategy: Recreate on RWO Longhorn PVCs, so expect a brief single-pod outage each. The OIDC block then executes for the first time in these apps' deployed lifetime. Both paths are idempotent (gitea takes auth update-oauth on the existing id; nextcloud's user_oidc:provider is create-or-update), so the expected result is a no-op re-write plus the verify loop logging success. If the SOPS client secret ever diverged from Keycloak's, this is when it gets corrected instead of silently left stale. Tail both pods after merge. Serves: R-IDP-02, R-IDP-06, R-ENV-02, R-GITOPS-02. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-01 01:29:40 -05:00
jakman85	b8602c939c	docs: INCIDENT — Flux post-build substitution fails OPEN, briefly broke Cilium ... Merging PR #4 put cilium agents into Init:Error on 2/3 dev nodes for ~6 minutes. Self-healed; cluster fully green. Recorded because the mechanism is systemic and far more dangerous on a cold prod bootstrap. Root cause: cluster-vars (which DEFINES a knob) is owned by the flux-system Kustomization, while platform/ (which REFERENCES it) is owned by the platform Kustomization, with no dependency ordering between defining a variable and using it. Flux post-build substitution is NON-STRICT by default, so during the window where platform rendered the new manifest against the old cluster-vars, ${K3S_API_HOST} became the EMPTY STRING. Cilium upgraded with k8sServiceHost: "" and fell back to ClusterIP 10.43.0.1 — which only the Cilium agent can translate, and that agent was the thing restarting. Its config init container then could not reach the API server at all. Proven by the HelmRelease config digest going e83bf695 -> eebd6c88 -> e83bf695: the config changed and then changed back to BYTE-IDENTICAL. The pre-merge byte-identity verification was correct — the steady state is identical; the damage was a transient intermediate render. optional: false does NOT protect against this: it requires the ConfigMap to exist, not that every referenced key is defined. Fix recommended, NOT applied unilaterally: enable the StrictPostBuildSubstitutions feature gate on kustomize-controller so an undefined variable FAILS the Kustomization instead of silently emptying it — fail-closed, per this project's own doctrine. Requires first proving every ${VAR} in every substituted path is defined or correctly escaped, or it would fail legitimate embedded shell expansions. Interim discipline: add a knob in one commit, reference it in a later one. Serves: R-ENV-02, R-GITOPS-01, R-PLAT-01.	2026-09-01 01:00:16 -05:00
gitea-admin	dce0e8ffe9	Merge pull request 'Phase 7 (prod) opened — prep only: T-7.1 checklist, R-ENV-02 tokenization, node_health, CI false-green fix' (#4) from docs/phase-7-open-prod-node-1 into main	2026-09-01 05:43:25 +00:00
jakman85andClaude Opus 5	ff0533106b	feat(prod): operator decisions — node DNS, service VIPs, become vault, collections path ... 1. PROD NODE DNS = 8.8.8.8 + 1.1.1.1 (operator, 2026-09-01). Two INDEPENDENT public upstreams, so the single-upstream SPOF that broke containerd pulls and CoreDNS forwarding on dev cannot recur. Prod deliberately differs from dev, which is PiHole-first because dev needs the local jackson.lan zone. SCOPE CORRECTED IN THE COMMENTS. Three resolvers answer *.home.keithjackson.dev and only one is configured here: pods -> CoreDNS -> platform/coredns/coredns-custom.yaml, whose `template` plugin answers the WHOLE zone with ${GATEWAY_VIP}. Self-contained in-cluster; PiHole is NOT in this path. Prod is already correct (.20). LAN -> PiHole. Not a cluster concern. nodes -> THIS FILE. The one thing public node DNS breaks: the k3s API SERVER is a node process carrying oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab and reaches the issuer through the NODE resolver, so prod would get the PUBLIC record instead of the prod Gateway VIP — breaking kubectl OIDC SSO. Dev avoids this only because its node resolver IS PiHole. Not blocking (prod has no cluster or Keycloak yet); options are recorded in PROD-PROVISIONING.md §4 for the operator to pick. 2. SERVICE VIPs DECIDED: UNIFI_L4_VIP 192.168.2.21, CRAFTY_L4_VIP 192.168.2.22 — the first two addresses after GATEWAY_VIP (.20), mirroring dev's .53/.54/.55 ordering within the .20-.30 pool. S3_BACKUP_ENDPOINT stays TBD-PROD-* : no prod backup target exists, and dev's versitygw is across the pfSense boundary on the dev host, so it is not DR for prod. 3. PROD BECOME VAULT committed, ansible-vault encrypted, matching the hypervisor and proxmox_hosts pattern. Verified working: `ansible k3s_prod -m command -a 'id -un' --become` returns root. The L0 no-plaintext-secrets check asserts every ansible vault.yml starts with $ANSIBLE_VAULT, so this stays honest by gate. 4. TOOLCHAIN GAP FIXED: the devcontainer image does NOT ship the Galaxy collections, so EVERY ansible play died at parse time on community.general.modprobe. ansible.cfg now sets collections_path = collections, and the collections are installed there from the existing collections/requirements.yml (gitignored — pinned by requirements, fetched at setup, vendored bytes are not source). 5. DRIVE HEALTH MEASURED and recorded in §2 with the raw attribute table so the numbers are auditable. Verdict: usable, but not for production etcd/Longhorn state — 31% of rated life consumed, 15 grown bad blocks, 187 erase failures, and 32 unexpected power losses on a consumer drive with NO power-loss protection, in a single failure domain. etcd is the worst possible tenant: it fsyncs constantly and unclean power loss without PLP is exactly how a member corrupts. 6. NON-ECC RAM: RISK ACCEPTED by the operator, recorded as a CLOSED decision, not an open item — do not re-litigate. Rationale: 3-node k3s homelab, not enterprise; corruption on one node is outweighed by the other two and causes node ejection. That holds for the dominant mode (etcd raft CRC + quorum; Longhorn 3 replicas + hard anti-affinity + rebuild). Recorded residual: corruption introduced UPSTREAM of replication replicates faithfully to all three replicas, so quorum sees no disagreement — low probability, and the only mitigation is different hardware. Follow-up noted: Longhorn's snapshotDataIntegrity is left at the chart default rather than set explicitly, and it is the mechanism that operationalises the operator's own argument. Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03, R-SEC-01. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-01 00:40:29 -05:00
jakman85andClaude Opus 5	0b3db6e778	fix(node_health): SMART verdict was wrong by 4096x — judge on portable signals ... The first real SMART run FAILED k3s-prod-1's drive on reallocated=61440. That verdict was WRONG, and the failure mode is instructive: the check trusted a VENDOR-DEFINED raw value as if it were an ATA-standard sector count. Micron TN-FD-20 ("SMART Attributes for the C400 SSD") names attribute 5 "Reallocated BLOCK Count" and defines its raw as grown-bad-blocks x Block_Sector_Count, where Block_Sector_Count is 256 x 16 = 4096 for the 128Gb-NAND parts (256GB/512GB). This is the 512GB C400-MTFDDAC512MAM: 61440 / 4096 = 15. Triply corroborated on the drive itself — attr 170 Grown_Failing_Block_Ct = 15, attr 196 Reallocated_Event_Count = 15, attr 5 normalised value 100 against thresh 10, and `when_failed` empty on every attribute. FIFTEEN grown bad blocks, not sixty-one thousand bad sectors. smartctl(8) states it outright: since ATA/ATAPI-4 rev 4 the meaning of raw Attribute fields is "entirely vendor-specific". So a raw value can never be a portable verdict. FAIL now comes only from signals that mean the same thing on every drive: the drive's own smart_status.passed; NORMALISED value vs the MANUFACTURER's own threshold via smartctl's when_failed ("now"/"past"/""), split Pre-fail -> FAIL and Old_age-or-past -> WARN; the ATA-standard counters 197/198/187 (199 CRC is WARN-only — it is link, not media); and for NVMe critical_warning / media_errors / percentage_used. Wear reads from the NORMALISED lifetime indicator (202/231/233, or NVMe percentage_used) — a percentage by definition. Vendor raw counts are still collected and PRINTED WITH THEIR NUMBERS, and may raise a WARN, but can never drive a FAIL. Attribute 5's raw is now shown beside the normalised reading that contradicts it, labelled vendor-encoded, with a mechanically-derived "(= 15 x 4096)". Endurance attributes are matched by THE NAME smartctl's drivedb resolved for that model, not by bare ID — drivedb is the portability layer; on an unrecognised drive the name is Unknown_Attribute, nothing matches, and the field stays null with no claim made. The STORAGE section now prints 170/172/173/174/189/206, whose previous omission is exactly what let the misread stand unchallenged. Also documented from TN-FD-20: on this family attr 197 "will always be 0, as reallocation will be done on-the-fly" — so pending=0 is NOT evidence of health here. The role reports the number and infers nothing from it. Branches real hardware cannot exercise were verified with a harness rendering the actual role expressions against synthetic smartctl payloads: virtio/no-stdout -> SKIP; NVMe healthy -> PASS; NVMe degraded -> FAIL; a genuinely dying ATA drive (attr 5 normalised 8 <= thresh 10, pending 48) -> FAIL driven by the threshold trip and standard counters, not the raw; unknown-vendor attrs -> PASS with the unnamed counter ignored; failed-in-the-past -> WARN. NEW VERDICTS. k3s-prod-1: 0 FAIL / 4 WARN / 12 PASS / 1 SKIP — SMART is a WARN naming the real findings (31% of rated life consumed, 15 grown bad blocks, 187 erase failures, 32 unexpected power losses on a drive with no power-loss protection). k3s-dev-1: unchanged, all three virtio disks still SKIP with their stated reason. BREAKING (JSON artifact): the fact `reallocated_sectors` is renamed and gone — that name asserted a unit ATA does not guarantee. Replaced by realloc_attr (the whole attribute, normalised + raw), realloc_events, and grown_bad_blocks. Nothing in the repo reads it. Serves: R-ENV-03, R-OBS-04. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-01 00:40:29 -05:00
jakman85andClaude Opus 5	ce1966acb0	docs: PM handoff — Phase-7 prep session complete (PR #4, 13 commits, CI green) ... Records what landed, the numbers (coverage 37->40/51, safety-critical scaffolds 1->0, PROVEN still 0 by design), the corrections made mid-session so they are not re-litigated (prod NOPASSWD rejected; the '$i envsubst bug' disproven by running the real tool; the stranded DNS branch dropped rather than landed), three near-misses worth remembering, the six items blocked on the operator, and what is knowingly left undone. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-01 00:15:28 -05:00
jakman85andClaude Opus 5	c1f24e4a6d	fix(dns): resilient node resolvers — mechanism-detecting, per-environment (R-DNS-01/03) ... Re-authors the fix stranded on origin/fix/node-dns-resilient-upstream (7a34590). That commit was NOT landed: it wrote a netplan drop-in and ran `netplan apply`, but netplan does not exist on these nodes any more (Debian 13 trixie generic cloud image; not in node_baseline_packages). Landing it would have failed the handler with command-not-found and BROKEN node-baseline.yml on every node — worse than not landing it. Its dev-LAN literals also sat in role defaults, which prod inherits, pointing prod nodes at a PiHole on a network they cannot reach. The underlying problem is real and verified: a SINGLE upstream resolver is a hard SPOF. PiHole answers jackson.lan instantly but its external recursion flaps — 0/25 for public names (registry-1.docker.io, quay.io, ghcr.io) while gitea.jackson.lan resolved 5/5 — breaking containerd image pulls and CoreDNS `forward . /etc/resolv.conf`. MECHANISM DETECTION (new `dns` tag in node_baseline). Five read-only probes feed one set_fact yielding three outcomes, because the two environments no longer share a resolver stack (dev-1 is behind resolved's 127.0.0.53 stub; prod-1 has a direct resolv.conf and no cloud-init at all): * systemd-resolved — resolved running AND owning /etc/resolv.conf (proven by the symlink pointing into /run/systemd/resolve/, or by every nameserver being a resolved loopback). Writes resolved.conf.d/99-k3s-dns.conf with an ordered DNS=, Domains=, and FallbackDNS= EXPLICITLY EMPTIED so systemd's compiled-in public resolvers can never silently bypass the local zone and ad-blocking. * resolv-conf-file — plain directly-managed file (prod's shape). Ordered nameserver lines + options timeout:2 attempts:2, backup: true. Deliberately NO `rotate`, which would destroy PiHole-first ordering. * contended — resolvconf/openresolv present or NetworkManager active. Writes NOTHING and fails loudly: a change that silently gets reverted is worse than no change. Resolved-running-but-not-owning-resolv.conf correctly falls through to the file branch. Ordering is the policy that keeps the local zone working — the second server is only consulted after the first fails, so a gateway NXDOMAIN can never beat PiHole to a jackson.lan answer. SEAMS. Dev values in role defaults ([192.168.1.18, 192.168.1.1], search jackson.lan); prod values in inventory/group_vars/k3s_prod/vars.yml. NOT cluster-vars — that ConfigMap is consumed by Flux postBuild for platform/ and apps/, and no Ansible code reads it. PROD'S SECOND UPSTREAM IS AN OPEN OPERATOR DECISION and is left as a loud BLOCKED-pending-second-upstream placeholder, the same precedent as the node_baseline disk devices, with a dns-tagged assert that stops the play before any file is written. A public resolver was NOT invented as a default: it would bypass the local zone, which is a decision, not a default. PROD-PROVISIONING.md §4 was rewritten (the old bullet pointed at the now-dropped branch) to state the choice, including that prod routing to the dev PiHole is unverified across the pfSense boundary. tofu: servers = [var.dns_server, var.network_gateway] — the one hunk of the old branch worth taking as-is. Dev-only by nature. ALSO FIXES A FALSE WARN THIS WOULD OTHERWISE CREATE: node_health_collect.py parsed /etc/resolv.conf only, so on dev it read the 127.0.0.53 stub and would report a permanent single-upstream WARN even after a correct fix. It now asks resolvectl for the real upstreams when it sees the stub — and if that fails, nameservers is NULL with the reason recorded, never the stub and never a guess. assess.yml turns null into SKIP, not WARN and not PASS. The honesty discipline is preserved exactly. Verified with NO play run against any node: syntax-check + yamllint + tofu fmt clean; --tags packages still selects ONLY the apt task (the prod package-floor escape hatch bypasses DNS entirely); --list-hosts unchanged; -c local simulation driving the REAL detection expression and REAL file bodies extracted from the role by script (so the test cannot drift) passes all 5 scenarios — dev stub -> systemd-resolved, prod shape -> resolv-conf-file, NetworkManager and resolvconf -> contended; dev defaults pass the assert byte-identically while prod overrides fail it by name; collector unit-tested against 8 resolver shapes. Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-01 00:11:49 -05:00
jakman85andClaude Opus 5	14be9b68f6	test(conformance): dr-restore actually asserts off-cluster restore (R-STATE-03) ... Closes the LAST scaffold-only safety-critical requirement. The operator has attested DR works (Velero -> Proxmox versitygw S3, restore demonstrated), but the L3 test carried the right tag and layer label over a 5-line TODO body — so coverage counted it and reported "0 safety-critical gaps" while the suite asserted nothing. That was test debt, not capability debt. This makes the suite assert what the operator already demonstrated. Three steps: 1. assert-backup-target-is-off-cluster (read-only): BSL must be phase Available (Velero itself reached the store), s3Url host is not localhost/*.svc/*.cluster.local, and the resolved IPv4 is none of — loopback, any Node Internal/ExternalIP, any Service ClusterIP, any Pod IP, inside any node podCIDR, inside any ServiceCIDR. A BSL that exists but never reaches Available FAILS (a live DR defect), it does not skip. 2. assert-backups-are-scheduled: R-STATE-03 says SCHEDULED off-cluster backups — Velero + BSL but zero Schedule in phase Enabled is a fail. 3. The drill: seed a nanosecond-unique canary onto a longhorn PVC in its own scratch namespace; Backup to Completed; assert a PodVolumeBackup for that volume is Completed WITH a non-empty status.snapshotID (the kopia snapshot of volume DATA reached the remote repo, not just resource JSON); ANNIHILATE the source (delete ns, poll until gone, force-delete a lingering PV, FAIL if the original PV survived — a survivor makes the restore prove nothing); Restore with namespaceMapping into a clean namespace; assert the restored PVC bound a DIFFERENT PV and that the canary bytes are identical. Off-cluster proof = endpoint outside every cluster address + BSL Available + kopia snapshotID present + every in-cluster copy destroyed + restore landing on a new PV. The bytes could only have come from outside. Skips GREEN on: no longhorn SC, Velero API absent, no BSL, Velero/node-agent not ready — all three steps re-derive the same gates so it can never half-run. An RBAC denial is deliberately NOT a skip: it fails loudly as a harness defect. Teardown is a step-level `finally` that ALWAYS runs — deletes the Restore, issues a DeleteBackupRequest so objects leave the bucket (2h ttl as backstop), deletes both scratch namespaces by name and by label, releases leftover PVs whose claimRef is one of its own. Everything it destroys is something it created. conformance/incluster/rbac.yaml: authored the R-STATE-03 grants (velero.io backups/ restores/deletebackuprequests create+delete+read; BSL/schedules/podvolume* read-only; persistentvolumes get/list/DELETE — the off-cluster proof requires destroying the PV). Also two pre-existing gaps other tests already depended on: storageclasses get/list (the skip gate of five tests) and apps deployments/daemonsets read. No write verbs on platform controllers. NOT ASSERTED, stated in the file header rather than implied: off-HOST (on dev the nodes are VMs on the same Proxmox host as the S3 target, so this does not survive hypervisor loss); no direct S3 client read (that would hand a privileged test identity the backup credential for weaker evidence); real service data (Nextcloud/Gitea/CNPG drills stay operator exercises); bare-metal rebuild DR; encryption-at-rest; retention/GC. status: scaffold-only 5 -> 4; safety-critical w/o executable L3 1 -> 0 (of 6); executable 35 -> 36. coverage unchanged at 40/51 (correct — coverage only asks whether a tagged test exists). PROVEN stayed 0 / not-derivable: it did NOT flip to proven just because the body is non-empty. AUTHORED, NEVER EXECUTED — carries a conformance.home.k3s/never-executed annotation. No cluster access from this workstation. To convert authored -> proven the operator must, IN A MAINTENANCE WINDOW (it deletes namespaces and PVs and writes/deletes bucket objects): apply the updated rbac.yaml, run `./conformance/run.sh --requirements R-STATE-03 --layer l3`, confirm the log shows DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK and NOT DR_RESTORE_SKIPPED_OK (green but proves nothing), confirm no leaked *-dr-src/*-dr-dst namespaces or orphan PVs or *-dr-backup objects, and record the green run. Serves: R-STATE-03, R-TEST-05. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-08-31 23:53:36 -05:00
jakman85andClaude Opus 5	e623d27e2d	feat(ansible): parameterise target groups so prod reuses the roles unchanged (R-ENV-02) ... roles/k3s_server, roles/cilium and every node-host playbook hardcoded the dev group names k3s_primary/k3s_secondary/k3s_servers, so the prod inventory groups could not be selected at all. That broke the R-ENV-02 promise that prod reuses the SAME roles, and blocked PROD-PROVISIONING.md §6 outright. TWO KNOBS, deliberately separate — Ansible templates a play's `hosts:` BEFORE any host context exists, so group_vars is unreadable there, while roles run WITH host context: * which hosts a play RUNS ON — k3s_hosts, k3s_primary_hosts/k3s_secondary_hosts (k3s.yml's split), vpn_egress_gateway_hosts; each `| default(<dev group>)`, set by -e only. Follows node-health.yml's established `<name>_hosts | default(dev)` pattern. * which group a ROLE means by "my primary"/"my servers" — k3s_primary_group, k3s_servers_group in group_vars/all.yml (dev), overridden in group_vars/k3s_prod. Follows the k3s_api_host convention already in this repo. No per-run flag. DEV IS UNCHANGED — the key evidence. --list-hosts/--list-tasks/--syntax-check captured for all 9 playbooks at HEAD and after: sorted host lists IDENTICAL for all 9, same primary/secondary split, same 6-play site.yml. (Raw --list-hosts PRINT ORDER varies run to run; proven to be Ansible's own set ordering by running UNMODIFIED HEAD cilium.yml four times -> 3,1,2 / 1,3,2 / 1,2,3 / 1,2,3. Hence sorted comparison.) --list-tasks differs in exactly two files, only by added TAGS annotations and one new assert; no task reordered, renamed or removed. Var resolution verified per environment with -c local: dev -> k3s_primary/192.168.1.50/sans .50,.49,127.0.0.1,.51,.52; prod -> k3s_prod_primary/192.168.2.10/vip .9/sans .10,.9,127.0.0.1,.11,.12. BREAKS THE SMARTMONTOOLS CIRCULAR DEPENDENCY. node_baseline tasks are now tagged packages/system/disks, so the package install can run alone: ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages Skipping is the guard — no disks-tagged task is selected, so the BLOCKED-pending-disk-decision paths are never templated, let alone passed to mkfs. Belt-and-braces, a new disks-tagged assert fires first in a FULL run and names §2 explicitly instead of failing obscurely at mkfs. `tags: never` was deliberately NOT used (it would change the default run). FOOTGUN FOUND AND FIXED: roles/cilium, cluster_verify and flux_bootstrap all hardcoded .cache/dev.kubeconfig — the exact file infra/devcontainer/dev.sh exports as $KUBECONFIG. Once prod became targetable, a prod cilium/verify run would have SILENTLY OVERWRITTEN it, leaving the operator's kubectl quietly answering from PROD. Now cluster_kubeconfig_name (dev.kubeconfig -> prod.kubeconfig). Dev path byte-identical. KNOWN, DOCUMENTED, NOT FIXED HERE: flux-bootstrap.yml retargets but roles/flux_bootstrap shells out to bootstrap/flux/bootstrap.sh with a hardcoded --path=clusters/dev — a bootstrap/ subtree change, not a target-group change (§6, cross-referenced to §7). vpn.egress.gateway.yml is parameterised but prod has no egress group yet and would need its own vaulted PIA credentials. --check against prod is still impossible until the prod vault exists — even the packages-only run needs it. Nothing was run against the dev cluster: --list-hosts/--list-tasks/--syntax-check, plus debug/assert with -c local, only. Serves: R-ENV-01, R-ENV-02, R-ENV-03. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-08-31 23:53:36 -05:00
jakman85andClaude Opus 5	1157c1b071	fix(gitea-actions): wait for dind to ACCEPT connections, not just write certs ... Re-authors the fix stranded on origin/fix/act-runner-dind-readiness (242951a), which applied cleanly but had never been merged. The act_runner startup waited only for the dind TLS CERT FILE (/certs/client/ca.pem), but dockerd writes the certs before it begins accepting TLS connections on :2376. A runner proceeding on the file alone still hard-exits with "cannot ping the docker daemon at tcp://localhost:2376" — observed crashlooping 11 times in ~8 minutes on a fresh pod after a node bounce. Adds a second bounded wait (nc -z -w1 127.0.0.1 2376, 60s cap) mirroring the existing cert-file loop, so the runner starts only once the daemon actually answers. busybox nc in the Alpine-based gitea/act_runner:0.2.11 supports -z/-w. This Deployment is the ENFORCED MERGE GATE for the whole repo, so it ships alone. A REPORTED "latent envsubst bug" in this file was INVESTIGATED AND DISPROVEN. The claim was that Flux envsubst expands bare `$i`, making the existing `[ "$i" -ge 60 ]` bound a silent no-op. Running the real `flux envsubst` (v2.4.0) shows bare `$i` is NOT substituted — even with i set in the environment, `$i` passes through literally while `${i}` expands. This matches the repo's own contract in clusters/base/apps-vms.yaml, which was itself written after empirical probing: "`$VAR` -> left alone by envsubst => bare shell vars need NO escaping." The existing 60s bound was always real. NO escaping was applied: `$${i}` would be an ACTIVE REGRESSION, correct only on the envsubst path — on a plain `kustomize build`/`kubectl apply -k`, sh reads `$$` as its own PID and the bound compares garbage against 60. Bare is correct on both paths. A comment now records the real contract so nobody "fixes" the bare form into escaping. NEAR MISS worth recording: the first draft of that comment spelled the dangerous forms out literally, and `flux envsubst` HARD-FAILED on it — "unable to parse variable name". Flux's envsubst runs over the entire manifest INCLUDING YAML COMMENTS, so a braced name containing punctuation is a fatal parse error that would have failed the whole apps-wave-1 Kustomization — breaking every merge in the repo. From a comment. The shipped comment names the forms in words; the file now contains no braced-dollar sequence at all. Verified without a cluster: flux envsubst over the file is byte-identical (exit 0, and --strict 0); post-envsubst YAML parses with containers unchanged; `sh -n` clean; a dry-run with a stubbed failing nc fires the stage-2 bound and exits 1; render==golden green (gitea-actions.yaml is an EXTRA_UNITS entry — render.py places the filename into a wave and never emits its bytes, so it is not among the 10 compared goldens). Follow-up documented in-file, deliberately NOT done here: convert dind to an initContainers sidecar (restartPolicy: Always) with a startupProbe that execs `docker info` over TLS — NOT a bare tcpSocket probe, which would reopen the same "port open but certs not ready" gap stage 1 closes — periodSeconds 1 / failureThreshold 60 to preserve the cap, after which runner drops its command: override. k3s v1.36 has GA sidecars, so this is the correct long-term shape; it ships separately because a failure in the merge gate must be unambiguous. Serves: R-GITOPS-01, R-TEST-03. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-08-31 23:49:43 -05:00
jakman85andClaude Opus 5	6e77150ae1	fix(ci): kill the false-green base resolution; cover R-SEC-01/02 + R-TEST-06 (L0) ... Unblocks the enforced Gitea gate, which correctly BLOCKED this branch on ['R-SEC-01','R-SEC-02','R-TEST-06'] being touched but uncovered. Tests were authored; the coverage rule was NOT narrowed. === THE FALSE GREEN (root cause) === `make -C ci gate` locally reported PASS having checked NOTHING. resolve_base() validated nothing, and EVERY way of failing to find a base degraded silently to an empty touched set + exit 0: an `HEAD~1` last resort, a two-dot diff fallback masking a failing three-dot diff, and no check that HEAD is even AHEAD of the base. Run while sitting on `main` (main == origin/main), `origin/main...HEAD` is empty -> "no changed files vs base" -> the empty set means run.sh is invoked with NO --requirements -> coverage.py's --require gate never runs at all. Green by vacuum. Now: candidates are BASE_REF > GITHUB_BASE_REF > origin/main > main, each verified to resolve. HEAD~1 and the two-dot fallback are gone. Unresolvable base / no merge base / failed diff / HEAD-not-ahead -> loud banner, EMPTY stdout, exit 3, and a machine-readable reason (on-base | no-base | diff-failed). gate.sh turns exit 3 into an UNDECIDED decision: `on-base` -> "NOT APPLICABLE — no branch under gate" (exit 0 so push-to-main runs don't break, but it never reads as PASS); `no-base`/`diff-failed` -> BLOCKED. It also now prints the diff base it used. A stale local main warns (widens, never falsely greens). Also fixed `make -C ci test`, which was SWALLOWING failing suites. Regression: ci/tests/touched-requirements-base.test.sh — 31 assertions over 8 throwaway-repo scenarios including the exact false-green case, an already-merged branch, no base ref, a single-commit repo, bogus BASE_REF, a shallow GITHUB_BASE_REF, and origin/main-preferred-over-stale-main. === NEW L0 STATIC LAYER (conformance/tests/l0-static/) === Where docs/14 actually puts these invariants. Each check.sh really executes offline via `run.sh --layer l0` and is wired into ci/l0.sh, so it runs on every PR. No scaffolds. R-SEC-01 no-plaintext-secrets (165 assertions / 445 tracked files): .sops.yaml holds only PUBLIC age recipients; every committed Secret is SOPS-wrapped with a mac and ENC[...] values; ansible vault.yml files are $ANSIBLE_VAULT; no secret-valued ansible key carries a literal outside a vault; no PEM key / age secret key / kubeconfig client-key-data is tracked; Flux declares sops decryption; the disallow-plaintext-secrets policy Enforces and has a proving `result: fail` row. Exemptions are SELF-POLICING, not path allowlists — a .example must be referenced by no kustomization and hold only placeholders, and the Kyverno fixture is exempt ONLY while a test asserts the policy rejects it. Proven against 7 injected violations, including deleting the negative row and weakening Enforce->Audit. R-SEC-02 least-privilege-rbac (41 assertions): no wildcard apiGroup/resource/verb and no escalate|bind|impersonate in any in-repo Role/ClusterRole; no ServiceAccount bound to cluster-admin or a wildcard role; the conformance runner's own test identity is scoped and holds no rights over RBAC objects; the default-deny/host-namespace/egress-class policies Enforce and each carry a negative row. Proven against 6 injected violations. R-TEST-06 spec-immutability-enforced (16 assertions): drives the REAL ci/spec-immutability.sh against HERMETIC throwaway spec repos and asserts its exit code in every state — clean-at-pin 0, edits under docs/ or decisions/ 1, untracked addition 1, unadopted drift 1, explicit pin bump 0, ABSENT spec repo 3 (honest skip, never green), unpinned SPEC_PIN 1 — plus that gate.sh sets BLOCK=1 on its failure, so the mechanism is load-bearing. The real spec repo is never written to. Each check documents what it does NOT cover (runtime decryption, git history, runtime least-privilege, branch protection) rather than overclaiming. status.py now reads check.sh too, so a placeholder L0 body still outranks an `executable: true` annotation. Coverage 37/51 -> 40/51 (11 gaps); executable tests 32 -> 35; scaffold-only stays 5 — no sixth scaffold was added. Serves: R-SEC-01, R-SEC-02, R-TEST-01..06. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-08-31 23:26:49 -05:00
jakman85andClaude Opus 5	9f59c36000	feat(ansible): node_health role — read-only hardware health for prod + dev nodes ... Operator ask: implement the roles needed to check prod node hardware health, and run them to find out our state. This is that role; the run's findings are below. roles/node_health is a read-only sibling to cluster_verify (cluster health vs hardware health), styled after node_baseline. One unprivileged collector script does the work in a single SSH round trip — reading /sys/class/dmi, thermal + hwmon, EDAC, cpufreq + thermal_throttle, /proc/pressure, /sys/block incl. SCSI ioerr/iotmo counters, lsblk, lscpu, mounts + statvfs, per-NIC stats, resolv.conf. THE DESIGN POINT: an unreadable field is emitted as null WITH A RECORDED REASON, never as zero. The summary ends with a "NOT READ (and why) — these are UNKNOWNS, not passes" section. A fake zero on a reallocated-sector count is exactly the kind of false green this project refuses. 22 checks report PASS/WARN/FAIL/SKIP/INFO; every SKIP says why. Report-only by default; thresholds live in defaults/main.yml. Portable across bare metal AND VMs — validated against k3s-prod-1 (bare metal, no root) and k3s-dev-1 (VM, root). VM-absent things (SMART on virtio, coretemp, EDAC) skip cleanly, and iSCSI/Longhorn volumes are split out of the physical-disk set so a live replica is never mistaken for a spare device. playbooks/node-health.yml takes its target group as a variable (-e node_health_hosts= k3s_prod), deliberately unlike the other playbooks that hardcode k3s_servers — see the group-name gap already flagged in hosts.ini. Two become facts found the hard way and documented in the role: a group_vars `ansible_become: true` OUTRANKS `become: false` as a play/task keyword, so unprivileged tasks must override the variable; and `failed_when: false` does NOT catch "Missing sudo password" — only `ignore_errors` does. FIRST RUN, k3s-prod-1, 2026-08-31: 0 FAIL · 3 WARN · 12 PASS · 3 SKIP · 4 INFO. Box is healthy, cool and unstressed: i7-9700T 8C/8T, VT-x + /dev/kvm (KubeVirt OK), zero throttle events, hottest sensor 33C, PSI 0.00 across the board, 31.1 GiB RAM, eno1 up at 1000 Mb/s, BIOS R21 02.26.00, no reboot pending. DRIVE HEALTH IS STILL UNKNOWN — the headline. SMART did not run, blocked TWICE over: no vaulted ansible_become_password for prod, AND smartmontools is not on the node at all (node_baseline has never run there). Power-on hours, wear, reallocated/pending sectors and drive temperature all remain unmeasured on a ~2011 consumer SSD. WARNs: kernel ioerr_cnt=6 on sda (read cautiously — four freshly attached iSCSI disks on dev each showed a flat 7, suggesting a fixed per-device discovery cost, not media trouble; attribution needs dmesg/root); 0.95 GiB swap still active (node_baseline does the swapoff); one IPv4 resolver only (the other two upstreams are IPv6 Google), which refines rather than resolves the DNS-SPOF finding. ESCALATION — ECC status is UNKNOWABLE on this hardware: /sys/devices/system/edac/mc exists but no controller is bound. Non-ECC consumer RAM. Silent memory corruption on a prod node would be undetectable. Recorded as a SKIP with reason, not a pass. Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-08-31 23:20:25 -05:00
jakman85andClaude Opus 5	677f9ea5f0	docs: CI is the enforcement point; local gate produced a FALSE GREEN ... Operator directive: conformance and gates are enforced at the Gitea runner / CI-CD level — use it. The authority on mergeability is the Actions run, not a local `make -C ci gate`; local runs are a fast pre-check only. Proven immediately on this branch. Same commit (a1fb31a): local -> "no changed files vs base; empty touched-requirement set" -> MERGE GATE: PASS CI -> 17 touched requirements -> REQUIRE: FAILED, uncovered ['R-SEC-01','R-SEC-02','R-TEST-06'] -> MERGE GATE: BLOCKED The local run resolved an empty diff base, checked nothing, and reported green. A local false green is this project's most dangerous failure mode — it hid real coverage debt. Fix in progress so local agrees with CI or announces loudly that it cannot determine a base; an honest noisy skip beats a quiet false pass (ci/README.md exit-code contract). The gate was RIGHT: we authored a vaulted-secret flow (R-SEC-01) and status tooling reasoning about spec immutability (R-TEST-06) while those requirements had NO covering test. Tests are being authored; the rule is NOT being narrowed. Also records how to read CI from a workstation — the API paths are non-obvious on this Gitea version (actions/runs/<run>/jobs 404s; use actions/tasks + actions/jobs/<id>/logs, noting the task-id -> job-id offset). OPERATOR ATTESTATIONS: DR is proven (Velero -> Proxmox server, restore demonstrated) and dev-cluster resilience is proven (slow only because Longhorn is on HDD; etcd is on its dedicated SSD). Both recorded as fact. Kept distinct from automated proof: the status report's R-STATE-03 finding is TEST debt, not capability debt — the system does the thing, the suite does not yet assert it. The attestation does not close the requirement and the gate must not be softened to make it appear closed. Serves: R-TEST-03, R-TEST-05, R-TEST-06, R-STATE-03. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-08-31 23:05:57 -05:00
jakman85andClaude Opus 5	a1fb31a7ef	docs(prod): record the decided prod address plan; packages-in-role correction ... PROD-PROVISIONING.md §4 now carries the full operator-decided address table (nodes .10/.11/.12, control-plane VIP .9, ingress Gateway VIP .20, Cilium LB pool .20-.30) with the two invariants that make it safe: the control-plane VIP sits outside the LB pool so kube-vip and LB-IPAM cannot contend, and GATEWAY_VIP is the FIRST address of the pool because the shared Gateway pins nothing and LB-IPAM hands it the first free address. §5 no longer tells the operator to hand-install smartmontools — package installation is the node_baseline role's job (operator ruling). §0 gains the node 2/3 addresses. Task: T-7.1. Gate: MERGE GATE: PASS. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-08-31 22:44:10 -05:00
jakman85andClaude Opus 5	1252823d80	feat(conformance): repo-derived spec status report (conformance/run.sh --status) ... Operator ruling: "I care about the state of the repository and what the status of each spec is based on that state" — not a hand-maintained task board. This is that report. conformance/lib/status.py (stdlib-only, --selftest) sits beside lib/coverage.py and IMPORTS it rather than forking the registry/test parsers. Driven by a new `conformance/run.sh --status`; artifacts land in _artifacts/status/ alongside coverage and are git-ignored by the existing convention. Exit codes follow ci/lib.sh (0/1/3); --strict-coverage also applies the status hygiene gate. Signals derived per requirement: is there an implementing artifact; is it RECONCILED (walk resources: from clusters/dev, expand each Flux Kustomization's spec.path, recurse — so a commented-out resource correctly does not count); is there a tagged test and is its body executable or a TODO scaffold (TODO markers OUTRANK an executable: "true" annotation — ambiguity never upgrades a row). DELIBERATELY NOT DERIVED, and reported as such rather than defaulting to green: proof-of-pass (nothing tracked in git records a run result — _artifacts/ and results/ are git-ignored) and cluster liveness. "Reconciled" means Flux WOULD apply it, not that it is healthy. PROVEN is reported as "not derivable" for all 51. WHAT IT SHOWS THAT coverage.md DOES NOT. Coverage reports 37/51 covered, 0 safety-critical gaps. Three of those claims do not survive repo state: 1. 5 of the 37 "covered" requirements have only a TODO-body scaffold (R-STATE-03, R-OBS-02, R-OBS-04, R-GITOPS-03, R-GITOPS-02). Coverage counts a tagged file; it never looks inside it. 2. R-STATE-03 is SAFETY-CRITICAL and its L3 fault test asserts nothing. Coverage reports "0 safety-critical gaps" only because dr-restore/chainsaw-test.yaml exists and carries an L3 label. --strict now fails on this. 3. R-UPD-01/02/03 are NOT reconciled: platform/image-automation/ is commented out of platform/kustomization.yaml. Coverage shows them covered with executable L3 tests, which reads as delivered; automated updates are actually 0/3 wired+testable. Headline: 51 requirements | reconciled 20 (+12 partial) | executable test 32 | wired+testable 22 (+8 partial) | scaffold-only 5 | PROVEN 0 (not derivable). Also surfaces 18 requirements with no reconciled manifest citing their R-ID — listed as a traceability gap (R-TEST-04) rather than guessed either way. Annotating those manifests makes the report self-correct, so it acts as a forcing function instead of another hand-maintained map. Serves: R-TEST-01..06. Gate: MERGE GATE: PASS. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-08-31 22:44:10 -05:00
jakman85andClaude Opus 5	666cc0d809	feat(ansible): prod node inventory + vaulted become; smartmontools into node_baseline ... Prod inventory scaffolding for the three bare-metal nodes, and the become posture the operator ruled for production. BECOME: prod does NOT get dev's NOPASSWD sudo. Prod keeps a real sudo boundary and supplies the password to Ansible as an ansible-vault-encrypted `ansible_become_password` in inventory/group_vars/k3s_prod/vault.yml, mirroring the existing hypervisor and proxmox_hosts groups. The decrypting passphrase stays outside the repo (~/.config/home-k3s/ansible-vault-pass), so plays still run unattended with no plaintext secret in git. Committed here: README.md + vault.yml.example only — the operator creates the encrypted vault (`ansible-vault create`), as with every other vaulted group. PACKAGES BELONG IN THE ROLE (operator ruling): smartmontools is now declared in node_baseline_packages, not hand-installed on a node. Drive health tooling is a permanent part of a bare-metal node's baseline — dev nodes are VMs and never needed it. Same click-ops-never principle as the rest of the repo. ADDRESSING: nodes .10/.11/.12, reached DIRECTLY on the LAN (no pfSense NAT hop, unlike dev's 192.168.2.102:225x mapping), so ansible_host is the real node IP. kube-vip control-plane VIP is 192.168.2.9 — deliberately BELOW the nodes and OUTSIDE the Cilium LB pool (.20-.30) so kube-vip and LB-IPAM can never contend for an address. The ingress Gateway VIP (.20) and the LB pool are Flux/cluster-vars concerns and deliberately do not appear in Ansible. BLOCKED-BY-DESIGN: node_baseline_longhorn_device / node_baseline_etcd_device are present but set to an obviously invalid /dev/BLOCKED-... path pending the operator's disk-layout decision (node 1 is a single SATA SSD; the role's /dev/vdb,/dev/vdc defaults are virtio names that do not exist on bare metal). Leaving them unset would silently fall back to those equally-nonexistent defaults — a confusing failure that looks like a wiring bug. This fails just as hard and says exactly why. KNOWN FOLLOW-UP: roles/k3s_server and every node-host playbook hardcode the literal group names k3s_primary/k3s_secondary/k3s_servers, so the k3s_prod_* groups are not selected by them as-is. Harmless now (nothing can accidentally run against prod) but the "same roles, unchanged" reuse R-ENV-02 promises needs the target group parameterized before §6 of PROD-PROVISIONING.md can execute. Documented in hosts.ini and both READMEs. Serves: R-ENV-01, R-ENV-03, R-SEC-01. Task: T-7.1. Gate: MERGE GATE: PASS. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-08-31 22:44:10 -05:00
jakman85andClaude Opus 5	3f980d2a17	feat(env): tokenize hardcoded dev LAN addresses into cluster-vars (R-ENV-02) ... R-ENV-02 promises dev and prod differ ONLY by scoped cluster-vars overrides. In fact dev LAN addresses were hardcoded into base manifests, so clusters/prod would have come up ANNOUNCING DEV ADDRESSES (192.168.1.x) on the prod LAN (192.168.2.x). New cluster-vars knobs (base default = dev literal, per the file's existing convention): K3S_API_HOST platform/cilium/helmrelease.yaml (+ the values.yaml mirror) CILIUM_LB_POOL_START platform-config/cilium/l2-ip-pool.yaml CILIUM_LB_POOL_STOP " UNIFI_L4_VIP apps/vms/unifi/50-service-l4.yaml CRAFTY_L4_VIP platform/crafty/service-l4.yaml S3_BACKUP_ENDPOINT platform/keycloak/postgres-cluster.yaml, apps/wave-0/{nextcloud, semaphore}-db.yaml, and platform/velero/helmrelease.yaml Also fixes GATEWAY_VIP's base default, which was still the RETIRED libvirt address 10.73.0.240, and adds the GATEWAY_VIP patch that clusters/prod was missing entirely. clusters/base/apps-vms.yaml gains postBuild.substituteFrom (it was the one consuming Kustomization without it, so a ${TOKEN} there would have shipped as a literal string). The envsubst contract is documented there: ${VAR} substitutes and escapes as $${VAR}; $VAR and $(cmd) are left alone; ${ARR[0]} hard-errors. The three UOS_URL expansions in the unifi cloud-init are escaped accordingly. PROD ADDRESS PLAN (operator, 2026-08-31): nodes .10/.11/.12 · control-plane VIP .9 (Ansible-side) · GATEWAY_VIP .20 Cilium LB pool .20-.30 CONTAINMENT INVARIANT, now documented at the pool and in the prod overlay: GATEWAY_VIP MUST equal CILIUM_LB_POOL_START. The shared Gateway pins no address, so Cilium LB-IPAM gives it the first free address of a matching pool block. An initial prod proposal put .20 outside a .21-.30 pool; the Gateway would have come up on .21 while split-horizon coredns-custom kept answering *.home.keithjackson.dev with .20 — an address nothing announces — blackholing in-cluster OIDC hairpin traffic while reporting healthy. The pool was widened to .20-.30. The control-plane VIP (.9) stays outside the pool so kube-vip and LB-IPAM can never contend. Values prod still cannot know stay INVALID `TBD-PROD-*` placeholders on purpose: base defaults are dev values, so an omitted patch means prod silently inherits a dev address — far worse than a loud failed apply. DEV IS BYTE-IDENTICAL. Verified by kustomize build over 13 paths before/after, then flux envsubst with the dev cluster-vars values applied to both sides: zero drift. The only deltas are the intended additive keys in the overlay ConfigMap itself. No unresolved ${TOKEN} survives. render-golden green (10 goldens); yamllint clean. Serves: R-ENV-02. Task: T-7.1 (prerequisite). Gate: MERGE GATE: PASS. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-08-31 22:43:36 -05:00
jakman85andClaude Opus 5	e3c44368c8	docs(prod): correct sudo posture to vaulted become; record status-surface ruling ... Two operator corrections: 1. PROD DOES NOT GET NOPASSWD SUDO. My initial checklist recommended mirroring dev's passwordless sudo onto prod — rejected, correctly. Dev's NOPASSWD is a convenience that must not propagate to production. Prod instead keeps a real sudo boundary and drives Ansible `become` with an ansible-vault-encrypted `ansible_become_password` in inventory/group_vars/k3s_prod/vault.yml, mirroring the EXISTING hypervisor and proxmox_hosts groups. The decrypting passphrase stays outside the repo at ~/.config/home-k3s/ansible-vault-pass (ansible.cfg :: vault_password_file), so plays still run unattended with no plaintext secret in git. 2. THE TASK BOARD IS NOT THE STATUS SURFACE — THE REPOSITORY IS. Operator ruling closes the escalation raised in the previous commit. Requirement/spec status is to be DERIVED from repo state (what is authored, reconciled, and covered by a green requirement-tagged test), not from a hand-maintained board. The spec board is left as a historical seed; no agent edits it and no SPEC_PIN churn is spent on it. ci/spec-immutability.sh keeps its full strictness — no path exception was added, since weakening an R-TEST-06 gate for a bookkeeping file would be a defect, not a fix. A repo-derived spec status report becomes the artifact of record alongside the coverage report and this log. Serves: R-ENV-03 (prod provisioning posture), R-SEC-01 (no plaintext secrets), R-TEST-06. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-08-31 22:29:39 -05:00
jakman85andClaude Opus 5	574088dfeb	docs(prod): open Phase 7 (prep only) + T-7.1 bare-metal provisioning checklist ... The operator added the first production node (k3s-prod-1, 192.168.2.10 — HP EliteDesk 800 G5 Mini, i7-9700T/31GiB, Debian 13, VT-x + /dev/kvm) and opened roadmap Phase 7 for PREP WORK ONLY: no k3s install until the R-ENV-02 blockers close and the gate is green. Records the Phase-7 entry-gate position honestly: the operator has declared hardware ready, but "Phases 1-6 proven in dev/test" is NOT met (37/51 conformance; R-ENV-01/02/03 themselves uncovered). Proceeding is an explicit, accepted risk — not an oversight. Adds infra/PROD-PROVISIONING.md (T-7.1, design-only): firmware/BIOS, disk layout, OS, network, access, Ansible reuse, repo blockers, build sequence, Phase-7 exit criteria. Key findings recorded in DECISIONS-LOG: - R-ENV-02 is violated in the base — dev LAN addresses hardcoded into base manifests (cilium k8sServiceHost, the Cilium LB pool, crafty/UniFi LB IPs, 3 S3 backup endpoints). Prod is on 192.168.2.x, so prod would build WRONG today. - clusters/prod never sets GATEWAY_VIP -> would inherit the base default 10.73.0.240, an address on the RETIRED libvirt network. - No NOPASSWD sudo on the prod node -> all Ansible fails at its first become. - node_baseline device defaults are virtio (/dev/vdb,vdc) and absent on bare metal. - ESCALATION: agents/task-board.md is structurally unmaintainable — AGENTS.md designates it the one agent-editable surface, but ci/spec-immutability.sh fails the gate on ANY spec-repo dirt with no path exception. Explains the all-todo board after six phases. NOT worked around: exempting it would weaken an R-TEST-06 gate. Operator decision. Operator decisions: Phase 7 open (prep only); wait for all 3 nodes before cluster-init; check drive health before choosing the single-SSD disk layout. Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-PLAT-01/03. Task: T-7.1. Gate: MERGE GATE: PASS. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-08-31 22:17:51 -05:00
jakman85andClaude Opus 4.8	c182888133	docs: PM handoff — 2026-08-31 current-state, gotchas + next steps; PHASE1-STATUS 8 vCPU ... Appends the definitive current-state/handoff entry to DECISIONS-LOG (SSO fixed via CVE-2025-30162 egress allow, 5 forward-auth UIs gated, 8 vCPU, Gitea Actions CI live, external-snapshotter vendored; operational gotchas; prioritized Next Steps → Longhorn-SSD, L2/L3-CI, forward-auth long-term, Phase-6 onboarding). Updates the reconnect guide (6→8 vCPU) and points new agents at the handoff. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-31 21:23:04 -05:00
jakman85andClaude Opus 4.8	7a79cea94a	fix(flux): vendor external-snapshotter manifests locally (stop remote-base build flake) ... The kustomization pulled two REMOTE github bases (?ref=v8.2.0) that Flux's kustomize-controller git-fetched on every reconcile; those fetches intermittently exceeded the 27s timeout -> 'kustomize build failed' BuildFailed events -> the whole platform tier stalled whenever GitHub was slow. Vendored the exact v8.2.0 output to local crds.yaml + snapshot-controller.yaml (built with kubectl kustomize; images transformer still re-pins the controller to v8.2.0), so the build is now offline + deterministic. No change to what's deployed. platform/kubevirt has the same latent pattern. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-31 10:54:43 -05:00
jakman85andClaude Opus 4.8	d1a547fd15	docs(plan): L2/L3 conformance CI wiring plan; gate.yml placeholder -> plan pointer ... HAND-OFF 2 (L2/L3 conformance wiring) — DEFERRED with a concrete plan, per the hand-off's escape hatch (too risky to wire live cluster access from CI autonomously; do NOT hand CI a broad kubeconfig). The intended seam (in-cluster Chainsaw Job under its own scoped SA; CI holds only a narrow trigger credential) is mostly scaffolded but not safe to finish autonomously: - ci/conformance.sh still runs run.sh locally rather than triggering the in-cluster Job. - conformance/incluster/chainsaw-job.yaml has no repo-clone init-container and the stock chainsaw image lacks bash/python3/kubectl/git that run.sh needs. - No trigger SA/RBAC/credential exists, and the core blocker is a privilege-escalation design question: any `create jobs` credential in the conformance namespace can create a Job running as the privileged conformance-runner SA. This needs an admission constraint (recommended: narrow trigger SA + a Kyverno policy pinning the admissible Job shape), or a GitOps-native zero-credential trigger — an operator decision. - conformance/incluster/ isn't wired into Flux; Gitea vars/secrets aren't provisioned. docs/plans/l2-l3-conformance-ci-wiring.md captures current state, the security analysis, the functional gaps, a file-by-file step sequence, validation/rollback, and the open decisions. gate.yml's conformance placeholder now points to it (still if:false, honest). Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-31 09:01:18 -05:00
jakman85andClaude Opus 4.8	27bc02471c	ci: exclude comment-only prose from touched-requirements scan; gate.yml -> make -C ci gate ... HAND-OFF 1 (make -C ci gate parity). touched-requirements.sh gained a third scoping rule: after stripping the diff marker, whole-line YAML/shell `#` comments are dropped BEFORE the R-ID grep. A requirement cited only in a code-file comment is prose (a REFERENCE), not a declaration -- the same logic rule 2 already applies to `*.md`. This unblocks the self-referential trap where editing gate.yml's header (which cites R-TEST-06, a known-uncovered ID enforced by the spec-immutability STAGE, not a Chainsaw test) re-attributed R-TEST-06 so the coverage `--require` check false-blocked the merge. Conservative: only whole-line comments are dropped; an R-ID on a real code/spec line is still detected, and authoritative declarations never live in a bare comment. gate.yml now invokes the aggregate `make -C ci gate` (ci/gate.sh) instead of the individual stage scripts, so CI runs byte-for-byte the same gate a human runs in the devcontainer (R-TEST-02) and applies the R-TEST-03 decision. A BLOCKED gate exits non-zero and fails the job. The cluster-gated L2/L3 stage SKIPS honestly (no CLUSTER_AVAILABLE). spec-immutability is retained as an explicit early UI step. Adds ci/tests/touched-requirements.test.sh (+ `make -C ci test`) asserting rule 3: comment-only R-IDs (added or removed) are not attributed while real-line R-IDs are. ci/lib.sh REPO_ROOT is now env-overridable so tests can run the real script against a throwaway git repo (default unchanged). Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-31 13:54:29 +00:00
jakman85andClaude Opus 4.8	b2752aec48	feat(auth): gate jackett/qbittorrent/radarr/sonarr UIs with oauth2-proxy sidecar (rollout) ... Roll the verified Option-B per-app oauth2-proxy sidecar out to the remaining four forward-auth apps after the pihole canary was confirmed fail-closed (unauthenticated -> 302 to Keycloak, pod 2/2, clean sidecar OIDC discovery). Each app's Gateway route now terminates at its oauth2-proxy sidecar (:4180, --allowed-group=media-users, --upstream=http://127.0.0.1:<uiport>); the gateway-ingress CiliumNetworkPolicy allows the Gateway to reach ONLY :4180, so the bare UI is never served un-authenticated (fail-closed; R-IDP-02/06). The apps' own app/API ports stay in-Service for EAST-WEST callers (arr -> qbittorrent:8080 / jackett:9117) via the existing reciprocal-ingress allows. - keycloak: confidential jackett/qbittorrent/radarr/sonarr clients (realm-import + realm-clients mirror), provisioned live via kcadm (groups membership mapper). - secrets: four SOPS/age <app>-oauth2-proxy-credentials, wired into apps/wave-0 and apps/wave-1 by the renderer's sidecar-creds wave rule. - ci: remove the temporary canary deferral (all five goldens now reproduce). Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-31 08:45:46 -05:00
jakman85andClaude Opus 4.8	72d4720d58	feat(auth): gate pihole UI with per-app oauth2-proxy sidecar (canary) ... Close the un-gated-UI security finding for forward-auth apps (R-IDP-02) by rendering the Option-B per-app oauth2-proxy reverse-proxy sidecar that crafty proves working. catalog/render.py now emits, for any forward-auth app with an exposed UI: the oauth2-proxy sidecar (upstream http://127.0.0.1:<uiport>, :4180, --allowed-group per auth.group), an authproxy :4180 Service port, an HTTPRoute backendRef -> :4180, and a gateway-ingress CiliumNetworkPolicy that allows the Gateway to reach ONLY :4180 (fail-closed: the bare UI is never served un-authenticated). oidc/native/none apps render byte-identical. This is the CANARY: only pihole's golden carries the sidecar and is verified fail-closed in-cluster first. jackett/qbittorrent/radarr/sonarr are deferred to the rollout commit (temporarily listed in ci/render-golden.sh) so their UIs keep serving exactly as today until their clients+secrets land. - keycloak: confidential pihole client (realm-import + realm-clients mirror), provisioned live via kcadm (groups membership mapper, bare names). - secret: SOPS/age pihole-oauth2-proxy-credentials (client-id/secret/cookie), wired into apps/wave-0 via the renderer's new sidecar-creds wave rule. - L1 render goldens regenerated for the sidecar (forward-auth fixtures). Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-31 08:36:03 -05:00
jakman85andClaude Opus 4.8	a04251bbc8	docs: plan for Longhorn SSD tiering (hot-tier move off HDD) ... Investigation + phased implementation plan to move the churn-heavy, latency-sensitive Longhorn volumes (Prometheus, the CNPG Postgres DBs, Loki, gitea) off the slow/failing HDD 'Storage' pool onto a tagged SSD-backed Longhorn disk on local-zfs, ending the reboot rebuild storm. Key finding: the recorded 'SSD does not fit Longhorn' blocker was based on the 384 GiB provisioned figure; actual working set is ~79 GB cluster-wide (~26 GB/node) since all bulk media is on NFS, not Longhorn. Hot tier is ~51 GB actual / 114 GiB provisioned and fits the ~323 GiB SSD AVAIL with headroom. Read-only investigation; no live storage touched. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-31 08:27:21 -05:00
jakman85andClaude Opus 4.8	d60e5531b1	test(conformance): correct stale ImageVolume-gate rationale for k3s v1.36.4 ... The three KubeVirt VM conformance tests (vm-lifecycle, vm-default-deny, vm-data-survival) carried comments asserting the ALPHA `ImageVolume` feature gate is DISABLED "on this k3s v1.31 cluster" and that the apiserver therefore prunes the container-disk `image` volume, breaking the containerDisk boot path. The cluster was upgraded v1.31 -> v1.36.4. Revalidated the gate live: - kubernetes_feature_enabled{name="ImageVolume"} = 1 (enabled by default; stage="" — no longer ALPHA) - a server-side `kubectl apply --dry-run=server` of a Pod with an `image` volume RETAINS the volume + mount instead of pruning them. So the old premise is stale: containerDisk roots would boot on 1.36.4. The tests never asserted the gate state and never used containerDisk — they boot from CDI DataVolume roots, which remains the correct choice because it is the spec-required, rebuild-surviving disk model (docs/06, R-STATE-01); an ephemeral containerDisk could never satisfy data-survival. Test logic is therefore unchanged; only the explanatory comments are corrected (v1.31 -> v1.36.4, and the "containerDisk is broken here" rationale marked stale with the live evidence). Merge gate: PASS (lint, l0, render-golden, l1 green). Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-31 01:19:46 -05:00
jakman85andClaude Opus 4.8	b745bab26b	docs(plan): evaluate Cilium 1.20.1 native Gateway ExternalAuth vs oauth2-proxy ... Research + implementation plan (no live change) on whether Cilium 1.20.1's native Gateway API ExternalAuth filter (GEP-1494) can replace the un-wired forward-auth CEC template and Crafty's oauth2-proxy sidecar. Verdict: DEFER. The filter exists in 1.20.1 code (shipped 1.20.0, PR #45739) but is EXPERIMENTAL-channel Gateway API; the cluster runs the standard-channel CRDs (verified: HTTPRoute filters[].type enum lacks ExternalAuth), so it is not even installable today. Also: fail-open bug (#47178, fixed #47265) in a brand-new security filter, no 1.20 docs, and the ext_authz hop would hit the same enforce_policy_on_l7lb hairpin re-evaluation just fixed for Keycloak (68690e5). Plan is phased/reversible (pihole canary first) and keeps the working sidecar. Resolves the DECISIONS-LOG open item on ext_authz workaround relevance under 1.20.1. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-31 01:19:36 -05:00
jakman85andClaude Opus 4.8	676c37a247	tofu: node vCPU 6 -> 8 (CPU spike headroom) ... Steady-state CPU PSI 15-23% + 15-min load 5.6-8.0 on 6 cores (run queue at/over core count) left no headroom for bursty workloads. Proxmox host has 24 cores at ~4% load, so 3x8 = 24 vCPU is 1:1 with zero oversubscription. In-place tofu update (cores change is pending until a per-node VM stop/start); rolled one node at a time to keep etcd quorum. Comment also corrected 16->24 GiB (stale). Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-30 23:47:49 -05:00
jakman85andClaude Opus 4.8	8a424bcba3	test(l1): regenerate render goldens for the keycloak-egress renderer change ... The container-workload and egress-class L1 fixtures use auth.mode=forward-auth, so the renderer now emits the keycloak backend egress allow for them too. Regenerate the committed goldens to match the canonical render. MERGE GATE: PASS. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-30 23:35:20 -05:00