home-network-k3s-impl

0 Fork 0

Author	SHA1	Message	Date
jakman85andClaude Opus 5.5	d8b25a67fe	fix(vuln): survive OOM kills, isolate bad upstream strings, label findings per line Review findings on the integrated scanner: - An OOM kill no longer wedges the loop: inFlight is persisted before each scan, and on restart that image is marked failed and sits out full runs for 20h. - One bad upstream string can no longer freeze the report: safe_url requires an exact lowercase scheme, acceptance reasons are bounded, /triage urls use the same rule, and report.build repairs or drops any entry the schema would refuse. lastCompleteRunAt advances only after a report is actually published. - The private registry reports credential-required, not failed: VULN_PRIVATE_REGISTRIES names it, and Trivy is not invoked. - loadError clears after a full run against a good inventory. - Every image under platform/vuln-scanner stays out of Renovate auto-merge. - The page labels status and target state per line, and mixed acceptance reads as mixed. The Keycloak client is still an operator action: its key must go in an existing encrypted file that an agent must not modify. platform/vuln-scanner/README.md has the exact steps. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-11-09 12:28:29 -06:00
jakman85andClaude Opus 5.5	d06dff9a19	fix(vuln): no-per-cve-label check also matches escaped-quote label forms Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-11-09 10:29:05 -06:00
jakman85	b1a6fe6503	Merge branch 'wip/vuln-platform' into feat/vuln-visibility	2026-11-09 10:29:05 -06:00
jakman85	b265d08278	Merge branch 'wip/vuln-bot' into feat/vuln-visibility	2026-11-09 10:29:05 -06:00
jakman85andClaude Opus 5.5	b85d1e3ed7	feat(vuln): vuln-scanner platform manifests, Flux tier, alerts, L0/L2/L3 tests Hand-authored platform/vuln-scanner (non-gating Flux tier, digest-pinned Trivy 0.76.1 and python 3.14.9-alpine, read-only RBAC, per-pod oauth2-proxy, internal-only cve.<domain>), five alerts with promtool tests, R-VULN-01..09 registry entries, and vuln-* L0 checks with mutation self-tests. src/ and the bot workflow are integrated separately. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-11-09 09:42:52 -06:00
jakman85andClaude Opus 5.5	50ec2297ee	feat(vuln): scanner service and fixture-driven L1 tests Stdlib-only scanner per docs/17: Kubernetes API inventory, Trivy runner, persisted state, schema-validated report.json, strict acceptances parser, server-rendered page, aggregate metrics, internal API, and a scan loop that stays live while scanning. L1 tests under conformance/tests/l1-rendering/vuln-scanner-* run the actual modules against recorded fixtures with a fake Trivy runner. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-11-09 09:41:23 -06:00
jakman85andClaude Opus 5.5	c3c72c47bb	feat(vuln): CVE triage bot workflow, script and L1 tests Scheduled Gitea Actions bot in the Renovate shape: maintains a counts-only CVE Dashboard issue, derives remediation states from Renovate's dashboard and PRs, ticks approve rows for exact-matched images with an unaccepted CRITICAL with a fix, and posts the heartbeat. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-11-09 09:39:24 -06:00
jakman85	da0515c8c8	Merge pull request 'fix(vpn): enrol keith-phone-home and admit the WireGuard subnet to PiHole DNS' (#158 ) from fix/wg-home-peer-pihole-sources into main gate / gate (push) Successful in 55s Details gate / conformance (push) Skipped Details Reviewed-on: #158	2026-11-09 02:50:04 +00:00
jakman85andClaude Opus 5.5	f13d336630	fix(vpn): enrol keith-phone-home and admit the WireGuard subnet to PiHole DNS gate / gate (pull_request) Successful in 1m12s Details gate / conformance (pull_request) Skipped Details Add the at-home peer (10.90.0.3) and list ${WG_CLIENT_CIDR} in the PiHole DNS frontends' loadBalancerSourceRanges. wg0 is a Cilium device, so tunnel clients reach the source-range check untranslated; a LAN-only list is predicted to drop their DNS. Adds an L0 guard that both ranges are present and every range is RFC1918, and corrects the README's SNAT and IPv6 claims. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-11-08 20:45:48 -06:00
jakman85andClaude Opus 5.5	8fe8d5df8a	feat(dns): PiHole answers an explicit LAN name list, not a zone wildcard gate / gate (pull_request) Successful in 1m14s Details gate / conformance (pull_request) Skipped Details Public names are proxied CNAMEs to the Cloudflare Tunnel since PR #156 and work from the LAN, so they should resolve publicly there too. Replace the address=/${BASE_DOMAIN}/ wildcard with explicit entries: the LAN-only names (derived from every controller:none HTTPRoute), gitea/nextcloud (off Cloudflare's 100 MB / 100 s limits), keycloak/auth, and the L4 VIPs. Add L0 check pihole-lan-names, mutation-tested. Correct docs that still described public names as CNAME to wan. and the cloudflared entrypoint comment. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-11-08 19:51:49 -06:00
jakman85andClaude Opus 5.5	92944486d1	feat(dns): publish public names as proxied CNAMEs to the Cloudflare Tunnel gate / gate (pull_request) Successful in 45s Details gate / conformance (pull_request) Skipped Details Gateway target -> fc81938b-...cfargotunnel.com, --cloudflare-proxied globally, L4 service publishers (unifi, crafty-mc) pinned DNS-only, Plex web UI and Headlamp disowned (LAN/VPN-only). L0 internal-services-publish-no-public-dns gains assertion (9) tying the Gateway target to the tunnel UUID, requiring cloudflare-proxied=false on every service-source publisher, and keeping the tunnel's 404 list and the unpublished set consistent. L2 dns-exposure fixture opts out of proxying. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-11-08 17:55:41 -06:00
jakman85andClaude Opus 5.5	d9ab57f2d1	fix(cloudflared): fsGroup so uid 65532 can read the 0440 credentials Secret gate / gate (pull_request) Successful in 42s Details gate / conformance (pull_request) Skipped Details The pod ran as 65532:65532 with no fsGroup, so the kubelet laid the Secret's credentials.json down root:root 0440 — unreadable to the container. cloudflared logged "permission denied" and then the red-herring "Error locating origin cert". - deployment.yaml: securityContext.fsGroup 65532 (mode stays 0440) - L0 cloudflared-ingress-rules (7): Secret volumes must be readable by the container's run-as uid/groups; mutation self-tested - runbook: permission-denied troubleshooting entry, credential rotation section - decisions/2026-11-08-cloudflared-creds-permission.md Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-11-08 15:49:56 -06:00
jakman85andClaude Opus 5.5	09f219fe7f	fix(cloudflared): auditable Service+ServiceMonitor scrape; placeholder-only template value gate / gate (pull_request) Successful in 42s Details gate / conformance (pull_request) Skipped Details servicemonitor-ports-resolve could not audit a PodMonitor with no in-repo Service, and no-plaintext-secrets requires a template's secret value to be a bare placeholder. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-11-08 12:39:56 -06:00
jakman85andClaude Opus 5.5	aabc8c60b6	feat(cloudflared): Cloudflare Tunnel connector, Phase 1 (isolated, no traffic change) Adds platform/cloudflared: a locally-managed tunnel whose ingress rules live in git. LAN/VPN-only hosts (plex, plex-vip, jackett, pihole, pihole-dns, semaphore, qbittorrent, radarr, sonarr, headlamp, unifi, crafty-mc) map to 404. The apex and *.home go to cilium-gateway-shared.gateway over TLS with matchSNItoHost. Last comes a catch-all 404. Nothing routes to the Kubernetes API. - 2-replica Deployment, hard node spread, non-root, pinned digest, PDB - default-deny plus an egress CNP: Cloudflare edge 7844/443, the Gateway, and Gateway backends by gateway-access=shared minus the LAN-only namespaces (needed because of the CVE-2025-30162 L7LB egress re-check) - PodMonitor plus two alerts that fire on absent series, with promtool tests - its own Flux Kustomization (dependsOn platform-config, wait:false, nothing depends on it), plus SOPS decryption patches in both overlays - placeholder credentials (SOPS-encrypted) so it can merge before enrolment - L0 cloudflared-ingress-rules, and a cloudflared_ scrape owner - runbooks/cloudflare-tunnel.md and decisions/2026-11-08-cloudflare-tunnel.md Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-11-08 12:38:22 -06:00
jakman85	0b0b5abc8e	Merge pull request 'feat(ci): run L0/L1 offline checks, mutation suites and yamllint in bounded parallel pools' (#149 ) from feat/parallel-l0-checks into main gate / gate (push) Successful in 39s Details gate / conformance (push) Skipped Details Reviewed-on: #149 Reviewed-by: Keith Jackson <[email protected]>	2026-11-08 06:53:42 +00:00
jakman85	7be693bf37	Merge pull request 'chore(deps): Update quay.io/oauth2-proxy/oauth2-proxy Docker tag to v7.15.6' (#136 ) from renovate/quay.io-oauth2-proxy-oauth2-proxy-7.x into main gate / gate (push) Canceled after 1m32s Details gate / conformance (push) Canceled after 0s Details Reviewed-on: #136 Reviewed-by: Keith Jackson <[email protected]>	2026-11-08 06:45:31 +00:00
jakman85andClaude Opus 5.5	a775082ae1	feat(ci): run L0/L1 offline checks, mutation suites and yamllint in bounded parallel pools - conformance/lib/run_l1_offline.py: checks run in a thread pool (--jobs / CONF_OFFLINE_JOBS, default min(8, usable CPUs)); results consumed in submission order, so log lines, JUnit, step summary and the 0/1/3 aggregation are unchanged. Exports a smaller MUTATION_JOBS to the children when parallel, so nested pools do not oversubscribe. - keycloak-{passkey,client-secret}-reconciler/mutations.py: anchors checked first (serial), then the --selftest subprocesses run in a pool (MUTATION_JOBS, default min(8, CPUs)); verdicts reported in MUTATIONS order. Output byte-identical to the serial loop. - ci/lint.sh: yamllint over LINT_JOBS contiguous slices concurrently, per-slice output buffered and printed in order; stage rc = highest slice rc. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-11-08 00:18:57 -06:00
jakman85andClaude Opus 5.5	3d7ecbcfe9	fix(conformance): regenerate L1 render goldens for oauth2-proxy v7.15.6 gate / gate (pull_request) Successful in 3m0s Details gate / conformance (pull_request) Skipped Details PR #136 bumps the renderer's OAUTH2_PROXY_IMAGE; the container-workload and egress-class (vpn + internet) goldens still pinned v7.8.1. Regenerated with the check scripts' own invocation; each diff is the image tag only. Also updates the crafty README's current-pin row. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-11-07 23:58:29 -06:00
jakman85andClaude Opus 5.5	9c51f60a35	feat(conformance): L0 guard that the external-dns annotationPrefix matches the tree gate / gate (pull_request) Successful in 2m58s Details gate / conformance (pull_request) Skipped Details Add assertion (8) to internal-services-publish-no-public-dns. The external-dns HelmRelease must declare annotationPrefix explicitly, and every external-dns annotation in apps/, platform/, runbooks/ and catalog/render.py must use that prefix. The v0.23.0 default-prefix change on 2026-11-07 ignored the controller: none, target and hostname annotations while assertions (1)-(3) stayed green. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-11-07 17:56:12 -06:00
jakman85	262c6e4a97	Merge branch 'main' into feat/keycloak-passkey-reconcile gate / gate (pull_request) Successful in 2m56s Details gate / conformance (pull_request) Has been skipped Details	2026-10-30 05:56:00 +00:00
jakman85andClaude Opus 5.5	04eb38584a	fix(conformance): workload-mTLS certificates may use a private CA, if they prove they are workload-internal gate / gate (pull_request) Successful in 1m58s Details images / images (pull_request) Failing after 7s Details gate / conformance (pull_request) Has been skipped Details R-TLS-01/docs/08 govern externally reachable TLS and ADR-0006 puts workload mTLS out of scope, but tls-lifecycle-automated required ACME for every Certificate, which platform/buildkit's internal CA cannot satisfy. Clause (5b) accepts the tls-scope=workload-mtls annotation only when the certificate proves it: no Gateway serves it, internal names only, namespaced selfSigned/ca issuer backed by an exempt CA certificate. Also fixes yamllint comma spacing in the buildkitd mounts. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-10-29 23:59:55 -05:00
jakman85andClaude Opus 5.5	f65c12b395	feat(images): build on rootless BuildKit over mTLS; verification moves into Containerfile verify stages Operator decision on PR #81: no CI job may reach a privileged daemon. The act runner is untouched. - platform/buildkit (+ clusters/base/buildkit.yaml): moby/buildkit v0.34.1-rootless pinned by digest, uid 1000, not privileged; caps drop ALL + SETUID/SETGID, seccomp/AppArmor Unconfined, read-only rootfs (each measured under podman). cert-manager CA + server/client certs, ClusterIP :1234, default-deny with ingress only from the act-runner pod (DinD masquerades jobs to it). No Kyverno exception: every applied policy matches app-namespace=true only. - Each Containerfile names its shipped stage and ends in a `verify` stage holding every former `docker run` assertion (uid 65532, RUN --network=none, writable-path inventory standing in for the read-only rootfs). The old `&& echo` checks that could never fail are real assertions now. - images.sh drives buildkitd with a pinned buildx (sha256-checked) via bake; PRs build verify targets with agent-runner FROM the in-run toolbox (target: context); main verifies, then pushes. Fails loudly when the builder secrets are absent or buildkitd is unreachable. - infra/gitea/buildkit-client-secret.sh copies the client cert into Actions secrets. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-10-29 23:59:55 -05:00
jakman85andClaude Opus 5.5	e539109830	feat(images): build, verify and publish the three images in CI; Renovate carries the digests Operator rule 2026-10-29: image creation, hosting and lifecycle belong to CI/CD, never to a workstation build/push. Nothing had ever been pushed to the Gitea registry. - .gitea/workflows/images.yaml + scripts/images.sh: PRs build+verify (no credential); main builds, verifies and pushes toolbox, agent-runner (FROM toolbox) and conformance-runner as sha-<commit> + main, reading the digest back from the registry. Only changed images build. - Each image's own build-image.sh does the build and in-image verification, now engine-neutral (DinD: no podman-only calls, probe over stdin not a bind mount). New toolbox build-image.sh. - Digest references carry `# renovate-digest:` markers; renovate.json5 gains a custom.regex manager for them. The worker Job template is deliberately untracked (it would loop). - renovate.yaml pins renovate-version 44.167.0: the action's default (39.264.0) rejects the current config's managerFilePatterns, so every run would have died at CONFIG_VALIDATION. - Push credential: REGISTRY_PUSH_TOKEN secret; the Actions token has no package scope in Gitea 1.24.3 (container.go Authenticate). Retention: infra/gitea/registry-retention.sh. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-10-29 23:59:55 -05:00
jakman85andClaude Opus 5.5	17eed6e8ae	fix(keycloak-passkey): bind only on git's say-so; protect real sub-flows; screen the requirement PUT by body id Review findings on the passkey reconciler, fixed: 1. HIGH - the bind was automatic and rollback did not stick. A git-declared switch, `bind-browser-flow` in passkey-desired-state.yaml, now decides whether pass 3 may PUT browserFlow. Default "false": build + verify the candidate (or reuse a verified one), log WOULD BIND <alias>, never touch the binding - including after an earlier bind, so an operator's rollback sticks, and a hand-edited bound candidate is left alone rather than shadowed by an -r2 build. "true" binds as before. The renderer carries the one hand-edited line through and --check refuses anything but the quoted strings "true"/"false". --no-bind can only narrow it. 2. HIGH - the protected-subtree guard read `alias` on sub-flow rows. Keycloak 26.0.6 AuthenticationManagementResource.recurseExecutions sets displayName = flowRef.getAlias() and flowId on a sub-flow row and never sets alias (alias is the CONFIG alias on authenticator rows), so the protected set was only the top-level alias. Sub-flows are now named by displayName (getFlows() lists top-level flows only, so flowId cannot be resolved through it); a sub-flow row without one leaves the subtree undetermined and the flow pass fail-closed. The selftest stub now returns 26.0.6-shaped rows, and a real run is asserted to refuse writes into browser-mfa-forms and browser-mfa-conditional-otp. 3. MEDIUM - PUT /flows/{alias}/executions was screened on the path alias only, but updateExecutions resolves the execution realm-wide by body id. body.id is now screened against the protected ids AND required to be in the tree this run is building (recorded from the build's own readback); the config POST is held to the same build scope; and the body may carry only id/requirement/priority (the same handler renames any flow named by flowId). 4. Stale "audit-only" docs rewritten (platform-config kustomization, chainsaw test, desired-state header and its renderer template, README) to describe what ships. 5. backoffLimit 0 (a retried deterministic build failure burns the candidate-alias budget); the stale "Renovate does not manage image: tags" comment corrected and the image pinned by index digest. 6. The ~160-line header duplicated in the YAML and the script is now kept once, in the script, with a short pointer above it. Selftest: 259 assertions (was 216). mutations.py: 40 injected regressions (was 30), including one per finding above, all caught. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-10-29 21:41:14 -05:00
jakman85andClaude Opus 5	f5931944fa	feat(keycloak): reconcile the passkey realm config via GitOps, the browser flow blue/green PR #72 declared the WebAuthn passwordless policy and a restructured browser-mfa flow in the realm import. That declaration could not take effect: a KeycloakRealmImport is a ONE-SHOT, and prod's reads Done=True / Started=False, so every realm-level field and flow declared after the first import is inert -- in git, applied by Flux, reported Ready, reaching the realm never. Merging it changed nothing about how anyone logs in, and nothing anywhere said so. Same failure class as the 2026-09-06 client-secret incident, third field over. A SIBLING reconciler converges all three declared things every 15 minutes. It is a sibling, not a second pass of client-secret-reconciler.yaml, so that reconciler's allow-list -- and its guarantee that prod's real user and the media-users / platform-admins / viewers groups are structurally unreachable -- stays literally true. Its selftest still passes unchanged. pass 1 webAuthnPolicyPasswordless* -- strict read-modify-write of GET/PUT /admin/realms/homelab, readback-verified. pass 2 webauthn-register-passwordless -- strict read-modify-write of ONE literally-named alias; enabled, defaultAction false so enrolment stays opt-in. pass 3 the browser flow -- BLUE/GREEN. PASS 3 NEVER MUTATES THE FLOW USERS ARE LOGGING IN THROUGH. Converging a five-flow, ten-execution tree needs create-subflow, create-execution, set-requirement and bind-config as SEPARATE untransacted calls; doing that to the BOUND flow means a run dying between them leaves a half-built login path for every SSO app and kubectl, and this repo has had a total SSO outage from a realm change. So instead: 1. BUILD under a NEW, unused alias (declared alias + a hash of the desired shape, applied to EVERY alias in the tree -- reusing browser-mfa-forms would mean writing INTO the bound tree). A failure leaves an orphaned, unbound flow and zero user-visible effect. 2. VERIFY by readback -- levels, ordering, authenticators, requirements, and the conditional-OTP config CONTENTS -- before anything is bound, so the regression that once 403'd every platform-admin is caught BEFORE it bites. 3. BIND with ONE atomic PUT setting browserFlow. 4. ROLLBACK is one write; no DELETE exists anywhere in the allow-list, so the old flow survives as the rollback target. The widened allow-list is made safe structurally. It cannot say "but not the bound flow" -- its write entries are alias-parameterised -- so a second guard screens every write against the live browserFlow's WHOLE subtree (sub-flow aliases and execution ids) before a socket opens, FAIL-CLOSED until that binding has been read. The create-flow endpoint carries no alias in its path, so it is screened on its request BODY; the selftest caught that it otherwise escaped the guard. Idempotence is load-bearing: a rebuild-and-reflip every 15 minutes would churn the login path on a schedule. The bound flow is compared STRUCTURALLY and a match produces zero writes and no new flow. Sub-flow and config ALIASES are deliberately not compared -- a built flow carries a hash suffix, so comparing labels would make it permanently unrecognisable as converged. Single source of truth: the desired state is GENERATED from the realm import by ci/render-keycloak-passkey-policy.py and gated with --check, so the RP ID -- baked into every credential at enrolment and never re-keyable -- is declared once. The pod keeps automountServiceAccountToken: false and reads a mounted ConfigMap, so no rbac.yaml is added anywhere. Read from Keycloak 26.0.6 source before being relied on: updateRealm (DefaultExportImportManager 725-894) makes no getUsers/getClients/getGroups/ getRoles/getRequiredActions/getAuthenticationFlows call; it DOES read getBrowserFlow(), which is what makes the atomic bind possible and why the policy write pins it; the passwordless policy is applied all-or-nothing only when RpEntityName is non-empty, with defaults substituted for omitted keys; and updateExecutions compares rep.getPriority(), so a body omitting `priority` would silently reshuffle the whole execution order. Gate coverage: a new whole-repo L0 check extracts the script and runs its --selftest (216 assertions, including a build killed at four different points each asserting the old flow is still bound and the bound subtree was never written), plus mutations.py (30 injected regressions, all caught) proving the selftest can fail. An earlier iteration of this work only AUDITED the flow and left the restructure manual; decisions/2026-10-18-keycloak-passkey-reconciler.md records that design and why it was reconsidered. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-29 21:22:08 -05:00
jakman85andClaude Opus 5	793c3940a8	fix(conformance): one regex made the merge gate take 28 minutes instead of 2 gate / gate (pull_request) Successful in 1m55s Details gate / conformance (pull_request) Has been skipped Details ha-control-plane-as-code/check.sh had: re.search(r"--server https://.?\n(?:.\n){0,10}?\swhen:...'secondary'", K3S_TEXT, re.S) With re.S every `.` matches newlines, so `.?` and the nested `(?:.\n){0,10}?` can each consume the whole document -- nested unbounded repetition, which backtracks exponentially on a failed or late match. MEASURED in CI via `docker top` on the live job container: python3 at 19:54 of CPU time, which is essentially the entire ~28 minute gate. Localised to the exact line by extracting the inline python and running it under faulthandler.dump_traceback_later, which named re/__init__.py:177 in search from this call -- not inferred from reading. Replaced `.` with `[^\n]` and dropped re.S, so the {0,10} is a real ten-line window with nothing to backtrack into. On the real input (22,179 chars): 0.2 ms, match still found. Whole check: 0.199s, rc 0, 115 assertions, identical success message. WHY IT SURFACED NOW, AND IT WAS MINE: K3S_TEXT is every .yml in roles/k3s_server concatenated. The regex was latent for as long as it existed; PR #71 added ~100 lines to that role (kubelet graceful-shutdown config), growing the input past the point where exponential backtracking stopped being survivable. WHY ONLY infra/* WAS SLOW: touched-requirements.sh maps infra/* to R-ENV-01, so the L0 run becomes --include-scope whole-repo --requirements R-ENV-01, which selects this whole-repo check. Doc/platform changes never did -- #69 1m55s and #72 1m53s stayed fast while #71 27m44s, the #73 merge 27m46s and #75 all ran ~28m. The pattern is unique in the repo (grep over all l0-static check.sh files). Not re-timed in CI post-fix; this PR's own run is the first datapoint. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-18 02:38:23 -05:00
jakman85andClaude Opus 5	a2440a3125	feat(upgrades): automated OS reboots (kured) + security-only unattended-upgrades, and give Renovate a runner gate / gate (pull_request) Successful in 1m57s Details gate / conformance (pull_request) Has been skipped Details Two halves of one repeatable upgrade process. Neither works alone. (a) OS upgrades and reboots platform/kured/ — Kubernetes Reboot Daemon, chart 6.1.0/app 1.23.0, in its own Flux Kustomization (dependsOn platform, wait:true). NOT in platform/kustomization.yaml: that tier gates platform-config and every apps wave, so a wedged reboot daemon there could stall every application. A component added to make maintenance safer must not be able to cause an outage. infra/ansible/roles/unattended_upgrades/ — new role (node_baseline is touched by other open branches, and this is a different lifecycle), enabled for prod in group_vars. The human gate is the SENTINEL, which is how this honours node_baseline's "IT DOES NOT REBOOT" (k3s-prod-2 hung on a warm reboot 2026-09-03) and cluster-upgrade-plan.md's rejection of automating away the gate between nodes: u-u is security-only with the kernel blacklisted, so it essentially never writes /var/run/reboot-required. That file appears when an operator deliberately runs node-baseline.yml --tags kernel. kured then does only the mechanical half, including the ordering rule node_baseline has to shout. Quorum: concurrent reboots are structurally impossible, not merely unlikely. concurrency:1 plus kured's single lock annotation (API-server optimistic concurrency), and lockTtl:0 — an expiring lock is exactly the 3-node etcd catastrophe, so a hung node holds it forever and nothing cascades. Longhorn: measured 25 volumes x 3 replicas, hard anti-affinity, 600s replenishment wait, serialized rebuild — so every drain degrades ALL volumes to 2/3 and the inter-node gate must be volume robustness, not pod readiness. Doubled: lockReleaseDelay 30m (unconditional) plus kured's Prometheus alert gate (semantic). alertFilterMatchOnly:true is load-bearing — the flag inverts the regexp's meaning, and under the chart default kured would have been blocked on day one by the five alerts measured firing with no end in sight, and never rebooted anything. Kernel floor (kernel >= 6.13, EFI memattr): excluded TWICE. The running kernel comes from o=Debian Backports, which is not an allowed origin, so u-u cannot see a kernel upgrade; and ^linux- is blacklisted anyway so widening origins later cannot re-arm it. Asserted against apt-config dump, not against the file just written. 6.12 kernels are still installed and bootable on all three nodes, and nothing was held. Closes a silent no-op: update-notifier-common was absent on all three nodes, so NOTHING could create the sentinel. kured alone would have idled forever at 1/1 Running. platform/kured/alerts.yaml — three rules over kured's only metric, with a companion promtool suite pinning each both ways, plus runbooks/node-reboot-automation.md. Window is DAILY 03:00-06:00 because a weekly one forces for:8d, longer than this cluster's 7d Prometheus retention. (b) Image/dependency automation Renovate was aspirational: renovate.json present and complete, with no workflow, no in-cluster workload, and its docker+kubernetes managers disabled — so every plain image: tag was maintained by nothing while seven files claimed otherwise. .gitea/workflows/renovate.yaml — scheduled daily, dry_run default true for a safe first run, and a preflight that FAILS (not skips) without RENOVATE_TOKEN, because a skipped Gitea job reports success and that is how this stayed inert. Renovate now owns both halves; Flux image-automation is formally closed: it pushes commits rather than opening PRs (bypassing CLAUDE.md §1), and it cannot work here anyway — main is protected user_can_push:false and the flux-system GitRepository has no secretRef. Its controllers are installed and have idled 43 days with zero CRs. conformance/: +34/-0, one new SCRAPE_OWNERS entry claiming kured_. Weakens nothing; tightens the check for this component (requires service.create+metrics.create, both chart-default false, plus a companion NetworkPolicy requirement). Required because assertion (3) fails an alert over an unclaimed metric and KNOWN_UNWIRED_METRICS is shrink-only. Sensitive paths touched: clusters/, conformance/, .gitea/** — needs a non-author approving review; /approve-override deliberately NOT posted. NOT verified: no OS upgrade run, no node rebooted, no lock observed, the Ansible role never executed, and whether CNPG switches its primary on a drain-induced eviction (the most likely way the first real reboot fails). Full list in the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-18 00:10:29 -05:00
jakman85andClaude Opus 5	b156ee6cfa	fix(obs): open 44180 to ns monitoring so the forward-auth gate sidecars are actually scrapable gate / gate (pull_request) Successful in 1m51s Details gate / conformance (pull_request) Has been skipped Details platform/oauth2-proxy/podmonitor.yaml landed 2026-10-02 and discovered six targets that have never once scraped. Measured on prod 2026-10-04: all six read health=down with lastError "context deadline exceeded" and lastScrapeDuration exactly 10.0s — the configured scrapeTimeout, i.e. packets DROPPED rather than refused. Cause: every app namespace ships a default-deny NetworkPolicy with policyTypes: [Ingress, Egress] and NO ingress key at all (deny-all-ingress), and the only ingress allowance is <app>-allow-gateway-ingress: fromEntities: [ingress] -> 4180. Prometheus matched no clause. Verified identical in all six of jackett, radarr, sonarr, qbittorrent, pihole, crafty; zero namespaces had any -allow-monitoring-scrape object. render.py already had the right idiom — r_monitoring_ingress_policy — but it returned None unless the service declared observability.metricsEndpoint, and NO ServiceDefinition in this catalog declares one. It was wired in at the call site and had never emitted a single object on any branch: correct, and entirely latent. It now also fires for the forward-auth gate sidecar's own metrics port, keyed on _needs_oauth2_sidecar — deliberately the same condition the scrape keys on, since podmonitor.yaml selects pods purely by the auth-mode label, so a target exists iff the sidecar exists. Keying on observability.metrics instead would let metrics: false produce a discovered-but-blocked target: the same "two halves authored apart" failure this function exists to prevent. Rendered here rather than in platform/oauth2-proxy because that tier would have to name app namespaces created by the sibling apps-wave-0/1 tiers — it would apply on a running cluster and fail on a cold bootstrap. podmonitor.yaml's header records that objection; rendering per-app dissolves it. Five CNPs result, one per forward-auth app, each opening only 44180 to only ns monitoring. apps/ re-rendered with the invocation it was generated by (--no-image-automation, verified byte-identical beforehand); --emit-waves deliberately not run, so no wave kustomization moved. Three L1 goldens are regenerated with it (+63 lines, -0): a renderer change that adds an object necessarily drifts the goldens that pin the renderer's output. The first gate run caught container-workload-render and egress-class-render as FAIL with exactly this CNP as the drift. default-deny-baseline-render's golden regenerated byte-identical, which is the evidence that the regen is faithful and not a rubber stamp. That puts conformance/** in the diff, which is a sensitive path: this PR needs a non-author approving review. The local gate cannot see that — offline it reports approval-policy green (N/A). Local toolbox gate: rc 3 UNDETERMINED, not 0. Every stage that ran is green and l1 went FAIL -> pass with the goldens; the only not-run stage is gitea-settings (no GITEA_SETTINGS_TOKEN). L2/L3 conformance is UNVERIFIED for all 8 touched requirements. Not verified: that the scrape now succeeds. Nothing is applied until Flux reconciles main. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-04 16:05:44 -05:00
jakman85	4f16e83eec	Merge pull request 'feat(obs): make oauth2-proxy auth failures visible — the gates emit nothing' (#29 ) from feat/oauth2-proxy-auth-failure-visible into main gate / gate (push) Successful in 1m43s Details gate / conformance (push) Has been skipped Details Reviewed-on: #29 Reviewed-by: Keith Jackson <[email protected]>	2026-10-02 04:55:43 +00:00
jakman85andClaude Opus 5	99560c2d22	feat(auth): declare API paths per service, so --api-route can ship at all gate / gate (pull_request) Failing after 1m44s Details gate / conformance (pull_request) Has been skipped Details PR #28 fixed the cookie collision, the 5-minute session and the missing metrics listener in one renderer function, and deliberately declined `--api-route` because the config model could not express the difference between two apps. This adds the hook it was missing and sets the flag only where a measurement says to. THE DEFECT. oauth2-proxy answers an unauthenticated request with a 302 to Keycloak. Its AJAX guard would send 401 instead, but it fires only on an EXACT `Accept: application/json`, which these apps' JavaScript does not consistently send. So a browser XHR gets a 302, FOLLOWS IT INVISIBLY, has it satisfied silently by the live SSO session, and resolves 200 carrying the app's HTML where the JavaScript expected JSON. On a write it is worse: a browser following a 302 re-issues a POST as a GET and DROPS THE BODY, so the operator's change is discarded and the UI reports success. MEASURED, on two apps and two different paths -- which is the whole argument: jackett 2026-09-30: 33 x 302 vs 2 x 401 on /api/ paths, and 4 of 6 POSTs answered by the proxy and never reaching Jackett (upstream `-`). radarr 2026-10-01, 23h after PR #28 deployed (pod radarr-7d55cbffb9-42hmh, 0 restarts): 232 unauthenticated requests in the pod's whole life and ALL 232 are the same call -- POST /signalr/messages/negotiate from a browser, 302, upstream `-`. A left-open tab whose session the cookie rename orphaned, polling forever, silently redirected every time and NEVER re-prompting. The reported symptom, still in production. So radarr's broken path is /signalr/ and jackett's is /api/, and PIHOLE SERVES NO /api/ AT ALL. One function renders all five sidecars, so the cheap fix is a constant -- and a constant applies one app's measurement to four apps nobody measured. `auth` was `additionalProperties: false`, so there was no per-app hook. Adding one named, documented key is the change; the block stays closed. schemas/servicedefinition.schema.json 1.8.0 -> 1.9.0: optional `auth.apiRoutes`, a list (radarr needs two, and --api-route is natively repeatable) of anchored path regexes. Plus the shape invariant that a non-empty list requires mode: forward-auth, since no sidecar is rendered for oidc/native/none and declaring it there would be a SILENT NO-OP. catalog/render.py emits one --api-route per entry, in order, and RAISES on an unanchored entry. The flag is SINGULAR -- verified against the pinned v7.8.1 binary's own --help; --api-routes does not exist and the proxy refuses to start with it. catalog/{jackett,radarr}.yaml the two measured apps. catalog/{pihole,sonarr,qbittorrent}.yaml comments recording that they get NOTHING, and why. Sonarr and qBittorrent were measured (0 unauthenticated requests in 23h) and left alone: that is evidence of no TRAFFIC, not of no need, and sonarr being nearly radarr's codebase is what makes the guess tempting and still a guess. apps/wave-0/jackett.yaml, apps/wave-1/radarr.yaml regenerated. Only TWO goldens change; pihole/qbittorrent/sonarr are byte-identical to main, which is the per-app property proving itself on the real tree. BREADTH IS THE DANGEROUS DIRECTION AND IT IS NOT SYMMETRIC. A missing --api-route is a silent 302. A too-broad one answers a plain PAGE LOAD with 401, so a logged-out operator gets a bare error page and is never offered a sign-in -- the service looks permanently broken. Each entry must therefore match `^\^/[A-Za-z0-9._~/-]+(\.\\|\$)?$`: anchored at ^/ then literal path characters. `^/`, `^/.`, `.*`, `/api/` and `^/api/\|^/` are all rejected, all five provoked. Alternation is refused rather than supported because the flag is repeatable, so one entry per route costs nothing. NEW GATE CHECK: conformance/tests/l0-static/oauth2-proxy-api-routes-per-app (whole-repo scope, so ci/l0.sh's --include-scope union runs it on every invocation; no bespoke stage, which whole-repo-invariants-wired would fail). It is a SIBLING of oauth2-proxy-cookie-names-distinct, not an extension of it: that check's subject is cookie IDENTITY and shares neither inputs nor failure mode with this, so bundling them would make its name a lie and let a future api-route decision be expressed only by weakening a COOKIE check. The check exists for one reason the other gates cannot cover: render-golden compares the render to the goldens, so a RENDERER CONSTANT plus regenerated goldens AGREE WITH EACH OTHER while both being wrong. Provoked: the regen exits 0 -- render-golden is happy -- and this check exits 1 with 15 failures naming pihole, qbittorrent, radarr and sonarr individually. It also asserts the anchoring pattern is the SAME in the schema and the renderer, because three surfaces guard this and if they drift the loosest wins silently. Tested in both directions on throwaway trees outside the repo: compliant tree rc 0 / 43 assertions / 6 sidecars; pre-change tree rc 1 / 11 failures; and four targeted mutations (hand-edited golden, widened regex, renderer constant, deleted schema pattern) each firing their own assertion class and nothing else. ALSO FOLDED IN (coordinator's request; defect is in the file I own). _selftest() reached _cookie_name() through a bare `next(genexpr)` with NO default, so deleting the --cookie-name emission -- the exact mutation that assertion exists to catch -- raised StopIteration, which escaped _selftest() BEFORE its collected `SELFTEST FAIL:` lines were printed. Right verdict, zero diagnostics, and it masked every other assertion the same mutation broke (found by Desmond Oyelaran while wiring the selftest into the render-golden stage; his PR #33 is not in this base -- origin/main @ `cdf3e14` drives the selftest from no stage, checked). Fixed with `next(..., None)` plus an explicit check(), so a missing flag reports as a NAMED assertion. Provoking it exposed three more instances of the same class in the same block -- `side = fa_containers[1]`, `side["env"][0]`, and the fa_svc lookup all raise rather than assert when the sidecar is absent -- so all four are now None-tolerant, and the two fixtures added for --api-route were written that way from the start. Measured on throwaway copies: --cookie-name deleted: before rc 1 / 0 FAIL lines / StopIteration after rc 1 / 3 FAIL lines / no traceback sidecar not appended: before rc 1 / 0 FAIL lines / IndexError after rc 1 / 20 FAIL lines / no traceback Goldens are unchanged by this part; it touches _selftest() only. NOT verified, in full in the handoff entry: no browser round-trip by anyone for this flag -- that a 401 REPAIRS the operator's experience is reasoned from HTTP semantics, not watched. `^/api/` on radarr is PR #30's measurement of radarr's pre-fix pod, not mine; the current pod has zero /api/ traffic. Sonarr and qBittorrent are UNMEASURED, not cleared. --cookie-refresh is still unexercised (0 AuthSuccess on all five pods). Nothing was applied; the live pods are still the pre-change spec. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-01 22:07:08 -05:00
jakman85andClaude Opus 5	bce101d03f	feat(obs): wire the gate-sidecar scrape, and fix the alert that would have gone silent doing it gate / gate (pull_request) Failing after 1m42s Details gate / conformance (pull_request) Has been skipped Details PR #28 gave the five fleet oauth2-proxy sidecars --metrics-address=0.0.0.0:44180 and a named `metrics` port. This wires a scraper to them and, in the process, found a real bug in the alert this branch added. PodMonitor, not ServiceMonitor. Measured: no Service exposes 44180 on any of the five (jackett/radarr/sonarr/qbittorrent/pihole all publish only their app port + authproxy 4180), so a ServiceMonitor would select nothing. A PodMonitor targets pod ports directly and needs no change to catalog/render.py or the apps/wave-* renders and goldens. Selector is the pod label homelab.keithjackson.dev/auth-mode=forward-auth — present on exactly those five gate pods, absent from crafty's, which still has no metrics listener. The property, not an app list. THE BUG: `up{container="oauth2-proxy"}` -> `up{container="oauth2-proxy"} == 1`. A target that is discovered but FAILING still publishes `up`, with value 0. The series exists, so count by (namespace) returned 1, `unless` removed the namespace, and OAuth2ProxyGateUnobserved went QUIET while nothing was being scraped at all — this branch's own disease reproduced inside the alert written to cure it. Not hypothetical: the PodMonitor lands five targets whose scrape is dropped, so the bare form would have silently cleared on this very commit. Proven by mutation: reverting the `== 1` fails the two new unit-test scenarios (got: nil) and leaves every other assertion green. The scrape does NOT work yet, and that is documented rather than papered over. From a pod in ns monitoring to the jackett pod, :44180 AND :4180 both time out; the shared proxy's :44180 answers. ns jackett's default-deny is podSelector:{} with policyTypes:[Ingress,Egress] and NO ingress key (deny-all), and the one allowance is fromEntities:[ingress] to 4180 — the Gateway's Envoy only. The endpoint itself is healthy: port-forwarded, /metrics returns http=200, 10208 bytes, all three oauth2_proxy_* families including requests_total{code="302"}. Same bug and fingerprint as platform/longhorn/networkpolicy-metrics.yaml records for Longhorn; the fix is one narrow rule per app, and it cannot ship from this tier because oauth2-proxy and apps-wave-0 are Flux SIBLINGS (both dependsOn platform-config), so naming ns jackett here would apply today and fail on a cold rebuild. Needs an owner in the renderer. Touches conformance/** : one KNOWN_EXTERNAL_MONITORS entry, which is the mechanism the failing check explicitly instructs ("DECLARE it ... with who creates it") and which flux-podmonitor.yaml already uses. It records that the port name was measured on prod instead of resolved from a Service, and that the targets are currently down. That makes this PR require a non-author review on the sensitive-path rule. Also: ratio alert still NOT shipped. oauth2_proxy_requests_total now exists on the endpoint but not in the TSDB, so writing it today would make it inert for exactly the reason it was refused on 2026-09-30. Runbook gains policy-vs-no-target triage and the radarr /signalr/messages/negotiate worked case — one orphaned browser tab, 206 identical 302s, nobody ever re-prompted. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-01 21:56:07 -05:00
jakman85andClaude Opus 5	dc136a4943	fix(auth): the forward-auth sidecar also died at 5 minutes and served no metrics gate / gate (pull_request) Successful in 1m42s Details gate / conformance (pull_request) Has been skipped Details Widens the cookie-collision fix to the whole sidecar contract, because all three defects live in the same function and land on the same five goldens, so they are one atomic change or they are five conflicting ones. Added to _oauth2_sidecar_container(), on top of the cookie-name/cookie-domain fix: --cookie-refresh=2m30s The session died FIVE MINUTES after sign-in. Keycloak's homelab realm issues 300-second access tokens (MEASURED on a live session: expires - created = 300.000s exactly, with refresh_token:true on the same line), and --cookie-refresh defaults to 0 -- which the pinned binary's own help spells "0 to disable". So the proxy held a refresh token it was never told to spend. Refresh at HALF the lifetime: two chances to renew before expiry. This, not the collision, is what dominates the rejection VOLUME -- ~95% of radarr's 155 rejections in 12h carry no cookie at all. --metrics-address=0.0.0.0:44180 + a `metrics` container port Without the flag the /metrics listener never starts, so no per-app sidecar was serving metrics at all (`ports` was ['http'] on all six). 44180 is the platform's existing convention, measured not chosen: the shared proxy already runs the same flag and port with 2 ready endpoints. PREREQUISITE ONLY -- NOT scraped. A container port is not a Service port, and each namespace's default-deny baseline is `ingress: []`, so this is unreachable until a Service port (or PodMonitor), a monitoring allow and a ServiceMonitor land. Do not read this as "the sidecars have metrics now". EVERY FLAG VERIFIED AGAINST THE PINNED v7.8.1 BINARY (`--help`), not against docs or the shared proxy -- which runs v7.15.3 and could not have vouched for v7.8.1: * --metrics-address exists in 7.8.1, so this cannot crashloop the pods * --cookie-refresh ... "0 to disable" confirms the default * --cookie-name ... (default "_oauth2_proxy") -- this confirms from the IMPLEMENTATION the constant the new L0 check is built on * --api-route is SINGULAR; the plural does not exist and the proxy refuses to start * --session-cookie-minimal exists and is mutually exclusive with --cookie-refresh, so dropping --pass-access-token is a least-privilege argument, never a size fix NOT done, deliberately: --api-route=^/api/ renderer-wide. It was measured on jackett and PiHole does not serve /api/, so a renderer constant would apply one app's path shape to five. It needs a per-app hook the `auth` schema forbids (additionalProperties: false) -- a schema change, separate PR. Also records the mechanism that made this INTERMITTENT, as a renderer comment, because nobody would reconstruct it from the flags: the loser does not just fail to read the cookie, it calls ClearSession, which expires the cookie on its own --cookie-domain -- the whole zone. The *arr UIs poll (/api/v3/command, /api/v3/queue/details, /signalr/negotiate), so every poll carrying an undecryptable cookie clears it again: A LEFT-OPEN RADARR TAB DELETED JACKETT'S COOKIE EVERY FEW SECONDS, from another namespace, with no user action. That is why signing in again never helped for longer than the gap between two polls. Crafty's counter-evidence resolved: the 81:1 ratio was a 24h window straddling a 13h52m-old pod. Bounded to the fixed process, crafty's ENTIRE life is 3 rejections in 6743 log lines, with ZERO `removing session` and ZERO `cookie signature not valid` against unfixed radarr's 8 and 8. The eviction fingerprint is gone where the fix is deployed. It logged 0 AuthSuccess too, so that is absence-of-the-failure-mode rather than a session watched working -- stated as such in the handoff. Renderer selftest gains six assertions, including a --cookie-refresh check that PARSES the duration and requires 0 < refresh < 300s (so 2m30s/150s/2.5m all stay green while a silent widening past the token lifetime does not). NOTE, and it is in the handoff as the top follow-up: NOTHING IN ci/ RUNS `render.py --selftest`, so those assertions are dev-time only. conformance/tests/l0-static/auditability-proofs-producible/check.sh:291 already drives other tools' selftests for exactly this reason and is the one-line fix. Gate (toolbox, worktree shim with the spec repo mounted at /home-network-k3s): rc 3 UNDETERMINED. Green: spec-immutability, decisions-log, lint, l0, render-golden, l1, approval-policy. Skipped: gitea-settings (no local token), conformance (declared). The new L0 check PASSES on this branch -- all five apps fixed in one commit. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-30 13:07:43 -05:00
jakman85andClaude Opus 5	9a4489b733	fix(auth): five oauth2-proxy sidecars shared one cookie and evicted each other gate / gate (pull_request) Failing after 1m43s Details gate / conformance (pull_request) Has been skipped Details Every oauth2-proxy on this platform is an independent instance with its OWN cookie-secret. Measured on prod 2026-09-30: seven `-oauth2-proxy` Secrets, seven DISTINCT cookie-secret values (compared in-cluster, never printed). So no instance can DECRYPT another's cookie -- it can only OVERWRITE it. All of them were emitting oauth2-proxy's DEFAULT names (`_oauth2_proxy`, `_oauth2_proxy_csrf`) scoped to `.${BASE_DOMAIN}`, so the browser kept ONE cookie pair for SIX live instances (jackett, pihole, qbittorrent, radarr, sonarr, and the shared forward-auth proxy). Whichever authenticated last clobbered the rest; the loser logged `No valid authentication in request`, bounced to Keycloak, and -- the SSO session still being live -- was redirected straight back and clobbered it again. The symptom is "I signed into Keycloak but I'm still not signed in", which sends you hunting in Keycloak, where nothing is wrong. Crafty was fixed on its own side on 2026-09-29 (#22); this is that fix generalized to the pattern. THE DEFECT WAS NOT IN THE MANIFESTS. apps/wave-/{jackett,pihole,qbittorrent,radarr, sonarr}.yaml are GENERATED goldens; one function emits all five. So the fix is in catalog/render.py::_oauth2_sidecar_container(), and both flags are now DERIVED from the ServiceDefinition rather than defaulted: --cookie-name=_oauth2_proxy_{s['name']} unique by construction --cookie-domain={host} the app's own host, not the zone A per-service table was rejected for the reason twelve ServiceMonitors once acquired one identical wrong port name in a single edit: it would leave the NEXT app onboarded defaulting straight back into the collision. --whitelist-domain stays zone-wide; it governs post-login REDIRECT targets, not cookie scope. Host-scoping is defence in depth and is asserted deliberately: a zone-wide cookie is transmitted by the browser to every host in the zone, so each app's session token was being sent to every sibling -- and it is what would make a FUTURE name collision platform-wide instead of contained. New gate check (whole-repo scope, so it runs on EVERY gate invocation, not only when a touched path happens to attribute an identity requirement): conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct/ It resolves each instance's EFFECTIVE cookie name -- declared, or oauth2-proxy's built-in default when omitted, because an omitted flag is not an absence but a CLAIM on a name, and reading it as an absence is exactly why this survived five reviews. Then: no two collide; none is another's name + `_csrf` (oauth2-proxy derives the CSRF cookie that way); the default has exactly ONE owner, registered with its reason in a shrink-only register -- the SHARED proxy, which MUST keep `_oauth2_proxy` because renaming it signs every other gated surface out at once; every other instance is host-scoped; render.py derives rather than lists; and the SHIPPED renderer is proven end-to-end on two synthetic ServiceDefinitions, so the check stays falsifiable once the tree is clean. Tested in both directions: rc 0 on this tree (39 assertions, 7 instances), rc 1 with 26 failures against the pre-fix tree, plus one targeted mutation per assertion class (duplicate name, CSRF derivation, zone-wide domain, unregistered default heir, stale register entry) -- each fired, and nothing else did. Also regenerates the three L1 rendering goldens, which are expected-output fixtures of the same renderer. UNVERIFIED: no browser round-trip. The collision has still never been REPRODUCED in a browser -- doing so needs two gated apps visited in one session -- so this is reasoned from measured configuration and from the mechanism, not from a failure anyone watched. Nothing was applied; merging rolls five Deployments and invalidates every existing session on them once. See decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-30 10:55:06 -05:00
jakman85andClaude Opus 5	55b232e2ce	feat(obs): notify on Flux deploy start and completion gate / gate (pull_request) Successful in 1m40s Details gate / conformance (pull_request) Has been skipped Details The operator got no mail when Flux began applying a git change and none when it finished. Measured cause: notification-controller v1.4.0 is running and its CRDs serve v1beta3, but there are ZERO Provider and ZERO Alert objects in the cluster — nothing in git ever created any. The obvious fix does not work. Neither Flux event can be filtered into a deploy signal: ReconciliationSucceeded is a heartbeat (101 events in one hour across 16 Kustomizations, message byte-identical whether the apply changed anything), and Progressing is worse (five Kustomizations re-apply the same object on every tick — event counts 2505/2504/2501/2495/2490). gotk_* carries no revision label at all (gotk_reconcile_condition 120 series, gotk_suspend_status 40, none with one). The applied revision is the only signal that moves when and only when git moves, and it lives in object status alone. So export it via the kube-state-metrics custom-resource-state path that already serves the Velero Schedule metrics — kube_flux_kustomization_info/_suspended and kube_flux_gitrepository_info — and ask the question in PromQL: FluxRolloutInProgress (severity: info, no `for:`) fires the moment a non-suspended Kustomization's applied revision is not the revision the GitRepository fetched from main, and CLEARS when all have caught up. Fired mail = a deploy started; RESOLVED mail = it completed. The completion half is send_resolved, not a second rule. FluxRolloutStuck (severity: warning, for: 30m) is the same expression as the escalation for a rollout that started and never converged. Two guards, each with a test whose only job is to fail if the clause is removed: `and on () (count(gitrepository_info) > 0)` — without it a missing exporter makes `unless on (revision) <empty>` remove nothing and all 16 Kustomizations alert forever, the permanently-firing mode this repo shipped once already — and the suspended-Kustomization exclusion, without which a suspended tier holds the rollout open and the "applied" mail never arrives. A dedicated Alertmanager severity=info route + email-deploy receiver keeps deploy notices off the [WARNING] catch-all. podAnnotations carries a roll-counter because the chart sets no checksum over the custom-resource-state ConfigMap, so a config-only edit would leave KSM on the old collector set with the new alerts silently inert. Verified: promtool SUCCESS on 6 scenarios (both guards, plus the convergence test that proves the alert empties itself); lint/l0/render-golden/l1/decisions-log/ spec-immutability green in the pinned toolbox. NOT verified: no mail has been sent — the metrics do not exist until this merges. Full gap list, and the deliberately-cut notification-controller half, in the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-29 23:40:01 -05:00
jakman85andClaude Opus 5	8c7a770de2	fix(sec): the R-SEC-02 admission policy was never applied -- and nothing could tell gate / gate (pull_request) Successful in 1m37s Details gate / conformance (pull_request) Has been skipped Details PR #13 added conformance-trigger-job-shape.yaml to platform/kyverno/admission/ kustomization.yaml. That file applies NOTHING. Its own sibling says so: "kept as reference; THIS file is the buildable render root" -- and that render root (platform/kyverno/kustomization.yaml) ships only the namespace and the HelmRelease, noting that ClusterPolicies are applied a tier later, by platform-config, because they need the Kyverno CRDs first. So three lists exist and only platform-config/kustomization.yaml applies anything. The policy went into the wrong one and EVERYTHING WENT GREEN: the file parsed, kyverno test exercised it by path and passed, the merge gate passed, and Flux reported `platform` Ready at the new revision. Measured on prod afterwards: kubectl get clusterpolicy conformance-trigger-job-shape Error from server (NotFound) while ns conformance, the conformance-trigger ServiceAccount, its Role and its RoleBinding were all live. That is exactly the state that identity's own RBAC header says must never exist -- the trigger identity present with nothing bounding it. Blast radius was small only because no trigger kubeconfig exists yet, which is luck, not design. Wires the policy into platform-config (verified: the render goes from 7 ClusterPolicies to 8 and contains conformance-trigger-job-shape). CLOSES THE CLASS, which matters more than the instance. Adds the L0 check admission-policies-actually-applied: every ClusterPolicy file under platform/kyverno/ is either referenced by platform-config/kustomization.yaml or named in a CI_ONLY list with a reason -- and CI_ONLY may not name a policy that IS applied, so the exemption cannot rot in either direction. Declared scope whole-repo deliberately: a policy wired nowhere touches no requirement the diff attributes, so a path-scoped run would have skipped it on the exact commit that introduced the defect. Proven both ways, not assumed: with the fix reverted -> FAIL, naming conformance-trigger-job-shape as never applied with the fix -> PASS, 9 policy files, 8 applied, 1 CI-only with a reason `kyverno test` proves a policy is CORRECT. This proves it is DEPLOYED. Nothing did before. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-29 13:23:09 -05:00
jakman85andClaude Opus 5	8eb5baffa4	feat(sec): constrain the conformance CI trigger at admission (R-SEC-02) gate / gate (pull_request) Successful in 1m38s Details gate / conformance (pull_request) Has been skipped Details Implements the credential model the operator chose for wiring L2/L3 into CI: a narrow trigger SA bounded by a fail-closed Kyverno admission policy, rather than the zero-credential GitOps-native alternative (plan section 4, primary model). THE ESCALATION THIS CLOSES. Kubernetes places NO `use` check on a Job's serviceAccountName. Any principal that can `create jobs` in ns conformance can create a Job running as conformance-runner -- nodes/patch, pods/exec, namespaces/patch -- with an arbitrary command. A CI token with plain `create jobs` IS that privileged identity no matter how narrow its own Role looks. RBAC cannot fix this; only constraining the SHAPE of what the trigger may create can. Adds: * SA/Role/RoleBinding conformance-trigger -- create+get+list+watch jobs, get pods and pods/log, nothing else. Namespaced. No delete (a trigger that can delete can hide a failed run), no exec, no secrets. Results come back as a tarball on stdout read via pods/log, so no execution channel into the runner pod is ever granted. * ClusterPolicy conformance-trigger-job-shape -- Enforce, failurePolicy: Fail. It does NOT use ${KYVERNO_VALIDATION_ACTION}: a security boundary an overlay can downgrade to Audit is not a boundary. It constrains ONLY conformance-trigger, so a mistake here cannot wedge the rest of the cluster. * Flux Kustomization `conformance` -> ./conformance/incluster, wait:false and non-gating like apps-vms, so a test identity can never stall the platform or app tiers. * A kyverno test with seven cases, six of them real escalation paths. VERIFIED by `kyverno apply` (pass: 10, fail: 6, error: 0), not assumed: evil-command DENIED at /spec/template/spec/containers/0/command/ <- THE attack evil-image DENIED at .../image/ tag-not-digest DENIED at .../image/ (mutable tag is republishable) host-namespaces DENIED at /spec/template/spec/hostPID/ extra-volume DENIED by the volumes foreach injected-args DENIED by the args foreach good-run ADMITTED ALSO FIXES A FAIL-OPEN IN THE GATE ITSELF. `kyverno test` does NOT fail when a policy cannot LOAD: every row becomes "Skip / Invalid Policy" and it still prints "N tests passed" and exits 0. Measured while writing this policy -- a schema error (subjects in a separate match.all entry from resources) produced exactly that, and ci/l0.sh would have certified a security policy that does not load as green. This is not specific to this policy: a typo in restrict-host-namespaces or secret-provenance would silently disable it with the gate still green. ci/l0.sh now scans for "Invalid Policy" and fails. Proven: broken policy -> l0 rc=1, restored -> rc=0. IMAGE PINNING, HONEST LIMIT: the plan specifies pinning the exact runner image digest. That image does not exist yet, so this enforces the enforceable half -- the image must be under the conformance repository path AND digest-pinned, never a mutable tag. Residual surface: anyone who can PUSH to that path could publish a different digest. Tighten to the literal digest when the image is built; it is a one-line change to the pattern, reviewed in a PR, which is where a change to a security boundary belongs. NOT YET WIRED, deliberately: no runner image, no trigger kubeconfig, no CLUSTER_AVAILABLE. The identities here are inert until those exist, and PR #12 makes CLUSTER_AVAILABLE=true block rather than fake a green in the meantime. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-29 03:22:41 -05:00
jakman85andClaude Opus 5	712f4c9336	fix(conformance): "unit-tested" counted an alert MENTIONED in a comment as tested alert-rules-unit-tested decided whether an alert had a unit test with a bare word match over the whole test file. Any mention counted — including prose in a `#` comment. Hit for real while adding platform/monitoring/tests/vpn-egress.test.yaml. Its header explains that VPNTunnelDown stays quiet during an egress-steering failure, and that sentence alone made this check declare VPNTunnelDown "asserted on" and demand its removal from UNTESTED_ALERTS. Removing the entry would have recorded coverage that did not exist — the precise "looks fixed" outcome this check exists to prevent, produced by the check itself. (In that instance I made the coverage real instead, which also exposed that the fixtures were missing tun0's receive counter entirely.) Now an alert counts only when a test file names it in an assertion position, with comment lines stripped first so a commented-out assertion does not count either. BOTH promtool idioms are accepted, because both are real assertions: alert_rule_test: - alertname: CNPGPrimariesCoLocated promql_expr_test: - expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}' Matching only the first is not a stricter check, it is a WRONG one: it reported CNPGPrimariesCoLocated as untested when platform/cnpg/alerts.promtool-test.yaml pins it across four scenarios entirely through the second form. Believing that would have produced a redundant duplicate test for an alert already covered better than most — so the first version of this very fix had to be corrected before it landed. MEASURED BEFORE LANDING, because the registers are shrink-only and a prose-only alert would have required an entry this check forbids adding: of 61 alerts, 32 are asserted in one of the two idioms, 29 are not mentioned at all (already declared), and ZERO were prose-only. So this tightening failed nothing on the day it landed. My first measurement said 31/30/0 and was wrong in a way worth recording: it scanned only tests/*.test.yaml and missed the <manifest>.promtool-test.yaml companion convention, which is exactly what hid the CNPGPrimariesCoLocated false negative. Verified in both directions: 134 assertions pass with the real coverage intact, and a temporary prose-only mention of NodeExporterDown (a registered untested alert) leaves it correctly listed as untested instead of triggering the stale-entry failure. Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips, 0 unwired. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-19 01:00:59 -05:00
jakman85andClaude Opus 5	a3f4ddb971	merge: feat/egress-gateway-staleness-alert — finished so it can actually pass the gate Brings the two alerts for the silent egress-gateway failure: VPNEgressSteeringStale (critical) and VPNEgressUplinkLabelMissing (warning), plus the kube-state-metrics allowlist entry that makes them non-inert. WHY THE BRANCH SAT IN THE BACKLOG: it could not pass the merge gate. Three whole-repo checks failed on it, and each one was a real gap rather than a nuisance: * alert-runbooks-resolvable — both alerts link to /runbooks/vpn-egress-stale, which did not exist. An alert firing now would hand the operator a 404. * alert-rules-unit-tested — alerts.yaml is already a covered file, so new rules in it must be tested or a "tested" file quietly grows untested rules. * alert-metrics-have-scrape-targets — see the guard bug below. WHAT THE ALERTS CATCH. egress-class=vpn pods leave through one gateway node's tun0; Cilium resolves it to an ifindex and caches it. An OpenVPN reconnect destroys and re-creates tun0 with a NEW ifindex, and the node label homelab.keithjackson.dev/egress-uplink=<addr>-<ifindex> is what forces Cilium to re-resolve. If the label does not follow the tunnel, the cache points at a destroyed interface and every vpn-class pod is silently dropped — while tun0 stays up and VPNTunnelDown says nothing. That ran ~80 minutes undetected on the dev gateway. Verified this is live on prod, not inert: k3s-prod-3 carries egress-gateway=true and egress-uplink=10.150.192.86-139, and vpn-egress-steering has been in place 14 days. The label was exported on 0 series before this change, so the allowlist entry is what switches the alerts on. runbooks/vpn-egress-stale.md — written and wired into the configMapGenerator, with the four required operator sections. It leads with proving egress from a pod the policy selects, because the symptom is "outbound traffic silently disappears", and it says plainly not to "fix" this by deleting the policy: that does not restore egress, it sends vpn-class traffic out of the real ISP connection, which is the leak the class exists to prevent. platform/monitoring/tests/vpn-egress.test.yaml — four promtool fixtures. 1 and 2 are the SAME tunnel reconnect with and without the label following it, which isolates the mechanism the rule actually keys on: a label VALUE change is not a changing sample, it retires one series and starts another. 3 is the inert case (gateway node, no uplink label) where the critical rule CANNOT fire, which is why the warning exists. Two fixes the branch needed that were defects in the checks themselves: 1. alert-metrics-have-scrape-targets asserted the KSM allowlist matched `nodes=[...egress-gateway]` with a literal closing bracket — i.e. that the list contained EXACTLY that one label. Adding a second label correctly (egress-uplink, which these alerts need) therefore FAILED the check while the property it guards was perfectly satisfied. Loosened to `[,\]]`: in the list, not the whole list. A guard that breaks when you correctly extend the thing it guards teaches people to edit the guard, which is how it stops guarding. 2. UNTESTED_ALERTS listed VPNTunnelDown, and the check flagged it as stale because "asserted on" is a bare word match over the test file — my header COMMENT mentioning VPNTunnelDown was enough to count as coverage. Rather than delete the register entry and claim coverage that did not exist, the fixtures now assert VPNTunnelDown QUIET in all four cases. That required making them faithful: they had no node_network_receive_bytes_total{device="tun0"} at all, so VPNTunnelDown would have been firing throughout unnoticed, because promtool only checks the alertnames you name. A healthy tun0 passes traffic, and that silence IS the claim these rules exist for. NOTE FOR LATER, not fixed here: that word-match means any alert named in prose counts as tested. Tightening it to `alertname:` positions could reveal other prose-only "coverage", and the register is shrink-only, so the cascade wants doing deliberately rather than at 01:00 unattended. Conflicts resolved: kept HEAD's Velero custom-resource-state block and took only the branch's one-line allowlist change (14 lines down to 1); inserted the branch's 2026-09-12 DECISIONS-LOG entry in date order rather than appending it out of sequence. Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips, 0 unwired. The new unit tests pass under the pinned promtool 3.14.0. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-19 00:52:06 -05:00
jakman85andClaude Opus 5	755c08ab40	fix(conformance): the meta-check was layer-agnostic and reported failures silently Declaring whole-repo scope on the L1 tests made whole-repo-invariants-wired (R-TEST-03) fail. It was right to change behaviour and wrong about why, and it could not tell anyone either way. Two independent defects. 1. LAYER-AGNOSTIC WHERE THE DESIGN IS LAYER-BOUNDED The "every scope-declared test is selected despite a requirement filter matching none" assertion compared EVERY scope-declaring dir against an `--layer l0` selection. That silently assumed only L0 tests would ever declare the scope — while ci/l1.sh was written from the opposite premise, that a whole-repo invariant could appear at any layer. So the first L1 declaration made the check blame the union for correctly excluding another layer. Now partitioned by layer, and the L1 half of the property is asserted too: every scope-declared L1 test must be selected by `--layer l1 --include-scope`, and that selection must not leak non-L1 tests. That is what makes the L1 declaration load-bearing rather than decorative — previously nothing proved the union reached L1 at all, because nothing at L1 declared the scope. 2. EVERY ASSERTION FAILED SILENTLY The idiom throughout was a bare test on one line and `chk $?` on the next, under `set -euo pipefail`. A false test aborts the script BEFORE chk can print, so the diagnostic — which is the entire value of this check — was never reached. The real failure above emitted 26 `ok` lines, rc=1, and no message at all; it read as a tooling glitch rather than a finding. All 12 assertion sites now capture the status (`chk $arc`) so the check reports and continues. Proven by forcing MIN_DECLARED=999 against the real check: the failure prints its full explanation and the run proceeds to the remaining assertions, where before it died mute. A meta-check that cannot say why it failed is worse than no meta-check. Verified: 35 assertions, rc=0; full offline l0,l1,meta 25 tests / 0 failures. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-18 22:43:17 -05:00
jakman85andClaude Opus 5	b389566641	fix(conformance): L1 rendering never ran on a requirement-scoped gate All three L1 tests now declare `conformance.home.k3s/scope: "whole-repo"`. ci/l1.sh has passed --include-scope whole-repo since it was written, with a comment saying no L1 test declared the scope yet and it was wired ahead of time so the next whole-repo invariant "at any layer" would be picked up by annotation alone. Nothing ever declared it. The consequence is that the entire L1 rendering layer only ran when a touched path happened to attribute R-PLAT-02, R-NET-01 or R-NET-05/06/07 — a fail-open hole for every other change in the repo. The coupling requirement-scoped selection cannot see is that all three tests exercise ONE code path: catalog/render.py against schemas/servicedefinition.schema.json. Anything that changes rendered output invalidates these goldens no matter which requirement the touched path maps to. The 2026-09-04 external-dns merge is the proof, measured rather than asserted. ci/touched-requirements.sh attributes platform/externaldns/helmrelease.yaml to R-ENV-02, R-OBS-01/02/04, R-STATE-03 and R-TEST-01/02/03/04 — none of the rendering requirements — yet that change added external-dns.alpha.kubernetes.io/controller: none to the rendered output of all three goldens. Via --print-selection: requirement-scoped only (the 2026-09-04 behaviour): 0 L1 tests selected with --include-scope whole-repo (how ci/l1.sh runs): 3 L1 tests selected So the merge gated GREEN while leaving every rendering golden stale, and the staleness surfaced only on a later full run. Also corrects the now-false comment in ci/l1.sh that said no L1 test declares the scope, and records why the annotation is load-bearing there. Verified: offline l0,l1,meta --include-scope whole-repo in the pinned toolbox — 25 tests, 0 failures, 0 errors, exit 0. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-18 22:43:03 -05:00
jakman85andClaude Opus 5	79c703bc79	fix(conformance): whole-repo checks audited nested agent worktrees as source Six discovery sites walked ROOT without pruning dot-directories, so they descended into .claude/worktrees/agent-*/ — up to 18 full copies of the repo, created for isolated agent runs and gitignored (.gitignore:43). The visible half was false FAILs: four ServiceMonitors reported as selecting no Service, and nine PrometheusRule manifests reported as "under test but not in the repo", all of them worktree paths. The whole-repo L0 gate was red for reasons that had nothing to do with authored surface. The dangerous half is quieter. A check that asks "does anything in the repo declare X" can be satisfied by a worktree copy instead of real source, so the same bug that invents failures can also manufacture a pass. That is the reason to fix all six sites rather than only the ones that happened to be failing. egress-paths-selected-by-rules already had this fix and documents it (it once saw 5 CiliumEgressGatewayPolicy objects instead of the 1 authored); this applies the same dot-directory predicate everywhere else, which skips dot-DIRECTORIES while still matching dot-FILES such as .sops.yaml. platform/monitoring/tests/run.sh is the other side of the same failure: its `--list` reported each manifest once per worktree, and the L0 check compares that listing against the tree. Both find calls now prune .claude alongside .git. `--list` returns 21 real pairs where it previously returned them ~19x over. Verified in the pinned toolbox: L0 --include-scope whole-repo is 21 cases, 0 failures, 0 skips, 0 unwired (was 4 failures, then 1). L1 3/3. The 18 stale worktrees are all on already-merged branches and remain on disk; cleaning them up is a separate change and is no longer load-bearing for the gate. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-18 22:14:51 -05:00
jakman85andClaude Opus 5	05881d5a6a	fix(conformance): regenerate the three L1 goldens the external-dns change drifted gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details `b184a42` made the renderer stamp external-dns.alpha.kubernetes.io/controller: none on internal services' HTTPRoutes, but three L1 golden fixtures were never regenerated, so container-workload-render (R-PLAT-02) and egress-class-render (R-NET-05/06/07) failed. Exactly one line differs in each — the annotation and nothing else. WORTH RECORDING: the merge that caused this GATED GREEN. The requirement-scoped selection did not include R-PLAT-02 or R-NET-05/06/07, so neither check ran. A later docs-only commit shifted the touched set, selected them, and exposed a failure the earlier merge had introduced — meaning the gate reported PASS on a tree it had not actually checked in the relevant place. That is the same fail-open the whole-repo scope mechanism was built for, except that mechanism covers L0 only; L1 is still purely requirement-scoped. A renderer change can therefore drift any L1 golden whose requirements the change does not happen to attribute. Both affected checks are pure offline renders with no cluster dependency, so they are cheap to run unconditionally — declaring scope: whole-repo on the l1-rendering tests looks like the right follow-up. Also note the checks diff fail-fast: only egress-vpn was reported, while egress-internet had drifted identically and was never reached. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-18 20:59:19 -05:00
jakman85	b184a42729	merge: internal services stop advertising themselves in public DNS Six exposure:internal services published A records into the public Cloudflare zone answering 192.168.2.20, disclosing the internal subnet, the Gateway VIP and a full service inventory, and forming a DNS-rebind primitive that is why the SOHO router strips these answers in the first place. The renderer now stamps external-dns.alpha.kubernetes.io/controller: none on every HTTPRoute it renders for an internal service, derived from the exposure field alone — so a service that omits exposure defaults to internal and is likewise unpublishable. Established against the deployed v0.21.0's own source, where the controller check runs before hostname resolution in both configured sources, then proven live on a throwaway hostname including that adding the annotation to an existing record triggers the DELETE. Deletes exactly 12 records: the A and TXT pair for jackett, pihole, qbittorrent, radarr, semaphore and sonarr. A 39-record recovery baseline is committed alongside. Safe to land before the DHCP change, contrary to my earlier caution: the router returns NOTHING for all six names today, so these records serve nobody on the LAN, while Pi-hole at 192.168.2.23 already answers for anyone pointed at it. The five exposure:public services are untouched. They remain unreachable publicly, which is a separate problem: the legacy edge at 136.51.115.252 returns HTTP 403 for our hostnames and serves a placeholder certificate, so nothing is reverse-proxying them today.	2026-09-18 20:50:43 -05:00
jakman85andClaude Opus 5	4664b41db3	fix(l4): externalTrafficPolicy Cluster on unifi + crafty L4 VIPs Cilium's L2 announcer elects one node per Service to answer ARP for its LoadBalancer VIP, and that election is NOT backend-aware. Combined with externalTrafficPolicy: Local -- which lets a node deliver only to a backend on itself -- an elected announcer holding no backend answers ARP for the address and then has nowhere to send the packet, so the client gets ICMP port unreachable. It never self-heals: the lease holder keeps renewing after the backend moves away. Measured on prod 2026-09-14: unifi's VM ran on k3s-prod-1 with both leases on k3s-prod-2, and crafty's pod ran on k3s-prod-1 with the -tcp lease on k3s-prod-2 while its identical -udp twin happened to draw k3s-prod-1 and worked. Confirmed causally on the announcing node with `cilium-dbg service list`: under Local the frontend 192.168.2.22:25565 is absent from that node's service table entirely; under Cluster it appears as `1 => 10.0.0.11:25565 (active)`; flipped back, it vanishes again. Durability confirmed by cordoning k3s-prod-1 and moving the crafty pod to k3s-prod-3, after which the announcing node re-pointed to the new backend with no intervention. Both files also carried a comment claiming Local "pins the VIP to the node running the pod", which is the precise thing that is false here; both are rewritten. apps/wave-0/pihole-dns.yaml is the reference implementation. The client-IP cost is stated per service rather than waved away. For unifi it is ZERO: the VM's KubeVirt `masquerade` binding already SNATs everything into the guest, so Local was never preserving a client address the controller could see. For crafty it is real and accepted -- player identity is a Mojang/Xbox UUID so /ban and the whitelist are unaffected, but /ban-ip would hit every player at once and must not be used; no IP ban or per-IP rate limit is configured today. No loadBalancerSourceRanges here, deliberately departing from pihole-dns.yaml: these Services never had a source restriction (their CNPs admit `world`), so Cluster removes no protection, and adding a LAN-only filter during an outage fix could cut off off-LAN players or a remote site's device informing. No CiliumNetworkPolicy change is needed, and that was verified rather than assumed: Cilium's `cluster` entity subsumes `remote-node`, confirmed by dumping the crafty endpoint's datapath policy map with the CNP unmodified (`reserved:remote-node` was already an Allow on the game ports). Unlike pihole, whose policy omits `cluster` and must name remote-node explicitly. Adds an L0 whole-repo guard for the class. It fails on the pre-fix tree with exactly these four Services named and passes after. whole-repo scope is necessary: ci/touched-requirements.sh has no arm for apps/* or platform/crafty/*, so the change most likely to reintroduce this would attribute no requirement and be gated by nothing. HONEST LIMIT -- this does NOT end the reported outage. All four ports also have no listening backend: UniFi OS was never installed in the VM (its README flags this as the open blocker) and /crafty/servers is empty, so no Minecraft server exists. Probed directly from the node, bypassing the VIP entirely, 10.0.0.74:8443/:8080/:11443 and 10.0.0.11:25565 all refuse, while the two controls answer. This commit fixes a real latent defect that would have blackholed those VIPs once the applications exist; it cannot make an absent application answer. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-14 01:32:47 -05:00
jakman85andClaude Opus 5	4080065aa7	feat(dns): `exposure: internal` can no longer publish a public record docs/09 R-DNS-03 and schemas/renderer-contract.md both said an `internal` service renders no public DNS record. Nothing enforced either sentence. ExternalDNS's `gateway-httproute` source admitted EVERY HTTPRoute on the shared Gateway, so six `exposure: internal` services — jackett, pihole, qbittorrent, radarr, semaphore, sonarr — published `<name>.home.keithjackson.dev` into the public authoritative Cloudflare zone answering with the Gateway VIP 192.168.2.20. That published the internal subnet, the Gateway VIP and a full service inventory to anyone who could guess a label, and made each of those names a DNS-rebind primitive — which is why the SOHO router strips RFC1918 answers and LAN clients need per-client dnsmasq overrides to reach their own services. THE MECHANISM, established against the DEPLOYED version rather than from memory. The live prod Deployment runs registry.k8s.io/external-dns/external-dns:v0.21.0. In v0.21.0, source/annotations/processors.go IsControllerMismatch() skips a source object whose `external-dns.alpha.kubernetes.io/controller` annotation is present and != its own `dns-controller`, and source/gateway.go applies that check inside Endpoints() BEFORE any hostname is resolved; source/service.go applies the identical check, covering the second configured source. catalog/render.py now emits that annotation on every HTTPRoute it renders for an `internal` service, derived from `exposure` alone — no per-service list — so a service that omits `exposure` inherits the internal default and is unpublishable too. Verified LIVE on prod on a throwaway hostname with no existing record at risk: the annotated route published nothing, and adding the annotation to a route that already had a record produced action=DELETE for both its A record and its `a-` TXT registry entry one ~60s reconcile later, moving nothing else in the zone. WHY PER-RESOURCE AND NOT A CLUSTER-WIDE OPT-IN FILTER. `labelFilter` is the stronger default posture and v0.21.0 supports it for both configured sources, but nine publishing objects here are hand-authored platform/apps manifests with no `exposure` field, seven of which hold live records (auth, crafty, crafty-mc, grafana, headlamp, the apex, unifi). An opt-in filter deletes all seven at once, and whether each should be publicly resolvable is an operator decision per name. Per-resource exclusion has exactly the blast radius of the defect; the seven are enumerated in a shrink-only register so the unadjudicated set cannot grow silently, and the filter stays available as hardening afterwards. `policy: sync`, `txtOwnerId` and `domainFilters` are UNTOUCHED. Per-resource exclusion only ever removes an object from this instance's view, so nothing here makes prod capable of deleting a record it does not own. WHAT THIS DELETES, and it is not reversible by a revert alone: 12 records — the A record and the `a-`-prefixed TXT registry entry for each of jackett, pihole, qbittorrent, radarr, semaphore and sonarr. The zone has no `.home.keithjackson.dev` wildcard and `.keithjackson.dev` does not match two labels, so each becomes a public NXDOMAIN with no fallback. The complete pre-state is captured at the authority and committed as a recovery baseline (.standup-dns/dns-20260914-pre-externaldns-scope.txt), with the restore procedure for both halves of a record. MERGE ORDER IS NOT OPTIONAL: feat/pihole-lan-dns-authority must land and be verified first. Removing these records while the LAN still depends on them, with the router stripping RFC1918 and no internal authority yet, breaks name resolution outright. The five `exposure: public` services are LEFT UNCHANGED and still answer with the Gateway VIP, so they remain publicly unreachable. That is two open operator decisions, both written up in platform/externaldns/helmrelease.yaml: what address they should answer with (136.51.115.252 is the legacy non-k3s stack serving a placeholder cert, not established as correct for this cluster), and how to separate them from the non-public names that share the Gateway — because in v0.21.0 the target annotation is read only from the GATEWAY and a ListenerSet, never from an HTTPRoute, so a per-service override is not expressible while they all attach to `shared`. No knob is added for it yet: one that cannot take effect invites someone to set it and expect one. Guard: conformance/tests/l0-static/internal-services-publish-no-public-dns, a whole-repo-scoped L0 static check (no ci/l0.sh stage — the whole-repo-invariants-wired meta-test forbids one). It asserts both directions (present on internal, absent on public), the `service`-source half, that the renderer DERIVES the annotation from `exposure` and cannot emit external-dns's own claiming value, proves the shipped renderer end-to-end on internal/public/omitted-field fixtures, and ratchets the non-catalog register. Proven to fail on the pre-fix tree (9 of 53 assertions, naming all six services) and pass after (56/56). Schema 1.6.0 -> 1.7.0: no field changes, a mapping change to `exposure`. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-14 00:43:12 -05:00
jakman85andClaude Opus 5	3a72b07cda	chore(conformance): shrink UNTESTED_ALERTS by the two PSI detectors node-io-cause.test.yaml's healthy fixture asserts NodeIOPressureHigh and NodeIOPressureCritical quiet on the same measured prod state the eight new cause rules are pinned against — the point of that fixture being that the detectors and the cause rules must agree about what "healthy" is. The shrink-only register entries are therefore stale, and the gate correctly refused them. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 22:38:09 -05:00
jakman85andClaude Opus 5	b9f3143f42	feat(monitoring): name the CAUSE of node I/O pressure, not just the stall NodeIOPressureCritical has now fired twice on k3s-prod-1 for opposite causes and said the same thing both times. On 2026-09-14 02:40-02:45Z it fired at PSI(full) 20.98% — twice the threshold, four times the 2026-09-07 level — while every local device was idle (0.09-0.27 utilisation) and fast (1.4-4.8 ms per write), NFS retransmissions were zero and no Longhorn volume was degraded. Establishing that took eight ad-hoc PromQL queries after the event had already recovered. Adds the group platform-health.node-io-cause: eight rules, one per cause class, each naming its cause in its own summary. NodeDiskDeviceSaturated device saturation >0.85 busy, 10m NodeDiskWriteLatencyHigh device latency >20ms/write, 10m NodeDiskWriteLatencyCritical device latency >50ms/write, 5m NodeDiskReadLatencyHigh device latency >250ms/read, 15m NodeNFSRetransmissionRateHigh NFS transport >1% retransmitted, 10m NodeIOPressureNotLocalDisk not local disk PSI>10% with all of the above absent NodeEtcdRequestsSlow etcd >0.5% of etcd requests over 1s ContainerIOStallHigh which workload per-container PSI >15% Every threshold is measured on prod over 24-48h, not guessed, and each rule records its headroom. The PSI thresholds themselves are deliberately unchanged: the 21% stall was real, it fired twice in 24h for two real events on the one node that has a problem, and raising the bar would also have raised it above the 11.36% at which prod-1 was already losing etcd on 2026-09-07. Context reaches the operator at page time three ways: the summary states the cause class; every rule carries a diagnostic_query annotation with the exact companion PromQL; and node-pressure-dashboard.yaml gains a WHY row (seven panels) plotting saturation, write/read service time, NFS retransmissions, our own NFS op rate, the etcd slow-request fraction and per-container PSI side by side. runbooks/node-io.md gains a 10-row triage table with the real healthy / 2026-09-14 / 2026-09-07 columns, which turns those eight queries into a two-minute read. etcd and NFS metric availability, verified on prod rather than assumed: * etcd SERVER disk metrics do NOT exist. etcd_disk_wal_fsync_duration_seconds and etcd_disk_backend_commit_duration_seconds are absent (kubeEtcd disabled for k3s); a query for etcd_disk_/etcd_server_/etcd_mvcc_* returns nothing at all. NodeEtcdRequestsSlow uses the apiserver's client-side view of its own local member instead, and says in its own description that it is a proxy. * node_nfs_* DOES exist — the nfs collector is on by default, host-wide via /proc/net/rpc/nfs, confirmed on all three nodes. It gives op counts and retransmissions and NO latency, so it can see a dropping export but not a slow one. The mountstats collector is enabled here to close that gap; NO rule references node_mountstats_* yet, because the series do not exist until the HelmRelease rolls out and the collector's visibility through the node-exporter mount namespace must be verified on prod first. The runbook records the check and says to revert if it comes back empty. Proof, per the repo's two shipped alerting failure modes: * platform/monitoring/tests/node-io-cause.test.yaml — 7 fixtures, all from measured profiles: healthy (all 8 quiet), 2026-09-14 (NotLocalDisk fires and names sonarr), 2026-09-07 re-queried at offset 164h (write-latency and etcd fire, NotLocalDisk correctly silent), rebuild storm, NFS transport, read latency, and the `unless` semantics. * 10 mutations of the shipped rules, all killed. The mutation run found a real gap: the etcd threshold could be changed without failing any test, because it was only pinned by a promql_expr_test that restated the threshold in the test file. An alert_rule_test now pins it. * All 11 referenced metrics verified to return series on prod with `promtool query instant`, and all 8 expressions verified quiet on today's state. Also extends the alert-metrics-have-scrape-targets owner table for container_* (the kubelet ServiceMonitor's cAdvisor path, which this entry already claimed in its title) and for etcd_request_duration_seconds_* by EXACT NAME, deliberately not by an etcd_ prefix, so a future rule on the unscraped etcd_disk_* names still fails the gate. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 22:35:19 -05:00
jakman85	52398d4426	merge: Longhorn alerting measured the wrong things, three ways gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details (1) LonghornVolumeCapacityHigh alarmed on ALLOCATION while calling itself a capacity alarm. Prometheus's TSDB volume read 0.973 while the kubelet saw the same filesystem at 0.208, because 37.73 GiB of it is a Longhorn SYSTEM snapshot — the kind taken for the full replica rebuild that prod-1's drive replacement forced — growing +152 MB/h while the filesystem stayed flat. The rule was therefore unclearable by design, i.e. permanently firing. Deleted as redundant, shown rather than asserted: 23 of 24 volumes fall under the kubelet-based filling rules, and the 24th is volumeMode Block, which the deleted rule was not covering either. (2) LonghornVolumeDegraded and LonghornVolumeFaulted could not fire even with the scrape working. longhorn_volume_robustness on 1.12.1 is ONE-HOT over a state label with values 0/1, not the 0-3 enum the rules and their comment assumed, so a faulted volume would have alerted nobody. This survived the scrape fix landing hours earlier. (3) LonghornNodeStorageHigh was wrong twice: its 0.85 threshold sat above the 75% at which storage-minimal-available-percentage=25 already removes a disk from the scheduling pool, and longhorn_node_storage_usage_bytes SUMS a node's disks, so one full disk hid behind a half-empty sibling. Replaced with per-disk LonghornDiskAllocationHigh (>0.70) and LonghornDiskUnschedulable, the latter reading Longhorn's own verdict instead of re-deriving a formula from three settings no metric exports. Baseline green with 11/11 mutations red. retentionSize 45GiB of 50Gi and discard/TRIM are both REPORTED with recommendations, not changed: TRIM would not have helped here, since remove-snapshots-during-filesystem-trim is false and the 37.7 GiB is a snapshot. # Conflicts: # DECISIONS-LOG.md	2026-09-13 17:23:37 -05:00
jakman85andClaude Opus 5	4687521d65	fix(monitoring): delete the Longhorn rule that measured allocation and called it capacity LonghornVolumeCapacityHigh went pending within minutes of the Longhorn scrape landing today. Re-measured on prod: it read 0.9731 on the Prometheus TSDB volume (actual_size 52243681280 / capacity 53687091200) while the kubelet reported the SAME filesystem 20.8% used. actual_size is head + pinned snapshots, and one user_created=false system snapshot from prod-1's replica rebuild holds 40510484480 bytes (37.7 GiB) of it; Longhorn returns no freed blocks without discard/TRIM, and retentionSize 45GiB on a 50Gi volume targets 90% by design. The ratio is pinned near 1.0 and cannot fall back under 0.85 — a permanently-firing alert, the same shape as CNPGNoRecoverabilityPoint. DELETED, not re-thresholded. Raising the threshold would tune away a true statement about allocation. Redundancy proven, not asserted: of 24 Longhorn volumes on prod, 23 appear in kubelet_volume_stats_inodes > 0 and so fall inside MonitoringVolumeFillingUp (namespace="monitoring") or PersistentVolumeFillingUp / PersistentVolumeAlmostFull. The 24th is unifi/unifi-rootdisk, volumeMode Block — no filesystem for anyone to measure, and the deleted rule was not covering it either. AND A SECOND INERTNESS the scrape did not fix. longhorn_volume_robustness on longhorn 1.12.1 is one-hot over a `state` label with values 0/1, not the 0-3 enum the rules assumed: count(longhorn_volume_robustness) => 96 (24 volumes x 4 states) count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24 count(longhorn_volume_robustness == 2) => (empty) count(longhorn_volume_robustness == 3) => (empty) So a FAULTED volume would still have alerted nobody. Re-keyed to the label; the "44 series / 72 unknown(0, detached)" note in platform/longhorn/helmrelease.yaml was the same misreading and is corrected in place. The allocation question moved to where allocation is a budget. LonghornNodeStorageHigh fired at 0.85, but storage-minimal-available-percentage is 25 on this cluster, so a disk leaves the scheduling pool at 75% used — the old threshold could only fire after scheduling had already been refused. Replaced by: LonghornDiskAllocationHigh (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70, 30m LonghornDiskUnschedulable longhorn_disk_status{condition="schedulable"} == 0, 15m Per DISK, not per node: longhorn_node_storage_usage_bytes sums a node's disks, so one full disk could hide behind a half-empty sibling. Measured prod usage 0.2254 / 0.2306 / 0.2369 on prod-1/2/3. PROVEN NEITHER PERMANENTLY FIRING NOR INERT. platform/monitoring/tests/longhorn.test.yaml adds 6 cases pinning every rule quiet on today's measured prod state and firing on a constructed bad state, with every `for:` asserted, plus regression assertions that the old `== 2`/`== 3` forms match nothing and that per-disk 0.72 is visible where the node-summed 0.475 is not. Mutation-checked 11 ways; each turns the suite RED. Live on prod: all four new expressions return empty, while the sum of their selector cardinalities is 54. The L0 UNTESTED_ALERTS register shrinks by seven (7/56 -> 14/58 alert rules unit-tested), which also brings MonitoringVolumeFillingUp, PersistentVolumeFillingUp and PersistentVolumeAlmostFull under test since the redundancy claim rests on them. REPORTED, NOT CHANGED (DECISIONS-LOG, owner decisions): retentionSize 45GiB on a 50Gi volume is too tight — recommend 36GiB, with reasoning; and discard/TRIM on the longhorn StorageClass is feasible but not advisable now — remove-snapshots-during- filesystem-trim reads false on prod, so trim would not have reclaimed the 37.7 GiB anyway, and continuous discard on nodes where the etcd WAL shares the spindle points at this cluster's known failure mode. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 17:19:03 -05:00
jakman85andClaude Opus 5	cf7d97e546	fix(obs): twelve app SLI rules read a signal that exists; kill the dead ServiceMonitors Twelve `<app>-down` SLI rules were STRUCTURALLY INERT. Each read `up{job="<app>"} == 0`, and the ServiceMonitor rendered beside it named `endpoints[].port: metrics` — a port NO rendered app Service defined. Every monitor discovered ZERO targets, so no `up{job="<app>"}` series existed and the rule could not fire. Verified on prod before this change: `count by (job) (up)` listed 24 jobs and not one of the twelve appeared; `up{job="plex"}` and `up{job="gitea"}` both returned nothing. `PrometheusTargetsMissing` could not see it either — a target that was never discovered has no `up` series to be 0. THE PORT NAME WAS NEVER THE MISSING PIECE. Ground truth per app: Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud export NO Prometheus metrics without a separate exporter sidecar; Gitea can but only with app.ini metrics enabled; Keycloak does, on the operator's management port, already scraped by platform/keycloak/servicemonitor.yaml; vm-demo is a KubeVirt VM with no Service at all, so its monitor selected nothing whatsoever. `metrics: true` was asserting a fact none of them had. There was also a THIRD layer of inertness nobody had hit yet: every app namespace ships default-deny with `ingress: []` and the renderer emitted no monitoring allow, so even a correct port would have been dropped by policy — while renderer-contract.md claimed the baseline carried such an allow. FIXED IN THE GENERATOR, not in twelve rendered files: * `slis: [up]` now renders a `<app>-down` alert against kube-state-metrics readiness of the service's own workload object — `kube_deployment_status_replicas_available{namespace,deployment} == 0 or absent(...)`, `kube_statefulset_status_replicas_ready` for a StatefulSet, and `absent(kubevirt_vmi_info{namespace,name,phase="running"})` for a VM. The same idiom KeycloakDown has used since it was fixed. kube-state-metrics IS scraped, so this needs no exporter and no new target. The `or absent(...)` half is load-bearing: the signal DISAPPEARING now fires the alert instead of silencing it. * schema 1.6.0 adds `observability.metricsEndpoint` — WHERE metrics are served. A ServiceMonitor is rendered ONLY from it, and it renders all three halves of a working scrape together: the named Service port, the monitor endpoint, and the monitoring-namespace CiliumNetworkPolicy allow. No catalog entry declares one, so twelve dead ServiceMonitors are gone. `metrics: true` keeps meaning logs-to-Loki plus kube-state-metrics liveness. * schema 1.6.0 also adds `observability.livenessWorkload`, and catalog/keycloak.yaml uses it. Keycloak's server is a StatefulSet run by the operator; the default Deployment shape would have made its `absent()` half fire FOREVER — the other failure mode this repo has shipped. * renderer selftest gains cases 9/9a/9b/9c: no metricsEndpoint => no monitor; the up SLI must not read `up{job=...}`; endpoint port must exist on the Service; the metrics port must NOT reach the Gateway allow; an unknown livenessWorkload.kind is REFUSED. PROOF, both directions, for all twelve (apps/wave-{0,1,2}/tests/app-slis.test.yaml, run by platform/monitoring/tests/run.sh over the RENDERED manifests): * quiet on the healthy state; firing on no-ready-replica; firing when the series is ABSENT; the VM asserted both gone and in a non-running phase; per-app isolation asserted, and keycloak asserted un-maskable by its neighbours (keycloak-operator Deployment, keycloak-db StatefulSet ready in the same namespace). * mutation-checked: six mutations, each turning its suite red — reverting plex to `up{job=...} == 0`; dropping seerr's `absent()` half; pointing keycloak at a Deployment; capitalising the VMI phase to `Running`; widening keycloak's matcher to the namespace; inverting plex's `== 0` to `!= 0`. Control run green. * live on prod with `promtool query instant`: all ten deployed app Deployments export `kube_deployment_status_replicas_available`, `kube_statefulset_status_replicas_ready{namespace="keycloak",statefulset="keycloak"}` = 2, `kubevirt_vmi_info` present with a lowercase `phase="running"`; and evaluating the twelve SHIPPED expressions returns no series for eleven (quiet, not permanently firing). The twelfth is vm-demo, which is not in apps/wave-0/kustomization.yaml and so is not deployed — its rule is not applied either. DELETION WAS NOT THE ANSWER, and the evidence is recorded rather than assumed: kube-prometheus-stack's `kubernetes-apps` group IS live on prod (17 rules, KubePodNotReady / KubeDeploymentReplicasMismatch among them), so app pods are not unmonitored — but that is a generic warning keyed to a namespace, not the per-service CRITICAL SLI R-OBS-01 declares. platform/monitoring/alerts.yaml carries no generic app-readiness rule at all. GUARD SO IT CANNOT RECUR SILENTLY: new whole-repo L0 invariant conformance/tests/l0-static/servicemonitor-ports-resolve. It walks the MONITORS, not the alerts, so it needs no `up{job=...}` rule to reach one, and resolves each endpoint by SELECTOR across the whole tree rather than by same-file adjacency. Fixing the alerts DELETED the only thing that made the broken monitors visible to the sibling check, which is exactly why this axis needed its own home. It declares `conformance.home.k3s/scope: whole-repo` and adds no ci/l0.sh stage. Proven to FAIL on the pre-fix tree: 15 failures — eleven port disagreements, vm-demo's monitor matching no Service at all, and the three generator/contract assertions — and to pass after. Its resolver is additionally proven on in-check fixtures (agreeing pair accepted, disagreeing pair rejected, target-less endpoint rejected), because after the fix there are zero in-repo monitors left for it to judge and a guard whose only evidence is "nothing failed" cannot be told apart from one that stopped working. It also asserts r_servicemonitor does not HARDCODE a port name — the mechanism by which one edit gave twelve services the same wrong one. Also fixed, found on the way: * platform/monitoring/tests/run.sh discovered the SUITE convention only inside platform/monitoring/tests/ while documenting it as general — a suite added anywhere else was never run and still looked like coverage. Now discovered tree-wide; a misplaced .test.yaml fails loudly instead of vanishing. KNOWN_UNWIRED_JOBS in alert-metrics-have-scrape-targets is now EMPTY, and the twelve apps/* entries in alert-rules-unit-tested's UNTESTED_FILES are gone — both because the gaps were CLOSED, not waived. Both registers are shrink-only and a stale entry fails. * renderer-contract.md claimed the default-deny baseline carried a standing monitoring scrape allow. It never did; the claim is corrected and the allow is now real, per declared metrics endpoint and scoped to that one port. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 17:11:30 -05:00

1 2 3