f9446be381fb825f92ab1a5b9630cdebcde0de88
87
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
f9446be381 |
fix(obs): open 44180 to ns monitoring so the forward-auth gate sidecars are actually scrapable
platform/oauth2-proxy/podmonitor.yaml landed 2026-10-02 and discovered six targets that have never once scraped. Measured on prod 2026-10-04: all six read health=down with lastError "context deadline exceeded" and lastScrapeDuration exactly 10.0s — the configured scrapeTimeout, i.e. packets DROPPED rather than refused. Cause: every app namespace ships a default-deny NetworkPolicy with policyTypes: [Ingress, Egress] and NO ingress key at all (deny-all-ingress), and the only ingress allowance is <app>-allow-gateway-ingress: fromEntities: [ingress] -> 4180. Prometheus matched no clause. Verified identical in all six of jackett, radarr, sonarr, qbittorrent, pihole, crafty; zero namespaces had any -allow-monitoring-scrape object. render.py already had the right idiom — r_monitoring_ingress_policy — but it returned None unless the service declared observability.metricsEndpoint, and NO ServiceDefinition in this catalog declares one. It was wired in at the call site and had never emitted a single object on any branch: correct, and entirely latent. It now also fires for the forward-auth gate sidecar's own metrics port, keyed on _needs_oauth2_sidecar — deliberately the same condition the scrape keys on, since podmonitor.yaml selects pods purely by the auth-mode label, so a target exists iff the sidecar exists. Keying on observability.metrics instead would let metrics: false produce a discovered-but-blocked target: the same "two halves authored apart" failure this function exists to prevent. Rendered here rather than in platform/oauth2-proxy because that tier would have to name app namespaces created by the sibling apps-wave-0/1 tiers — it would apply on a running cluster and fail on a cold bootstrap. podmonitor.yaml's header records that objection; rendering per-app dissolves it. Five CNPs result, one per forward-auth app, each opening only 44180 to only ns monitoring. apps/ re-rendered with the invocation it was generated by (--no-image-automation, verified byte-identical beforehand); --emit-waves deliberately not run, so no wave kustomization moved. Three L1 goldens are regenerated with it (+63 lines, -0): a renderer change that adds an object necessarily drifts the goldens that pin the renderer's output. The first gate run caught container-workload-render and egress-class-render as FAIL with exactly this CNP as the drift. default-deny-baseline-render's golden regenerated byte-identical, which is the evidence that the regen is faithful and not a rubber stamp. That puts conformance/** in the diff, which is a sensitive path: this PR needs a non-author approving review. The local gate cannot see that — offline it reports approval-policy green (N/A). Local toolbox gate: rc 3 UNDETERMINED, not 0. Every stage that ran is green and l1 went FAIL -> pass with the goldens; the only not-run stage is gitea-settings (no GITEA_SETTINGS_TOKEN). L2/L3 conformance is UNVERIFIED for all 8 touched requirements. Not verified: that the scrape now succeeds. Nothing is applied until Flux reconciles main. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
4f16e83eec |
Merge pull request 'feat(obs): make oauth2-proxy auth failures visible — the gates emit nothing' (#29) from feat/oauth2-proxy-auth-failure-visible into main
Reviewed-on: #29 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
99560c2d22 |
feat(auth): declare API paths per service, so --api-route can ship at all
PR #28 fixed the cookie collision, the 5-minute session and the missing metrics
listener in one renderer function, and deliberately declined `--api-route`
because the config model could not express the difference between two apps.
This adds the hook it was missing and sets the flag only where a measurement
says to.
THE DEFECT. oauth2-proxy answers an unauthenticated request with a 302 to
Keycloak. Its AJAX guard would send 401 instead, but it fires only on an EXACT
`Accept: application/json`, which these apps' JavaScript does not consistently
send. So a browser XHR gets a 302, FOLLOWS IT INVISIBLY, has it satisfied
silently by the live SSO session, and resolves 200 carrying the app's HTML where
the JavaScript expected JSON. On a write it is worse: a browser following a 302
re-issues a POST as a GET and DROPS THE BODY, so the operator's change is
discarded and the UI reports success.
MEASURED, on two apps and two different paths -- which is the whole argument:
jackett 2026-09-30: 33 x 302 vs 2 x 401 on /api/ paths, and 4 of 6 POSTs
answered by the proxy and never reaching Jackett (upstream `-`).
radarr 2026-10-01, 23h after PR #28 deployed (pod radarr-7d55cbffb9-42hmh,
0 restarts): 232 unauthenticated requests in the pod's whole life and
ALL 232 are the same call -- POST /signalr/messages/negotiate from a
browser, 302, upstream `-`. A left-open tab whose session the cookie
rename orphaned, polling forever, silently redirected every time and
NEVER re-prompting. The reported symptom, still in production.
So radarr's broken path is /signalr/ and jackett's is /api/, and PIHOLE SERVES
NO /api/ AT ALL. One function renders all five sidecars, so the cheap fix is a
constant -- and a constant applies one app's measurement to four apps nobody
measured. `auth` was `additionalProperties: false`, so there was no per-app
hook. Adding one named, documented key is the change; the block stays closed.
schemas/servicedefinition.schema.json 1.8.0 -> 1.9.0: optional
`auth.apiRoutes`, a list (radarr needs two, and --api-route is natively
repeatable) of anchored path regexes. Plus the shape invariant that a
non-empty list requires mode: forward-auth, since no sidecar is rendered
for oidc/native/none and declaring it there would be a SILENT NO-OP.
catalog/render.py emits one --api-route per entry, in order, and RAISES on
an unanchored entry. The flag is SINGULAR -- verified against the pinned
v7.8.1 binary's own --help; --api-routes does not exist and the proxy
refuses to start with it.
catalog/{jackett,radarr}.yaml the two measured apps.
catalog/{pihole,sonarr,qbittorrent}.yaml comments recording that they get
NOTHING, and why. Sonarr and qBittorrent were measured (0 unauthenticated
requests in 23h) and left alone: that is evidence of no TRAFFIC, not of no
need, and sonarr being nearly radarr's codebase is what makes the guess
tempting and still a guess.
apps/wave-0/jackett.yaml, apps/wave-1/radarr.yaml regenerated. Only TWO
goldens change; pihole/qbittorrent/sonarr are byte-identical to main,
which is the per-app property proving itself on the real tree.
BREADTH IS THE DANGEROUS DIRECTION AND IT IS NOT SYMMETRIC. A missing
--api-route is a silent 302. A too-broad one answers a plain PAGE LOAD with
401, so a logged-out operator gets a bare error page and is never offered a
sign-in -- the service looks permanently broken. Each entry must therefore
match `^\^/[A-Za-z0-9._~/-]+(\.\*|\$)?$`: anchored at ^/ then literal path
characters. `^/`, `^/.*`, `.*`, `/api/` and `^/api/|^/` are all rejected, all
five provoked. Alternation is refused rather than supported because the flag is
repeatable, so one entry per route costs nothing.
NEW GATE CHECK: conformance/tests/l0-static/oauth2-proxy-api-routes-per-app
(whole-repo scope, so ci/l0.sh's --include-scope union runs it on every
invocation; no bespoke stage, which whole-repo-invariants-wired would fail).
It is a SIBLING of oauth2-proxy-cookie-names-distinct, not an extension of it:
that check's subject is cookie IDENTITY and shares neither inputs nor failure
mode with this, so bundling them would make its name a lie and let a future
api-route decision be expressed only by weakening a COOKIE check.
The check exists for one reason the other gates cannot cover: render-golden
compares the render to the goldens, so a RENDERER CONSTANT plus regenerated
goldens AGREE WITH EACH OTHER while both being wrong. Provoked: the regen exits
0 -- render-golden is happy -- and this check exits 1 with 15 failures naming
pihole, qbittorrent, radarr and sonarr individually. It also asserts the
anchoring pattern is the SAME in the schema and the renderer, because three
surfaces guard this and if they drift the loosest wins silently.
Tested in both directions on throwaway trees outside the repo: compliant tree
rc 0 / 43 assertions / 6 sidecars; pre-change tree rc 1 / 11 failures; and four
targeted mutations (hand-edited golden, widened regex, renderer constant,
deleted schema pattern) each firing their own assertion class and nothing else.
ALSO FOLDED IN (coordinator's request; defect is in the file I own). _selftest()
reached _cookie_name() through a bare `next(genexpr)` with NO default, so
deleting the --cookie-name emission -- the exact mutation that assertion exists
to catch -- raised StopIteration, which escaped _selftest() BEFORE its collected
`SELFTEST FAIL:` lines were printed. Right verdict, zero diagnostics, and it
masked every other assertion the same mutation broke (found by Desmond Oyelaran
while wiring the selftest into the render-golden stage; his PR #33 is not in this
base -- origin/main @
|
||
|
jakman85andClaude Opus 5
|
bce101d03f |
feat(obs): wire the gate-sidecar scrape, and fix the alert that would have gone silent doing it
PR #28 gave the five fleet oauth2-proxy sidecars --metrics-address=0.0.0.0:44180 and a named `metrics` port. This wires a scraper to them and, in the process, found a real bug in the alert this branch added. PodMonitor, not ServiceMonitor. Measured: no Service exposes 44180 on any of the five (jackett/radarr/sonarr/qbittorrent/pihole all publish only their app port + authproxy 4180), so a ServiceMonitor would select nothing. A PodMonitor targets pod ports directly and needs no change to catalog/render.py or the apps/wave-* renders and goldens. Selector is the pod label homelab.keithjackson.dev/auth-mode=forward-auth — present on exactly those five gate pods, absent from crafty's, which still has no metrics listener. The property, not an app list. THE BUG: `up{container="oauth2-proxy"}` -> `up{container="oauth2-proxy"} == 1`. A target that is discovered but FAILING still publishes `up`, with value 0. The series exists, so count by (namespace) returned 1, `unless` removed the namespace, and OAuth2ProxyGateUnobserved went QUIET while nothing was being scraped at all — this branch's own disease reproduced inside the alert written to cure it. Not hypothetical: the PodMonitor lands five targets whose scrape is dropped, so the bare form would have silently cleared on this very commit. Proven by mutation: reverting the `== 1` fails the two new unit-test scenarios (got: nil) and leaves every other assertion green. The scrape does NOT work yet, and that is documented rather than papered over. From a pod in ns monitoring to the jackett pod, :44180 AND :4180 both time out; the shared proxy's :44180 answers. ns jackett's default-deny is podSelector:{} with policyTypes:[Ingress,Egress] and NO ingress key (deny-all), and the one allowance is fromEntities:[ingress] to 4180 — the Gateway's Envoy only. The endpoint itself is healthy: port-forwarded, /metrics returns http=200, 10208 bytes, all three oauth2_proxy_* families including requests_total{code="302"}. Same bug and fingerprint as platform/longhorn/networkpolicy-metrics.yaml records for Longhorn; the fix is one narrow rule per app, and it cannot ship from this tier because oauth2-proxy and apps-wave-0 are Flux SIBLINGS (both dependsOn platform-config), so naming ns jackett here would apply today and fail on a cold rebuild. Needs an owner in the renderer. Touches conformance/** : one KNOWN_EXTERNAL_MONITORS entry, which is the mechanism the failing check explicitly instructs ("DECLARE it ... with who creates it") and which flux-podmonitor.yaml already uses. It records that the port name was measured on prod instead of resolved from a Service, and that the targets are currently down. That makes this PR require a non-author review on the sensitive-path rule. Also: ratio alert still NOT shipped. oauth2_proxy_requests_total now exists on the endpoint but not in the TSDB, so writing it today would make it inert for exactly the reason it was refused on 2026-09-30. Runbook gains policy-vs-no-target triage and the radarr /signalr/messages/negotiate worked case — one orphaned browser tab, 206 identical 302s, nobody ever re-prompted. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
dc136a4943 |
fix(auth): the forward-auth sidecar also died at 5 minutes and served no metrics
Widens the cookie-collision fix to the whole sidecar contract, because all three
defects live in the same function and land on the same five goldens, so they are one
atomic change or they are five conflicting ones.
Added to _oauth2_sidecar_container(), on top of the cookie-name/cookie-domain fix:
--cookie-refresh=2m30s
The session died FIVE MINUTES after sign-in. Keycloak's homelab realm issues
300-second access tokens (MEASURED on a live session: expires - created =
300.000s exactly, with refresh_token:true on the same line), and
--cookie-refresh defaults to 0 -- which the pinned binary's own help spells
"0 to disable". So the proxy held a refresh token it was never told to spend.
Refresh at HALF the lifetime: two chances to renew before expiry.
This, not the collision, is what dominates the rejection VOLUME -- ~95% of
radarr's 155 rejections in 12h carry no cookie at all.
--metrics-address=0.0.0.0:44180 + a `metrics` container port
Without the flag the /metrics listener never starts, so no per-app sidecar was
serving metrics at all (`ports` was ['http'] on all six). 44180 is the
platform's existing convention, measured not chosen: the shared proxy already
runs the same flag and port with 2 ready endpoints.
PREREQUISITE ONLY -- NOT scraped. A container port is not a Service port, and
each namespace's default-deny baseline is `ingress: []`, so this is unreachable
until a Service port (or PodMonitor), a monitoring allow and a ServiceMonitor
land. Do not read this as "the sidecars have metrics now".
EVERY FLAG VERIFIED AGAINST THE PINNED v7.8.1 BINARY (`--help`), not against docs or
the shared proxy -- which runs v7.15.3 and could not have vouched for v7.8.1:
* --metrics-address exists in 7.8.1, so this cannot crashloop the pods
* --cookie-refresh ... "0 to disable" confirms the default
* --cookie-name ... (default "_oauth2_proxy") -- this confirms from the
IMPLEMENTATION the constant the new L0 check is built on
* --api-route is SINGULAR; the plural does not exist and the proxy refuses to start
* --session-cookie-minimal exists and is mutually exclusive with --cookie-refresh,
so dropping --pass-access-token is a least-privilege argument, never a size fix
NOT done, deliberately: --api-route=^/api/ renderer-wide. It was measured on jackett
and PiHole does not serve /api/, so a renderer constant would apply one app's path
shape to five. It needs a per-app hook the `auth` schema forbids
(additionalProperties: false) -- a schema change, separate PR.
Also records the mechanism that made this INTERMITTENT, as a renderer comment, because
nobody would reconstruct it from the flags: the loser does not just fail to read the
cookie, it calls ClearSession, which expires the cookie on its own --cookie-domain --
the whole zone. The *arr UIs poll (/api/v3/command, /api/v3/queue/details,
/signalr/negotiate), so every poll carrying an undecryptable cookie clears it again:
A LEFT-OPEN RADARR TAB DELETED JACKETT'S COOKIE EVERY FEW SECONDS, from another
namespace, with no user action. That is why signing in again never helped for longer
than the gap between two polls.
Crafty's counter-evidence resolved: the 81:1 ratio was a 24h window straddling a
13h52m-old pod. Bounded to the fixed process, crafty's ENTIRE life is 3 rejections in
6743 log lines, with ZERO `removing session` and ZERO `cookie signature not valid`
against unfixed radarr's 8 and 8. The eviction fingerprint is gone where the fix is
deployed. It logged 0 AuthSuccess too, so that is absence-of-the-failure-mode rather
than a session watched working -- stated as such in the handoff.
Renderer selftest gains six assertions, including a --cookie-refresh check that PARSES
the duration and requires 0 < refresh < 300s (so 2m30s/150s/2.5m all stay green while a
silent widening past the token lifetime does not). NOTE, and it is in the handoff as the
top follow-up: NOTHING IN ci/ RUNS `render.py --selftest`, so those assertions are
dev-time only. conformance/tests/l0-static/auditability-proofs-producible/check.sh:291
already drives other tools' selftests for exactly this reason and is the one-line fix.
Gate (toolbox, worktree shim with the spec repo mounted at /home-network-k3s): rc 3
UNDETERMINED. Green: spec-immutability, decisions-log, lint, l0, render-golden, l1,
approval-policy. Skipped: gitea-settings (no local token), conformance (declared).
The new L0 check PASSES on this branch -- all five apps fixed in one commit.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
9a4489b733 |
fix(auth): five oauth2-proxy sidecars shared one cookie and evicted each other
Every oauth2-proxy on this platform is an independent instance with its OWN
cookie-secret. Measured on prod 2026-09-30: seven `*-oauth2-proxy` Secrets, seven
DISTINCT cookie-secret values (compared in-cluster, never printed). So no instance
can DECRYPT another's cookie -- it can only OVERWRITE it.
All of them were emitting oauth2-proxy's DEFAULT names (`_oauth2_proxy`,
`_oauth2_proxy_csrf`) scoped to `.${BASE_DOMAIN}`, so the browser kept ONE cookie
pair for SIX live instances (jackett, pihole, qbittorrent, radarr, sonarr, and the
shared forward-auth proxy). Whichever authenticated last clobbered the rest; the
loser logged `No valid authentication in request`, bounced to Keycloak, and -- the
SSO session still being live -- was redirected straight back and clobbered it again.
The symptom is "I signed into Keycloak but I'm still not signed in", which sends you
hunting in Keycloak, where nothing is wrong. Crafty was fixed on its own side on
2026-09-29 (#22); this is that fix generalized to the pattern.
THE DEFECT WAS NOT IN THE MANIFESTS. apps/wave-*/{jackett,pihole,qbittorrent,radarr,
sonarr}.yaml are GENERATED goldens; one function emits all five. So the fix is in
catalog/render.py::_oauth2_sidecar_container(), and both flags are now DERIVED from
the ServiceDefinition rather than defaulted:
--cookie-name=_oauth2_proxy_{s['name']} unique by construction
--cookie-domain={host} the app's own host, not the zone
A per-service table was rejected for the reason twelve ServiceMonitors once acquired
one identical wrong port name in a single edit: it would leave the NEXT app onboarded
defaulting straight back into the collision. --whitelist-domain stays zone-wide; it
governs post-login REDIRECT targets, not cookie scope.
Host-scoping is defence in depth and is asserted deliberately: a zone-wide cookie is
transmitted by the browser to every host in the zone, so each app's session token was
being sent to every sibling -- and it is what would make a FUTURE name collision
platform-wide instead of contained.
New gate check (whole-repo scope, so it runs on EVERY gate invocation, not only when
a touched path happens to attribute an identity requirement):
conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct/
It resolves each instance's EFFECTIVE cookie name -- declared, or oauth2-proxy's
built-in default when omitted, because an omitted flag is not an absence but a CLAIM
on a name, and reading it as an absence is exactly why this survived five reviews.
Then: no two collide; none is another's name + `_csrf` (oauth2-proxy derives the CSRF
cookie that way); the default has exactly ONE owner, registered with its reason in a
shrink-only register -- the SHARED proxy, which MUST keep `_oauth2_proxy` because
renaming it signs every other gated surface out at once; every other instance is
host-scoped; render.py derives rather than lists; and the SHIPPED renderer is proven
end-to-end on two synthetic ServiceDefinitions, so the check stays falsifiable once
the tree is clean.
Tested in both directions: rc 0 on this tree (39 assertions, 7 instances), rc 1 with
26 failures against the pre-fix tree, plus one targeted mutation per assertion class
(duplicate name, CSRF derivation, zone-wide domain, unregistered default heir, stale
register entry) -- each fired, and nothing else did.
Also regenerates the three L1 rendering goldens, which are expected-output fixtures
of the same renderer.
UNVERIFIED: no browser round-trip. The collision has still never been REPRODUCED in a
browser -- doing so needs two gated apps visited in one session -- so this is reasoned
from measured configuration and from the mechanism, not from a failure anyone watched.
Nothing was applied; merging rolls five Deployments and invalidates every existing
session on them once. See decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
55b232e2ce |
feat(obs): notify on Flux deploy start and completion
The operator got no mail when Flux began applying a git change and none when it finished. Measured cause: notification-controller v1.4.0 is running and its CRDs serve v1beta3, but there are ZERO Provider and ZERO Alert objects in the cluster — nothing in git ever created any. The obvious fix does not work. Neither Flux event can be filtered into a deploy signal: ReconciliationSucceeded is a heartbeat (101 events in one hour across 16 Kustomizations, message byte-identical whether the apply changed anything), and Progressing is worse (five Kustomizations re-apply the same object on every tick — event counts 2505/2504/2501/2495/2490). gotk_* carries no revision label at all (gotk_reconcile_condition 120 series, gotk_suspend_status 40, none with one). The applied revision is the only signal that moves when and only when git moves, and it lives in object status alone. So export it via the kube-state-metrics custom-resource-state path that already serves the Velero Schedule metrics — kube_flux_kustomization_info/_suspended and kube_flux_gitrepository_info — and ask the question in PromQL: FluxRolloutInProgress (severity: info, no `for:`) fires the moment a non-suspended Kustomization's applied revision is not the revision the GitRepository fetched from main, and CLEARS when all have caught up. Fired mail = a deploy started; RESOLVED mail = it completed. The completion half is send_resolved, not a second rule. FluxRolloutStuck (severity: warning, for: 30m) is the same expression as the escalation for a rollout that started and never converged. Two guards, each with a test whose only job is to fail if the clause is removed: `and on () (count(gitrepository_info) > 0)` — without it a missing exporter makes `unless on (revision) <empty>` remove nothing and all 16 Kustomizations alert forever, the permanently-firing mode this repo shipped once already — and the suspended-Kustomization exclusion, without which a suspended tier holds the rollout open and the "applied" mail never arrives. A dedicated Alertmanager severity=info route + email-deploy receiver keeps deploy notices off the [WARNING] catch-all. podAnnotations carries a roll-counter because the chart sets no checksum over the custom-resource-state ConfigMap, so a config-only edit would leave KSM on the old collector set with the new alerts silently inert. Verified: promtool SUCCESS on 6 scenarios (both guards, plus the convergence test that proves the alert empties itself); lint/l0/render-golden/l1/decisions-log/ spec-immutability green in the pinned toolbox. NOT verified: no mail has been sent — the metrics do not exist until this merges. Full gap list, and the deliberately-cut notification-controller half, in the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
8c7a770de2 |
fix(sec): the R-SEC-02 admission policy was never applied -- and nothing could tell
PR #13 added conformance-trigger-job-shape.yaml to platform/kyverno/admission/ kustomization.yaml. That file applies NOTHING. Its own sibling says so: "kept as reference; THIS file is the buildable render root" -- and that render root (platform/kyverno/kustomization.yaml) ships only the namespace and the HelmRelease, noting that ClusterPolicies are applied a tier later, by platform-config, because they need the Kyverno CRDs first. So three lists exist and only platform-config/kustomization.yaml applies anything. The policy went into the wrong one and EVERYTHING WENT GREEN: the file parsed, kyverno test exercised it by path and passed, the merge gate passed, and Flux reported `platform` Ready at the new revision. Measured on prod afterwards: kubectl get clusterpolicy conformance-trigger-job-shape Error from server (NotFound) while ns conformance, the conformance-trigger ServiceAccount, its Role and its RoleBinding were all live. That is exactly the state that identity's own RBAC header says must never exist -- the trigger identity present with nothing bounding it. Blast radius was small only because no trigger kubeconfig exists yet, which is luck, not design. Wires the policy into platform-config (verified: the render goes from 7 ClusterPolicies to 8 and contains conformance-trigger-job-shape). CLOSES THE CLASS, which matters more than the instance. Adds the L0 check admission-policies-actually-applied: every ClusterPolicy file under platform/kyverno/ is either referenced by platform-config/kustomization.yaml or named in a CI_ONLY list with a reason -- and CI_ONLY may not name a policy that IS applied, so the exemption cannot rot in either direction. Declared scope whole-repo deliberately: a policy wired nowhere touches no requirement the diff attributes, so a path-scoped run would have skipped it on the exact commit that introduced the defect. Proven both ways, not assumed: with the fix reverted -> FAIL, naming conformance-trigger-job-shape as never applied with the fix -> PASS, 9 policy files, 8 applied, 1 CI-only with a reason `kyverno test` proves a policy is CORRECT. This proves it is DEPLOYED. Nothing did before. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
8eb5baffa4 |
feat(sec): constrain the conformance CI trigger at admission (R-SEC-02)
Implements the credential model the operator chose for wiring L2/L3 into CI: a narrow
trigger SA bounded by a fail-closed Kyverno admission policy, rather than the
zero-credential GitOps-native alternative (plan section 4, primary model).
THE ESCALATION THIS CLOSES. Kubernetes places NO `use` check on a Job's
serviceAccountName. Any principal that can `create jobs` in ns conformance can create a Job
running as conformance-runner -- nodes/patch, pods/exec, namespaces/patch -- with an
arbitrary command. A CI token with plain `create jobs` IS that privileged identity no matter
how narrow its own Role looks. RBAC cannot fix this; only constraining the SHAPE of what the
trigger may create can.
Adds:
* SA/Role/RoleBinding conformance-trigger -- create+get+list+watch jobs, get pods and
pods/log, nothing else. Namespaced. No delete (a trigger that can delete can hide a
failed run), no exec, no secrets. Results come back as a tarball on stdout read via
pods/log, so no execution channel into the runner pod is ever granted.
* ClusterPolicy conformance-trigger-job-shape -- Enforce, failurePolicy: Fail. It does NOT
use ${KYVERNO_VALIDATION_ACTION}: a security boundary an overlay can downgrade to Audit
is not a boundary. It constrains ONLY conformance-trigger, so a mistake here cannot wedge
the rest of the cluster.
* Flux Kustomization `conformance` -> ./conformance/incluster, wait:false and non-gating
like apps-vms, so a test identity can never stall the platform or app tiers.
* A kyverno test with seven cases, six of them real escalation paths.
VERIFIED by `kyverno apply` (pass: 10, fail: 6, error: 0), not assumed:
evil-command DENIED at /spec/template/spec/containers/0/command/ <- THE attack
evil-image DENIED at .../image/
tag-not-digest DENIED at .../image/ (mutable tag is republishable)
host-namespaces DENIED at /spec/template/spec/hostPID/
extra-volume DENIED by the volumes foreach
injected-args DENIED by the args foreach
good-run ADMITTED
ALSO FIXES A FAIL-OPEN IN THE GATE ITSELF. `kyverno test` does NOT fail when a policy cannot
LOAD: every row becomes "Skip / Invalid Policy" and it still prints "N tests passed" and
exits 0. Measured while writing this policy -- a schema error (subjects in a separate
match.all entry from resources) produced exactly that, and ci/l0.sh would have certified a
security policy that does not load as green. This is not specific to this policy: a typo in
restrict-host-namespaces or secret-provenance would silently disable it with the gate still
green. ci/l0.sh now scans for "Invalid Policy" and fails. Proven: broken policy -> l0 rc=1,
restored -> rc=0.
IMAGE PINNING, HONEST LIMIT: the plan specifies pinning the exact runner image digest. That
image does not exist yet, so this enforces the enforceable half -- the image must be under
the conformance repository path AND digest-pinned, never a mutable tag. Residual surface:
anyone who can PUSH to that path could publish a different digest. Tighten to the literal
digest when the image is built; it is a one-line change to the pattern, reviewed in a PR,
which is where a change to a security boundary belongs.
NOT YET WIRED, deliberately: no runner image, no trigger kubeconfig, no CLUSTER_AVAILABLE.
The identities here are inert until those exist, and PR #12 makes CLUSTER_AVAILABLE=true
block rather than fake a green in the meantime.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
712f4c9336 |
fix(conformance): "unit-tested" counted an alert MENTIONED in a comment as tested
alert-rules-unit-tested decided whether an alert had a unit test with a bare word
match over the whole test file. Any mention counted — including prose in a `#`
comment.
Hit for real while adding platform/monitoring/tests/vpn-egress.test.yaml. Its header
explains that VPNTunnelDown stays quiet during an egress-steering failure, and that
sentence alone made this check declare VPNTunnelDown "asserted on" and demand its
removal from UNTESTED_ALERTS. Removing the entry would have recorded coverage that did
not exist — the precise "looks fixed" outcome this check exists to prevent, produced
by the check itself. (In that instance I made the coverage real instead, which also
exposed that the fixtures were missing tun0's receive counter entirely.)
Now an alert counts only when a test file names it in an assertion position, with
comment lines stripped first so a commented-out assertion does not count either.
BOTH promtool idioms are accepted, because both are real assertions:
alert_rule_test: - alertname: CNPGPrimariesCoLocated
promql_expr_test: - expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
Matching only the first is not a stricter check, it is a WRONG one: it reported
CNPGPrimariesCoLocated as untested when platform/cnpg/alerts.promtool-test.yaml pins
it across four scenarios entirely through the second form. Believing that would have
produced a redundant duplicate test for an alert already covered better than most —
so the first version of this very fix had to be corrected before it landed.
MEASURED BEFORE LANDING, because the registers are shrink-only and a prose-only alert
would have required an entry this check forbids adding: of 61 alerts, 32 are asserted
in one of the two idioms, 29 are not mentioned at all (already declared), and ZERO were
prose-only. So this tightening failed nothing on the day it landed.
My first measurement said 31/30/0 and was wrong in a way worth recording: it scanned
only tests/*.test.yaml and missed the <manifest>.promtool-test.yaml companion
convention, which is exactly what hid the CNPGPrimariesCoLocated false negative.
Verified in both directions: 134 assertions pass with the real coverage intact, and a
temporary prose-only mention of NodeExporterDown (a registered untested alert) leaves
it correctly listed as untested instead of triggering the stale-entry failure.
Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips,
0 unwired.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
a3f4ddb971 |
merge: feat/egress-gateway-staleness-alert — finished so it can actually pass the gate
Brings the two alerts for the silent egress-gateway failure: VPNEgressSteeringStale
(critical) and VPNEgressUplinkLabelMissing (warning), plus the kube-state-metrics
allowlist entry that makes them non-inert.
WHY THE BRANCH SAT IN THE BACKLOG: it could not pass the merge gate. Three
whole-repo checks failed on it, and each one was a real gap rather than a nuisance:
* alert-runbooks-resolvable — both alerts link to /runbooks/vpn-egress-stale,
which did not exist. An alert firing now would hand the operator a 404.
* alert-rules-unit-tested — alerts.yaml is already a covered file, so new rules
in it must be tested or a "tested" file quietly grows untested rules.
* alert-metrics-have-scrape-targets — see the guard bug below.
WHAT THE ALERTS CATCH. egress-class=vpn pods leave through one gateway node's tun0;
Cilium resolves it to an ifindex and caches it. An OpenVPN reconnect destroys and
re-creates tun0 with a NEW ifindex, and the node label
homelab.keithjackson.dev/egress-uplink=<addr>-<ifindex> is what forces Cilium to
re-resolve. If the label does not follow the tunnel, the cache points at a destroyed
interface and every vpn-class pod is silently dropped — while tun0 stays up and
VPNTunnelDown says nothing. That ran ~80 minutes undetected on the dev gateway.
Verified this is live on prod, not inert: k3s-prod-3 carries egress-gateway=true and
egress-uplink=10.150.192.86-139, and vpn-egress-steering has been in place 14 days.
The label was exported on 0 series before this change, so the allowlist entry is
what switches the alerts on.
runbooks/vpn-egress-stale.md — written and wired into the configMapGenerator, with
the four required operator sections. It leads with proving egress from a pod the
policy selects, because the symptom is "outbound traffic silently disappears", and
it says plainly not to "fix" this by deleting the policy: that does not restore
egress, it sends vpn-class traffic out of the real ISP connection, which is the leak
the class exists to prevent.
platform/monitoring/tests/vpn-egress.test.yaml — four promtool fixtures. 1 and 2 are
the SAME tunnel reconnect with and without the label following it, which isolates the
mechanism the rule actually keys on: a label VALUE change is not a changing sample,
it retires one series and starts another. 3 is the inert case (gateway node, no uplink
label) where the critical rule CANNOT fire, which is why the warning exists.
Two fixes the branch needed that were defects in the checks themselves:
1. alert-metrics-have-scrape-targets asserted the KSM allowlist matched
`nodes=[...egress-gateway]` with a literal closing bracket — i.e. that the list
contained EXACTLY that one label. Adding a second label correctly (egress-uplink,
which these alerts need) therefore FAILED the check while the property it guards
was perfectly satisfied. Loosened to `[,\]]`: in the list, not the whole list. A
guard that breaks when you correctly extend the thing it guards teaches people to
edit the guard, which is how it stops guarding.
2. UNTESTED_ALERTS listed VPNTunnelDown, and the check flagged it as stale because
"asserted on" is a bare word match over the test file — my header COMMENT
mentioning VPNTunnelDown was enough to count as coverage. Rather than delete the
register entry and claim coverage that did not exist, the fixtures now assert
VPNTunnelDown QUIET in all four cases. That required making them faithful: they
had no node_network_receive_bytes_total{device="tun0"} at all, so VPNTunnelDown
would have been firing throughout unnoticed, because promtool only checks the
alertnames you name. A healthy tun0 passes traffic, and that silence IS the claim
these rules exist for.
NOTE FOR LATER, not fixed here: that word-match means any alert named in prose counts
as tested. Tightening it to `alertname:` positions could reveal other prose-only
"coverage", and the register is shrink-only, so the cascade wants doing deliberately
rather than at 01:00 unattended.
Conflicts resolved: kept HEAD's Velero custom-resource-state block and took only the
branch's one-line allowlist change (14 lines down to 1); inserted the branch's
2026-09-12 DECISIONS-LOG entry in date order rather than appending it out of sequence.
Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips,
0 unwired. The new unit tests pass under the pinned promtool 3.14.0.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
755c08ab40 |
fix(conformance): the meta-check was layer-agnostic and reported failures silently
Declaring whole-repo scope on the L1 tests made whole-repo-invariants-wired (R-TEST-03) fail. It was right to change behaviour and wrong about why, and it could not tell anyone either way. Two independent defects. 1. LAYER-AGNOSTIC WHERE THE DESIGN IS LAYER-BOUNDED The "every scope-declared test is selected despite a requirement filter matching none" assertion compared EVERY scope-declaring dir against an `--layer l0` selection. That silently assumed only L0 tests would ever declare the scope — while ci/l1.sh was written from the opposite premise, that a whole-repo invariant could appear at any layer. So the first L1 declaration made the check blame the union for correctly excluding another layer. Now partitioned by layer, and the L1 half of the property is asserted too: every scope-declared L1 test must be selected by `--layer l1 --include-scope`, and that selection must not leak non-L1 tests. That is what makes the L1 declaration load-bearing rather than decorative — previously nothing proved the union reached L1 at all, because nothing at L1 declared the scope. 2. EVERY ASSERTION FAILED SILENTLY The idiom throughout was a bare test on one line and `chk $?` on the next, under `set -euo pipefail`. A false test aborts the script BEFORE chk can print, so the diagnostic — which is the entire value of this check — was never reached. The real failure above emitted 26 `ok` lines, rc=1, and no message at all; it read as a tooling glitch rather than a finding. All 12 assertion sites now capture the status (`chk $arc`) so the check reports and continues. Proven by forcing MIN_DECLARED=999 against the real check: the failure prints its full explanation and the run proceeds to the remaining assertions, where before it died mute. A meta-check that cannot say why it failed is worse than no meta-check. Verified: 35 assertions, rc=0; full offline l0,l1,meta 25 tests / 0 failures. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
b389566641 |
fix(conformance): L1 rendering never ran on a requirement-scoped gate
All three L1 tests now declare `conformance.home.k3s/scope: "whole-repo"`.
ci/l1.sh has passed --include-scope whole-repo since it was written, with a
comment saying no L1 test declared the scope yet and it was wired ahead of time
so the next whole-repo invariant "at any layer" would be picked up by annotation
alone. Nothing ever declared it. The consequence is that the entire L1 rendering
layer only ran when a touched path happened to attribute R-PLAT-02, R-NET-01 or
R-NET-05/06/07 — a fail-open hole for every other change in the repo.
The coupling requirement-scoped selection cannot see is that all three tests
exercise ONE code path: catalog/render.py against
schemas/servicedefinition.schema.json. Anything that changes rendered output
invalidates these goldens no matter which requirement the touched path maps to.
The 2026-09-04 external-dns merge is the proof, measured rather than asserted.
ci/touched-requirements.sh attributes platform/externaldns/helmrelease.yaml to
R-ENV-02, R-OBS-01/02/04, R-STATE-03 and R-TEST-01/02/03/04 — none of the
rendering requirements — yet that change added
external-dns.alpha.kubernetes.io/controller: none
to the rendered output of all three goldens. Via --print-selection:
requirement-scoped only (the 2026-09-04 behaviour): 0 L1 tests selected
with --include-scope whole-repo (how ci/l1.sh runs): 3 L1 tests selected
So the merge gated GREEN while leaving every rendering golden stale, and the
staleness surfaced only on a later full run.
Also corrects the now-false comment in ci/l1.sh that said no L1 test declares the
scope, and records why the annotation is load-bearing there.
Verified: offline l0,l1,meta --include-scope whole-repo in the pinned toolbox —
25 tests, 0 failures, 0 errors, exit 0.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
79c703bc79 |
fix(conformance): whole-repo checks audited nested agent worktrees as source
Six discovery sites walked ROOT without pruning dot-directories, so they descended into .claude/worktrees/agent-*/ — up to 18 full copies of the repo, created for isolated agent runs and gitignored (.gitignore:43). The visible half was false FAILs: four ServiceMonitors reported as selecting no Service, and nine PrometheusRule manifests reported as "under test but not in the repo", all of them worktree paths. The whole-repo L0 gate was red for reasons that had nothing to do with authored surface. The dangerous half is quieter. A check that asks "does anything in the repo declare X" can be satisfied by a worktree copy instead of real source, so the same bug that invents failures can also manufacture a pass. That is the reason to fix all six sites rather than only the ones that happened to be failing. egress-paths-selected-by-rules already had this fix and documents it (it once saw 5 CiliumEgressGatewayPolicy objects instead of the 1 authored); this applies the same dot-directory predicate everywhere else, which skips dot-DIRECTORIES while still matching dot-FILES such as .sops.yaml. platform/monitoring/tests/run.sh is the other side of the same failure: its `--list` reported each manifest once per worktree, and the L0 check compares that listing against the tree. Both find calls now prune .claude alongside .git. `--list` returns 21 real pairs where it previously returned them ~19x over. Verified in the pinned toolbox: L0 --include-scope whole-repo is 21 cases, 0 failures, 0 skips, 0 unwired (was 4 failures, then 1). L1 3/3. The 18 stale worktrees are all on already-merged branches and remain on disk; cleaning them up is a separate change and is no longer load-bearing for the gate. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
05881d5a6a |
fix(conformance): regenerate the three L1 goldens the external-dns change drifted
|
||
|
jakman85
|
b184a42729 |
merge: internal services stop advertising themselves in public DNS
Six exposure:internal services published A records into the public Cloudflare zone answering 192.168.2.20, disclosing the internal subnet, the Gateway VIP and a full service inventory, and forming a DNS-rebind primitive that is why the SOHO router strips these answers in the first place. The renderer now stamps external-dns.alpha.kubernetes.io/controller: none on every HTTPRoute it renders for an internal service, derived from the exposure field alone — so a service that omits exposure defaults to internal and is likewise unpublishable. Established against the deployed v0.21.0's own source, where the controller check runs before hostname resolution in both configured sources, then proven live on a throwaway hostname including that adding the annotation to an existing record triggers the DELETE. Deletes exactly 12 records: the A and TXT pair for jackett, pihole, qbittorrent, radarr, semaphore and sonarr. A 39-record recovery baseline is committed alongside. Safe to land before the DHCP change, contrary to my earlier caution: the router returns NOTHING for all six names today, so these records serve nobody on the LAN, while Pi-hole at 192.168.2.23 already answers for anyone pointed at it. The five exposure:public services are untouched. They remain unreachable publicly, which is a separate problem: the legacy edge at 136.51.115.252 returns HTTP 403 for our hostnames and serves a placeholder certificate, so nothing is reverse-proxying them today. |
||
|
jakman85andClaude Opus 5
|
4664b41db3 |
fix(l4): externalTrafficPolicy Cluster on unifi + crafty L4 VIPs
Cilium's L2 announcer elects one node per Service to answer ARP for its LoadBalancer VIP, and that election is NOT backend-aware. Combined with externalTrafficPolicy: Local -- which lets a node deliver only to a backend on itself -- an elected announcer holding no backend answers ARP for the address and then has nowhere to send the packet, so the client gets ICMP port unreachable. It never self-heals: the lease holder keeps renewing after the backend moves away. Measured on prod 2026-09-14: unifi's VM ran on k3s-prod-1 with both leases on k3s-prod-2, and crafty's pod ran on k3s-prod-1 with the -tcp lease on k3s-prod-2 while its identical -udp twin happened to draw k3s-prod-1 and worked. Confirmed causally on the announcing node with `cilium-dbg service list`: under Local the frontend 192.168.2.22:25565 is absent from that node's service table entirely; under Cluster it appears as `1 => 10.0.0.11:25565 (active)`; flipped back, it vanishes again. Durability confirmed by cordoning k3s-prod-1 and moving the crafty pod to k3s-prod-3, after which the announcing node re-pointed to the new backend with no intervention. Both files also carried a comment claiming Local "pins the VIP to the node running the pod", which is the precise thing that is false here; both are rewritten. apps/wave-0/pihole-dns.yaml is the reference implementation. The client-IP cost is stated per service rather than waved away. For unifi it is ZERO: the VM's KubeVirt `masquerade` binding already SNATs everything into the guest, so Local was never preserving a client address the controller could see. For crafty it is real and accepted -- player identity is a Mojang/Xbox UUID so /ban and the whitelist are unaffected, but /ban-ip would hit every player at once and must not be used; no IP ban or per-IP rate limit is configured today. No loadBalancerSourceRanges here, deliberately departing from pihole-dns.yaml: these Services never had a source restriction (their CNPs admit `world`), so Cluster removes no protection, and adding a LAN-only filter during an outage fix could cut off off-LAN players or a remote site's device informing. No CiliumNetworkPolicy change is needed, and that was verified rather than assumed: Cilium's `cluster` entity subsumes `remote-node`, confirmed by dumping the crafty endpoint's datapath policy map with the CNP unmodified (`reserved:remote-node` was already an Allow on the game ports). Unlike pihole, whose policy omits `cluster` and must name remote-node explicitly. Adds an L0 whole-repo guard for the class. It fails on the pre-fix tree with exactly these four Services named and passes after. whole-repo scope is necessary: ci/touched-requirements.sh has no arm for apps/* or platform/crafty/*, so the change most likely to reintroduce this would attribute no requirement and be gated by nothing. HONEST LIMIT -- this does NOT end the reported outage. All four ports also have no listening backend: UniFi OS was never installed in the VM (its README flags this as the open blocker) and /crafty/servers is empty, so no Minecraft server exists. Probed directly from the node, bypassing the VIP entirely, 10.0.0.74:8443/:8080/:11443 and 10.0.0.11:25565 all refuse, while the two controls answer. This commit fixes a real latent defect that would have blackholed those VIPs once the applications exist; it cannot make an absent application answer. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
4080065aa7 |
feat(dns): exposure: internal can no longer publish a public record
docs/09 R-DNS-03 and schemas/renderer-contract.md both said an `internal` service renders no public DNS record. Nothing enforced either sentence. ExternalDNS's `gateway-httproute` source admitted EVERY HTTPRoute on the shared Gateway, so six `exposure: internal` services — jackett, pihole, qbittorrent, radarr, semaphore, sonarr — published `<name>.home.keithjackson.dev` into the public authoritative Cloudflare zone answering with the Gateway VIP 192.168.2.20. That published the internal subnet, the Gateway VIP and a full service inventory to anyone who could guess a label, and made each of those names a DNS-rebind primitive — which is why the SOHO router strips RFC1918 answers and LAN clients need per-client dnsmasq overrides to reach their own services. THE MECHANISM, established against the DEPLOYED version rather than from memory. The live prod Deployment runs registry.k8s.io/external-dns/external-dns:v0.21.0. In v0.21.0, source/annotations/processors.go IsControllerMismatch() skips a source object whose `external-dns.alpha.kubernetes.io/controller` annotation is present and != its own `dns-controller`, and source/gateway.go applies that check inside Endpoints() BEFORE any hostname is resolved; source/service.go applies the identical check, covering the second configured source. catalog/render.py now emits that annotation on every HTTPRoute it renders for an `internal` service, derived from `exposure` alone — no per-service list — so a service that omits `exposure` inherits the internal default and is unpublishable too. Verified LIVE on prod on a throwaway hostname with no existing record at risk: the annotated route published nothing, and adding the annotation to a route that already had a record produced action=DELETE for both its A record and its `a-` TXT registry entry one ~60s reconcile later, moving nothing else in the zone. WHY PER-RESOURCE AND NOT A CLUSTER-WIDE OPT-IN FILTER. `labelFilter` is the stronger default posture and v0.21.0 supports it for both configured sources, but nine publishing objects here are hand-authored platform/apps manifests with no `exposure` field, seven of which hold live records (auth, crafty, crafty-mc, grafana, headlamp, the apex, unifi). An opt-in filter deletes all seven at once, and whether each should be publicly resolvable is an operator decision per name. Per-resource exclusion has exactly the blast radius of the defect; the seven are enumerated in a shrink-only register so the unadjudicated set cannot grow silently, and the filter stays available as hardening afterwards. `policy: sync`, `txtOwnerId` and `domainFilters` are UNTOUCHED. Per-resource exclusion only ever removes an object from this instance's view, so nothing here makes prod capable of deleting a record it does not own. WHAT THIS DELETES, and it is not reversible by a revert alone: 12 records — the A record and the `a-`-prefixed TXT registry entry for each of jackett, pihole, qbittorrent, radarr, semaphore and sonarr. The zone has no `*.home.keithjackson.dev` wildcard and `*.keithjackson.dev` does not match two labels, so each becomes a public NXDOMAIN with no fallback. The complete pre-state is captured at the authority and committed as a recovery baseline (.standup-dns/dns-20260914-pre-externaldns-scope.txt), with the restore procedure for both halves of a record. MERGE ORDER IS NOT OPTIONAL: feat/pihole-lan-dns-authority must land and be verified first. Removing these records while the LAN still depends on them, with the router stripping RFC1918 and no internal authority yet, breaks name resolution outright. The five `exposure: public` services are LEFT UNCHANGED and still answer with the Gateway VIP, so they remain publicly unreachable. That is two open operator decisions, both written up in platform/externaldns/helmrelease.yaml: what address they should answer with (136.51.115.252 is the legacy non-k3s stack serving a placeholder cert, not established as correct for this cluster), and how to separate them from the non-public names that share the Gateway — because in v0.21.0 the target annotation is read only from the GATEWAY and a ListenerSet, never from an HTTPRoute, so a per-service override is not expressible while they all attach to `shared`. No knob is added for it yet: one that cannot take effect invites someone to set it and expect one. Guard: conformance/tests/l0-static/internal-services-publish-no-public-dns, a whole-repo-scoped L0 static check (no ci/l0.sh stage — the whole-repo-invariants-wired meta-test forbids one). It asserts both directions (present on internal, absent on public), the `service`-source half, that the renderer DERIVES the annotation from `exposure` and cannot emit external-dns's own claiming value, proves the shipped renderer end-to-end on internal/public/omitted-field fixtures, and ratchets the non-catalog register. Proven to fail on the pre-fix tree (9 of 53 assertions, naming all six services) and pass after (56/56). Schema 1.6.0 -> 1.7.0: no field changes, a mapping change to `exposure`. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
3a72b07cda |
chore(conformance): shrink UNTESTED_ALERTS by the two PSI detectors
node-io-cause.test.yaml's healthy fixture asserts NodeIOPressureHigh and NodeIOPressureCritical quiet on the same measured prod state the eight new cause rules are pinned against — the point of that fixture being that the detectors and the cause rules must agree about what "healthy" is. The shrink-only register entries are therefore stale, and the gate correctly refused them. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
b9f3143f42 |
feat(monitoring): name the CAUSE of node I/O pressure, not just the stall
NodeIOPressureCritical has now fired twice on k3s-prod-1 for opposite causes and
said the same thing both times. On 2026-09-14 02:40-02:45Z it fired at PSI(full)
20.98% — twice the threshold, four times the 2026-09-07 level — while every local
device was idle (0.09-0.27 utilisation) and fast (1.4-4.8 ms per write), NFS
retransmissions were zero and no Longhorn volume was degraded. Establishing that
took eight ad-hoc PromQL queries after the event had already recovered.
Adds the group platform-health.node-io-cause: eight rules, one per cause class,
each naming its cause in its own summary.
NodeDiskDeviceSaturated device saturation >0.85 busy, 10m
NodeDiskWriteLatencyHigh device latency >20ms/write, 10m
NodeDiskWriteLatencyCritical device latency >50ms/write, 5m
NodeDiskReadLatencyHigh device latency >250ms/read, 15m
NodeNFSRetransmissionRateHigh NFS transport >1% retransmitted, 10m
NodeIOPressureNotLocalDisk not local disk PSI>10% with all of the above absent
NodeEtcdRequestsSlow etcd >0.5% of etcd requests over 1s
ContainerIOStallHigh which workload per-container PSI >15%
Every threshold is measured on prod over 24-48h, not guessed, and each rule
records its headroom. The PSI thresholds themselves are deliberately unchanged:
the 21% stall was real, it fired twice in 24h for two real events on the one node
that has a problem, and raising the bar would also have raised it above the 11.36%
at which prod-1 was already losing etcd on 2026-09-07.
Context reaches the operator at page time three ways: the summary states the cause
class; every rule carries a diagnostic_query annotation with the exact companion
PromQL; and node-pressure-dashboard.yaml gains a WHY row (seven panels) plotting
saturation, write/read service time, NFS retransmissions, our own NFS op rate,
the etcd slow-request fraction and per-container PSI side by side. runbooks/node-io.md
gains a 10-row triage table with the real healthy / 2026-09-14 / 2026-09-07 columns,
which turns those eight queries into a two-minute read.
etcd and NFS metric availability, verified on prod rather than assumed:
* etcd SERVER disk metrics do NOT exist. etcd_disk_wal_fsync_duration_seconds and
etcd_disk_backend_commit_duration_seconds are absent (kubeEtcd disabled for k3s);
a query for etcd_disk_*/etcd_server_*/etcd_mvcc_* returns nothing at all.
NodeEtcdRequestsSlow uses the apiserver's client-side view of its own local
member instead, and says in its own description that it is a proxy.
* node_nfs_* DOES exist — the nfs collector is on by default, host-wide via
/proc/net/rpc/nfs, confirmed on all three nodes. It gives op counts and
retransmissions and NO latency, so it can see a dropping export but not a slow
one. The mountstats collector is enabled here to close that gap; NO rule
references node_mountstats_* yet, because the series do not exist until the
HelmRelease rolls out and the collector's visibility through the node-exporter
mount namespace must be verified on prod first. The runbook records the check
and says to revert if it comes back empty.
Proof, per the repo's two shipped alerting failure modes:
* platform/monitoring/tests/node-io-cause.test.yaml — 7 fixtures, all from
measured profiles: healthy (all 8 quiet), 2026-09-14 (NotLocalDisk fires and
names sonarr), 2026-09-07 re-queried at offset 164h (write-latency and etcd
fire, NotLocalDisk correctly silent), rebuild storm, NFS transport, read
latency, and the `unless` semantics.
* 10 mutations of the shipped rules, all killed. The mutation run found a real
gap: the etcd threshold could be changed without failing any test, because it
was only pinned by a promql_expr_test that restated the threshold in the test
file. An alert_rule_test now pins it.
* All 11 referenced metrics verified to return series on prod with
`promtool query instant`, and all 8 expressions verified quiet on today's state.
Also extends the alert-metrics-have-scrape-targets owner table for container_*
(the kubelet ServiceMonitor's cAdvisor path, which this entry already claimed in
its title) and for etcd_request_duration_seconds_* by EXACT NAME, deliberately not
by an etcd_ prefix, so a future rule on the unscraped etcd_disk_* names still fails
the gate.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85
|
52398d4426 |
merge: Longhorn alerting measured the wrong things, three ways
(1) LonghornVolumeCapacityHigh alarmed on ALLOCATION while calling itself a capacity alarm. Prometheus's TSDB volume read 0.973 while the kubelet saw the same filesystem at 0.208, because 37.73 GiB of it is a Longhorn SYSTEM snapshot — the kind taken for the full replica rebuild that prod-1's drive replacement forced — growing +152 MB/h while the filesystem stayed flat. The rule was therefore unclearable by design, i.e. permanently firing. Deleted as redundant, shown rather than asserted: 23 of 24 volumes fall under the kubelet-based filling rules, and the 24th is volumeMode Block, which the deleted rule was not covering either. (2) LonghornVolumeDegraded and LonghornVolumeFaulted could not fire even with the scrape working. longhorn_volume_robustness on 1.12.1 is ONE-HOT over a state label with values 0/1, not the 0-3 enum the rules and their comment assumed, so a faulted volume would have alerted nobody. This survived the scrape fix landing hours earlier. (3) LonghornNodeStorageHigh was wrong twice: its 0.85 threshold sat above the 75% at which storage-minimal-available-percentage=25 already removes a disk from the scheduling pool, and longhorn_node_storage_usage_bytes SUMS a node's disks, so one full disk hid behind a half-empty sibling. Replaced with per-disk LonghornDiskAllocationHigh (>0.70) and LonghornDiskUnschedulable, the latter reading Longhorn's own verdict instead of re-deriving a formula from three settings no metric exports. Baseline green with 11/11 mutations red. retentionSize 45GiB of 50Gi and discard/TRIM are both REPORTED with recommendations, not changed: TRIM would not have helped here, since remove-snapshots-during-filesystem-trim is false and the 37.7 GiB is a snapshot. # Conflicts: # DECISIONS-LOG.md |
||
|
jakman85andClaude Opus 5
|
4687521d65 |
fix(monitoring): delete the Longhorn rule that measured allocation and called it capacity
LonghornVolumeCapacityHigh went pending within minutes of the Longhorn scrape
landing today. Re-measured on prod: it read 0.9731 on the Prometheus TSDB volume
(actual_size 52243681280 / capacity 53687091200) while the kubelet reported the
SAME filesystem 20.8% used. actual_size is head + pinned snapshots, and one
user_created=false system snapshot from prod-1's replica rebuild holds 40510484480
bytes (37.7 GiB) of it; Longhorn returns no freed blocks without discard/TRIM, and
retentionSize 45GiB on a 50Gi volume targets 90% by design. The ratio is pinned
near 1.0 and cannot fall back under 0.85 — a permanently-firing alert, the same
shape as CNPGNoRecoverabilityPoint.
DELETED, not re-thresholded. Raising the threshold would tune away a true
statement about allocation. Redundancy proven, not asserted: of 24 Longhorn
volumes on prod, 23 appear in kubelet_volume_stats_inodes > 0 and so fall inside
MonitoringVolumeFillingUp (namespace="monitoring") or PersistentVolumeFillingUp /
PersistentVolumeAlmostFull. The 24th is unifi/unifi-rootdisk, volumeMode Block —
no filesystem for anyone to measure, and the deleted rule was not covering it
either.
AND A SECOND INERTNESS the scrape did not fix. longhorn_volume_robustness on
longhorn 1.12.1 is one-hot over a `state` label with values 0/1, not the 0-3 enum
the rules assumed:
count(longhorn_volume_robustness) => 96 (24 volumes x 4 states)
count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
count(longhorn_volume_robustness == 2) => (empty)
count(longhorn_volume_robustness == 3) => (empty)
So a FAULTED volume would still have alerted nobody. Re-keyed to the label; the
"44 series / 72 unknown(0, detached)" note in platform/longhorn/helmrelease.yaml
was the same misreading and is corrected in place.
The allocation question moved to where allocation is a budget. LonghornNodeStorageHigh
fired at 0.85, but storage-minimal-available-percentage is 25 on this cluster, so a
disk leaves the scheduling pool at 75% used — the old threshold could only fire
after scheduling had already been refused. Replaced by:
LonghornDiskAllocationHigh (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70, 30m
LonghornDiskUnschedulable longhorn_disk_status{condition="schedulable"} == 0, 15m
Per DISK, not per node: longhorn_node_storage_usage_bytes sums a node's disks, so
one full disk could hide behind a half-empty sibling. Measured prod usage
0.2254 / 0.2306 / 0.2369 on prod-1/2/3.
PROVEN NEITHER PERMANENTLY FIRING NOR INERT. platform/monitoring/tests/longhorn.test.yaml
adds 6 cases pinning every rule quiet on today's measured prod state and firing on
a constructed bad state, with every `for:` asserted, plus regression assertions that
the old `== 2`/`== 3` forms match nothing and that per-disk 0.72 is visible where the
node-summed 0.475 is not. Mutation-checked 11 ways; each turns the suite RED. Live on
prod: all four new expressions return empty, while the sum of their selector
cardinalities is 54. The L0 UNTESTED_ALERTS register shrinks by seven (7/56 -> 14/58
alert rules unit-tested), which also brings MonitoringVolumeFillingUp,
PersistentVolumeFillingUp and PersistentVolumeAlmostFull under test since the
redundancy claim rests on them.
REPORTED, NOT CHANGED (DECISIONS-LOG, owner decisions): retentionSize 45GiB on a
50Gi volume is too tight — recommend 36GiB, with reasoning; and discard/TRIM on the
longhorn StorageClass is feasible but not advisable now — remove-snapshots-during-
filesystem-trim reads false on prod, so trim would not have reclaimed the 37.7 GiB
anyway, and continuous discard on nodes where the etcd WAL shares the spindle points
at this cluster's known failure mode.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
cf7d97e546 |
fix(obs): twelve app SLI rules read a signal that exists; kill the dead ServiceMonitors
Twelve `<app>-down` SLI rules were STRUCTURALLY INERT. Each read `up{job="<app>"} == 0`,
and the ServiceMonitor rendered beside it named `endpoints[].port: metrics` — a port NO
rendered app Service defined. Every monitor discovered ZERO targets, so no `up{job="<app>"}`
series existed and the rule could not fire. Verified on prod before this change:
`count by (job) (up)` listed 24 jobs and not one of the twelve appeared; `up{job="plex"}`
and `up{job="gitea"}` both returned nothing. `PrometheusTargetsMissing` could not see it
either — a target that was never discovered has no `up` series to be 0.
THE PORT NAME WAS NEVER THE MISSING PIECE. Ground truth per app: Plex, Pi-hole, Jackett,
qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud export NO Prometheus metrics
without a separate exporter sidecar; Gitea can but only with app.ini metrics enabled;
Keycloak does, on the operator's management port, already scraped by
platform/keycloak/servicemonitor.yaml; vm-demo is a KubeVirt VM with no Service at all, so
its monitor selected nothing whatsoever. `metrics: true` was asserting a fact none of them
had. There was also a THIRD layer of inertness nobody had hit yet: every app namespace ships
default-deny with `ingress: []` and the renderer emitted no monitoring allow, so even a
correct port would have been dropped by policy — while renderer-contract.md claimed the
baseline carried such an allow.
FIXED IN THE GENERATOR, not in twelve rendered files:
* `slis: [up]` now renders a `<app>-down` alert against kube-state-metrics readiness of the
service's own workload object —
`kube_deployment_status_replicas_available{namespace,deployment} == 0 or absent(...)`,
`kube_statefulset_status_replicas_ready` for a StatefulSet, and
`absent(kubevirt_vmi_info{namespace,name,phase="running"})` for a VM. The same idiom
KeycloakDown has used since it was fixed. kube-state-metrics IS scraped, so this needs no
exporter and no new target. The `or absent(...)` half is load-bearing: the signal
DISAPPEARING now fires the alert instead of silencing it.
* schema 1.6.0 adds `observability.metricsEndpoint` — WHERE metrics are served. A
ServiceMonitor is rendered ONLY from it, and it renders all three halves of a working
scrape together: the named Service port, the monitor endpoint, and the
monitoring-namespace CiliumNetworkPolicy allow. No catalog entry declares one, so twelve
dead ServiceMonitors are gone. `metrics: true` keeps meaning logs-to-Loki plus
kube-state-metrics liveness.
* schema 1.6.0 also adds `observability.livenessWorkload`, and catalog/keycloak.yaml uses
it. Keycloak's server is a StatefulSet run by the operator; the default Deployment shape
would have made its `absent()` half fire FOREVER — the other failure mode this repo has
shipped.
* renderer selftest gains cases 9/9a/9b/9c: no metricsEndpoint => no monitor; the up SLI
must not read `up{job=...}`; endpoint port must exist on the Service; the metrics port
must NOT reach the Gateway allow; an unknown livenessWorkload.kind is REFUSED.
PROOF, both directions, for all twelve (apps/wave-{0,1,2}/tests/app-slis.test.yaml, run by
platform/monitoring/tests/run.sh over the RENDERED manifests):
* quiet on the healthy state; firing on no-ready-replica; firing when the series is ABSENT;
the VM asserted both gone and in a non-running phase; per-app isolation asserted, and
keycloak asserted un-maskable by its neighbours (keycloak-operator Deployment,
keycloak-db StatefulSet ready in the same namespace).
* mutation-checked: six mutations, each turning its suite red — reverting plex to
`up{job=...} == 0`; dropping seerr's `absent()` half; pointing keycloak at a Deployment;
capitalising the VMI phase to `Running`; widening keycloak's matcher to the namespace;
inverting plex's `== 0` to `!= 0`. Control run green.
* live on prod with `promtool query instant`: all ten deployed app Deployments export
`kube_deployment_status_replicas_available`,
`kube_statefulset_status_replicas_ready{namespace="keycloak",statefulset="keycloak"}` = 2,
`kubevirt_vmi_info` present with a lowercase `phase="running"`; and evaluating the twelve
SHIPPED expressions returns no series for eleven (quiet, not permanently firing). The
twelfth is vm-demo, which is not in apps/wave-0/kustomization.yaml and so is not deployed
— its rule is not applied either.
DELETION WAS NOT THE ANSWER, and the evidence is recorded rather than assumed:
kube-prometheus-stack's `kubernetes-apps` group IS live on prod (17 rules, KubePodNotReady /
KubeDeploymentReplicasMismatch among them), so app pods are not unmonitored — but that is a
generic warning keyed to a namespace, not the per-service CRITICAL SLI R-OBS-01 declares.
platform/monitoring/alerts.yaml carries no generic app-readiness rule at all.
GUARD SO IT CANNOT RECUR SILENTLY: new whole-repo L0 invariant
conformance/tests/l0-static/servicemonitor-ports-resolve. It walks the MONITORS, not the
alerts, so it needs no `up{job=...}` rule to reach one, and resolves each endpoint by SELECTOR
across the whole tree rather than by same-file adjacency. Fixing the alerts DELETED the only
thing that made the broken monitors visible to the sibling check, which is exactly why this
axis needed its own home. It declares `conformance.home.k3s/scope: whole-repo` and adds no
ci/l0.sh stage. Proven to FAIL on the pre-fix tree: 15 failures — eleven port disagreements,
vm-demo's monitor matching no Service at all, and the three generator/contract assertions —
and to pass after. Its resolver is additionally proven on in-check fixtures (agreeing pair
accepted, disagreeing pair rejected, target-less endpoint rejected), because after the fix
there are zero in-repo monitors left for it to judge and a guard whose only evidence is
"nothing failed" cannot be told apart from one that stopped working. It also asserts
r_servicemonitor does not HARDCODE a port name — the mechanism by which one edit gave twelve
services the same wrong one.
Also fixed, found on the way:
* platform/monitoring/tests/run.sh discovered the SUITE convention only inside
platform/monitoring/tests/ while documenting it as general — a suite added anywhere else
was never run and still looked like coverage. Now discovered tree-wide; a misplaced
*.test.yaml fails loudly instead of vanishing.
* KNOWN_UNWIRED_JOBS in alert-metrics-have-scrape-targets is now EMPTY, and the twelve
apps/* entries in alert-rules-unit-tested's UNTESTED_FILES are gone — both because the
gaps were CLOSED, not waived. Both registers are shrink-only and a stale entry fails.
* renderer-contract.md claimed the default-deny baseline carried a standing monitoring
scrape allow. It never did; the claim is corrected and the allow is now real, per declared
metrics endpoint and scoped to that one port.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
a76ae3bdfa |
fix(ci): the content scan could not match R-C2-01/02 — a digit in the workstream segment
Checked as part of closing the C2 coverage gaps, and it was live on main. The content
scan's pattern was `R-[A-Z]+-[0-9]+`, which CANNOT match an ID whose workstream segment
carries a digit: `[A-Z]+` consumes the `C` of `R-C2-01` and then demands a `-` where the
`2` is. `R-C2-01` and `R-C2-02` are the only two such IDs in
conformance/requirements/registry.yaml, and the scan was blind to BOTH.
REPRODUCED before the fix, on a throwaway repo whose only change is a manifest declaring
both IDs on real (non-comment) lines, in a path the map keys to nothing:
$ REPO_ROOT=$T BASE_REF=main ci/touched-requirements.sh
R-NET-06 # the all-letter control ID was the ONLY output
It failed by UNDER-attributing, which is the fail-OPEN direction: a change that retyped a
`conformance.home.k3s/R-C2-01` tag, or a manifest self-declaring R-C2-02, contributed
those IDs to NOBODY, so the C2 tests were filtered out of the merge gate. Same defect
class as the map's `*) : ;;` arm — a miss that reads as "nothing touched".
Every other consumer in the repo already allowed the digit, which is why this was the lone
dissenter: test_requirements_for_path() twenty lines above in this same script,
conformance/lib/coverage.py, conformance/lib/status.py and conformance/lib/
run_l1_offline.py all use `R-[A-Z0-9]+-[0-9]+`. Aligned to that form, with the reason
recorded at the line.
SCOPE OF THE HOLE. path_requirements() has no arm for platform/homepage/* or
platform/headlamp/* either, so before this the ONLY route to R-C2-01/02 was detection rule
3 (editing the C2 conformance test itself). No path-map arm is added here: c2-root-surface
declares `scope: whole-repo`, so ci/l0.sh Stage 3b's --include-scope union selects it on
EVERY gate invocation regardless of the touched set — which is the right mechanism for an
invariant whose input surface is the whole tree, and strictly stronger than a two-component
path arm would be.
REGRESSION-TESTED, in ci/tests/touched-requirements.test.sh (the suite `make -C ci test`
runs): a fixture declaring R-C2-01 / R-C2-02 / R-OBS-04 from `notes/` — a path outside
path_requirements(), so the IDs can only come from the content scan — asserts both digit
IDs ARE attributed, that the all-letter form still is (control), and that a purely numeric
workstream is not invented (the pattern must not widen the gate on noise).
PROVEN TO DETECT BREAKAGE: with the pattern reverted to `R-[A-Z]+-[0-9]+` the suite reports
`FAIL (2 failed, 14 passed)` naming both C2 IDs as missing; restored, all three ci/ suites
pass (16 assertions in this one).
ALSO: conformance/tests/l0-static/whole-repo-invariants-wired/check.sh cites this pattern
in the comment explaining why its BOGUS_REQ fixture is lowercase. Citation updated so the
reasoning still reads true; the fixture is unaffected (grep -E is case-sensitive, and the
check still passes 26/26).
NOTE: another branch may be editing this same script concurrently. This is a one-line
change to one statement plus its comment — expect a trivial conflict and keep the
`[A-Z0-9]` form.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
f57c411953 |
test(conformance): L0 covers R-C2-01 / R-C2-02 — 49 assertions, and it is RED
The Test DECLARATION landed in the preceding WIP commit with no check.sh, so the test
could not run at all. This is the check: 49 offline assertions over the repository as
git tracks it, no cluster, no network, no spec repo (R-TEST-06).
THE RULE. Every home-zone hostname the repository routes over HTTP must be linked from
the apex dashboard's configuration; every home-zone link the dashboard offers must
resolve to something the repository actually serves (an HTTPRoute at L7 or a
hostname-publishing LoadBalancer Service at L4); and the two R-C2-02 surfaces must be
linked, reconciled and backed by real alert rules, with operational rights arriving
per-user through OIDC groups rather than through a shared cluster-admin identity.
IT IS RED ON MAIN, on the forward direction:
FAIL (8) every service the repo ROUTES over HTTP is surfaced on the C2 root.
Unsurfaced: crafty.${BASE_DOMAIN} (platform/crafty)
platform/crafty/httproute.yaml serves crafty.${BASE_DOMAIN} through the shared Gateway,
clusters/base/crafty.yaml reconciles it, and nothing on the C2 root links to it. R-C2-01
says the root MUST surface "the services running in the cluster and links to them", so
an unsurfaced route is a requirement gap, not a styling preference. Fixed in the next
commit, deliberately separate so the RED is in the history.
WHAT IS ASSERTED BEYOND THE PRIOR ATTEMPT (49c160d on feat/coverage-debt-paydown, whose
tree is ~107 commits behind and was NOT merged):
(0) THE INPUT SURFACE ITSELF. The forward direction is only as good as the set of
manifests it reads, so three assertions guard the guard: the scanned roots are
present, every manifest in them PARSES (an unparseable file is a service the check
cannot see), and EVERY Flux-reconciled path lies inside a scanned root — so a
future tier reconciling, say, ./services fails loudly instead of letting the
dashboard hide all of it. conformance/ is deliberately excluded from the scan: its
fixtures and goldens declare synthetic HTTPRoutes (paas-demo, gw-declared-
conformance) that are test data, not services the cluster serves. The prior
attempt excluded them only by accident, through a filename prefilter.
(5) postBuild.substituteFrom ON THE DASHBOARD'S TIER. Hostnames are ${BASE_DOMAIN}
tokens now, not the literals the prior attempt compared. Without substitution the
apex route binds the literal host `${BASE_DOMAIN}` (matching no listener) and
every tile links to a host that does not exist — while Flux still reports the
Kustomization Ready. It fails OPEN, which is this repo's most-repeated incident
shape.
(12) TOKENS, NEVER LITERALS. No href may name a literal `home.keithjackson.dev` host,
so prod's front door cannot link dev's services — the 2026-09-04 class (two
clusters authored to describe the same names, prod's external-dns then deleting
dev's records), applied to the C2.
(2) THE SECTIONNAME IS PROVEN LOAD-BEARING rather than asserted from prose: the
wildcard listeners are shown to exist AND to be distinct from the apex listener,
which is why `*.${BASE_DOMAIN}` cannot serve the bare apex.
(10) THE REVERSE DIRECTION NOW ACCEPTS L4. The UniFi tile points at :11443 on a
hostname published by apps/vms/unifi/50-service-l4.yaml (a KubeVirt VM behind a
pinned Cilium LoadBalancer VIP). That is something this repo serves, so it
resolves — no off-cluster declaration is needed for it any more. The declaration
hatch is kept and constrained (specific home-zone hostnames, may not shadow a
hostname the repo does serve) but is EMPTY today, which is the honest state.
WHOLE-REPO SCOPE, as the declaration says. The invariant quantifies over every tracked
manifest that can declare an HTTPRoute; the change that BREAKS it is a route added
somewhere else entirely (`apps/wave-1/newthing.yaml` attributes R-GITOPS-02 or nothing,
never R-C2-01), and ci/touched-requirements.sh has no path-map arm for platform/homepage
or platform/headlamp at all. Requirement-scoped selection would filter this check out of
exactly the change that introduces the bug — the R-TEST-03 blind spot. No ci/l0.sh stage
is added: Stage 3b's --include-scope union selects it, and whole-repo-invariants-wired
fails the gate if anyone hand-adds one.
HONEST GAPS, in the check's own header: it does not fetch the apex (L2, covered for the
Gateway/TLS half under R-DNS-01 / R-TLS-01); (19) reads a DECLARED auth posture because
no Gateway ext-authz attachment exists anywhere in this repo, and says so in the
assertion text; it does not judge whether a tile is useful; and an L4-ONLY endpoint
(crafty-mc, a Minecraft game port) is reported as an ADVISORY rather than required,
because a raw TCP/UDP endpoint has no URL a browser can follow.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
0fe12b8332 |
WIP: L0 test declaration for R-C2-01 / R-C2-02 (no check.sh yet)
INCOMPLETE — preserved by the PM after the authoring agent died to a server error mid-task (its third attempt; the first two died to a session rate limit). Do not merge. R-C2-01 and R-C2-02 are the only 2 remaining coverage gaps on main (49/51). This is the chainsaw Test declaration only: it quotes both requirements verbatim from the spec, cites docs/12-command-center-ui.md for which surface serves which half (Homepage at the root for R-C2-01, Headlamp for R-C2-02's 'controlled operational actions where safe', Grafana for dashboards/alerts), and argues why the deciding half is L0 rather than L2 — reachability is already covered by l2-behavioral/gateway-ingress-reachable, whereas whether the root surfaces THE SERVICES THE CLUSTER RUNS is a comparison of two sets that are both repository facts. WHAT REMAINS: check.sh does not exist, so this test cannot run at all. Also still open: whether 'a service the root hid' (from the prior art commit 49c160d on feat/coverage-debt-paydown) is still a live defect, whether ci/touched-requirements.sh can attribute R-C2-01/02 at all (a digit in the workstream segment reportedly broke the content scan), and the R-TLS-02 decision summary. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
37eea51f24 |
merge: the substitution check now catches a collapsed doubled dollar
The existing check audited only the inverse — a ${...} undefined or
unescaped — so it passed the manifest that broke prod: Postgres
dollar-quoting under a postBuild.substituteFrom path, which envsubst halves,
delivering 'DO $' and a syntax error on every reconciled run (fixed
|
||
|
jakman85andClaude Opus 5
|
830983b791 |
test(l0): prove the dollar-collapse guard fires — and name the real envsubst
The implementation landed in the preceding WIP commit had never been executed. This finishes it: the arithmetic is re-measured, the guard and its allowlist are attacked, and the header's one factual error is corrected. WHAT ENVSUBST ACTUALLY IS. The header credited `github.com/drone/envsubst`. The pinned flux 2.4.0 binary in the dev image embeds `github.com/fluxcd/pkg/[email protected]` — Flux's own fork of drone/envsubst ("This package is a fork of drone/envsubst", its README) — read straight out of the binary with `grep -ao`. Same behaviour, different module; the header now says so and says how to check, because a claim about a dependency that nobody can re-derive is how the last wrong assumption survived. RE-MEASURED, NOT ASSERTED. Every row of the DOLLAR-RUN ARITHMETIC table was re-run through `flux envsubst` (flux 2.4.0, DEFINED=VALUE) and every row holds: `$$`->`$`, `$$$`->`$$`, `$${D}`->`${D}`, `$$${D}`->`$VALUE` (odd run: the brace is LIVE), `$$$${D}`->`$${D}`, `DO $$ x $$;`->`DO $ x $;`, and `$$${ARR[0]}` dies with "missing closing brace" while `$${ARR[0]}` is fine. `--strict` output was byte-identical to non-strict, exit 0 both ways (`cmp`) — strict mode is blind to this whole class, which is why a source assertion is the only guard available. The header now carries the 30-second reproduction, flagged to be re-measured on any FLUX_VERSION bump. END TO END ON THE REAL PATH. With the pre-16b7fcd manifest restored, `kubectl kustomize apps/wave-0` emits `DO $$` — valid SQL, and exactly what the `kubectl apply` verification saw — while the same stream through `flux envsubst --strict` emits `DO $` and `$;` and exits 0. That two-command contrast IS the incident. THE GUARD IS NOT TRIVIALLY SATISFIABLE — 16 adversarial cases, all as intended, run against a throwaway copy of the tree so nothing could be left behind: * an unmarked `$$` under a substituted path FAILS (DOLLAR-COLLAPSE); so do an odd `$$${...}` (DOLLAR-HALF-ESCAPED) and an even `$$$${...}` (OVER-ESCAPED); * a stale entry FAILS three ways — context no longer in the file, context that only covers an already-accepted `$${` escape and so excuses nothing, and a path that is not a file in the repo; * the allowlist cannot blanket-exempt a path: a 2-char context is rejected (<24), a `.*` wildcard is matched VERBATIM and therefore rejected as stale, a 24-char context carrying no dollar run is rejected, a 4-char reason is rejected (<40) — and, the central case, a genuine entry covers only the span its context matches, so a SECOND `$$` elsewhere in the same file still fails. Two genuine entries for two real occurrences do pass, so the mechanism is usable rather than merely unsatisfiable; * the CLASS, not the instance: the pre-fix `DO $$ ... $$;` reconstructed in a NEW ConfigMap under apps/wave-2 fails at both dollar runs, and restoring the real pre-16b7fcd apps/wave-0/nextcloud-db.yaml verbatim fails the check at lines 384 and 391. The guard rejects the exact commit that broke production. The scratch fixtures lived only in the throwaway copy; `git status` on this branch stayed clean throughout, which is the other half of the last attempt's unfinished business. LEGITIMATE USES EXIST AND STAY GREEN. 18 `$${...}` escapes (unifi cloud-init, the gitea and nextcloud OIDC bootstrap scripts, two longhorn comments) pass unremarked, and the 2 non-braced `$$` occurrences are the two allowlist entries. Those two are shell comments inside block scalars, and `kubectl kustomize` confirms the comment text really does survive into the stream envsubst reads — so they are genuine, reaching content whose collapse is cosmetic, not a hypothetical the allowlist was invented to excuse. STILL NOT PROVEN, AND STILL NOT PROXIED: that a change was verified through a substituting path. Which command a human ran is a fact about a shell session; git stores the manifest, never the invocation, so it is not statically decidable here and no honest formulation exists. Grepping commit prose for `kubectl apply` would be satisfiable by rewording and would fire on the break-glass runbooks whose whole content is that command. The answer is to move the proof off the verification path and onto the source, which is what assertion (7) does. No ci/l0.sh stage: the `conformance.home.k3s/scope: "whole-repo"` annotation is what wires it, and whole-repo-invariants-wired fails the gate on a hand-written stage. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
a92fa11701 |
feat(monitoring): scrape Longhorn + cert-manager — 8 alerts were structurally inert
An alert rule whose metric is never scraped is not a broken alert, it is an ABSENT
alert that reads as a present one. `longhorn_volume_robustness == 2` over a TSDB
with no longhorn_* series returns no vector: permanently quiet, permanently
"healthy", indistinguishable from a correct rule. It survives review (the PromQL is
right), it survives `promtool test rules` (the unit test supplies the series
itself), and it appears on no dashboard.
VERIFIED ABSENT ON PROD before this change (re-verified 2026-09-13 21:0xZ, ~15h
after the first audit, because a stale reading is not evidence):
count(longhorn_volume_robustness) => (empty) 4 rules inert
count({__name__=~"certmanager.*"}) => (empty) 3 rules inert
count(hubble_drop_total) / {__name__=~"cilium.*"} => (empty) 1 rule inert
count by (job) (up) => 20 jobs, none of them
Longhorn, cert-manager,
Cilium or Hubble
Consequence: storage health was unmonitored on the cluster whose prod-1 had just
had a failing SSD, and TLS expiry was unmonitored with the live `wildcard-home`
wildcard 89 days from expiry on an ACME DNS-01 renewal that can fail silently.
WHAT THIS CHANGES
* platform/longhorn: `metrics.serviceMonitor.enabled: true` (the supported chart
way — rendering longhorn 1.12.1 emits exactly 1 ServiceMonitor with the flag and
0 without), plus networkpolicy-metrics.yaml. The NetworkPolicy is LOAD-BEARING,
not belt-and-braces: measured with the monitor applied and no allowance, all 3
targets were DISCOVERED and every scrape failed `context deadline exceeded`
after 10.0s — a drop, not a refusal — while curling the endpoint from inside
the manager pod returned 200 in 0.031s. Longhorn 1.12.1 ships 6 NetworkPolicies
of its own, guarded on `networkPolicies.restrictInternalTraffic` (default TRUE)
and NOT on `networkPolicies.enabled`, so they are effectively unconditional and
the manager policy admits only same-namespace Longhorn components. The one-line
alternative (restrictInternalTraffic: false) was rejected: it deletes the
internal-traffic restriction across all six components to open one port to one
namespace, trading an R-SEC-02 control for an observability one when the two do
not conflict. This policy is Ingress-only, ns monitoring -> app=longhorn-manager,
TCP 9500 only; 9502/9503 (webhook, recovery backend) stay closed.
* platform/cert-manager: `prometheus.servicemonitor.enabled: true`. Chart default
already serves 9402; only the monitor was missing. No NetworkPolicy needed —
neither ns cert-manager nor ns monitoring carries one. NOT paired with
prometheus.podmonitor.enabled: the chart calls `fail` if both are set.
* platform/kubevirt: KubeVirtVMLaunchFailing queried
`kubevirt_vmi_phase_count{phase="Failed"}`, which DOES NOT EXIST on the deployed
KubeVirt — a different flavour of the same defect, with the scrape target up and
healthy. Prod has only `kubevirt_vmi_info` and `kubevirt_vmi_number_of_outdated`;
the only phase-bearing names are the phase_transition_time histograms. Corrected
to `kubevirt_vmi_info{phase="failed"}` — LOWERCASE, because KubeVirt lowercases
the value (observed `phase="running"` for the unifi VMI) and a capital-F matcher
would select nothing and be exactly as inert.
PROVEN, NOT ASSERTED. The rendered ServiceMonitors were applied to prod by hand and
the metrics observed to land, then REMOVED again so the merge's Helm upgrade creates
and owns them rather than colliding with an unowned object:
count(up{job="longhorn-backend"}) => 3 (all three managers, up==1)
count({__name__=~"longhorn.*"}) => 8717 (was empty)
count(longhorn_volume_robustness) => 44
up{namespace="cert-manager"} => 3 targets, all 1
count({__name__=~"certmanager.*"}) => 104 (was empty)
wildcard-home days to expiry => 89.2, ready_status{condition="True"}=1
All 58 alert rules in the repo were then evaluated against prod one by one: every
one returns a result without error, and exactly ONE fires.
TRIAGE OF THE ONE NEWLY-FIRING ALERT — A TRUE POSITIVE, NOT TUNED AWAY.
LonghornVolumeCapacityHigh on the Prometheus TSDB volume:
`longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes` = 0.971
(48.6 GiB allocated of 50 GiB) while the kubelet's view of the same filesystem is
only 20.4% used. Longhorn is not reclaiming freed blocks, so ~48.6 GiB x 3 replicas
is committed for ~10 GiB of real data. On a cluster whose failure mode is disk
pressure and whose etcd WAL shares a spindle with Longhorn, that is worth knowing
and was invisible until now. The threshold is left alone; the follow-up is
filesystem trim, not a bigger number.
THE GUARD (conformance/tests/l0-static/alert-metrics-have-scrape-targets)
Whole-repo L0 invariant, 118 assertions. It crosses from each alert to the scrape
config that makes its metric exist: every metric name is claimed by exactly one
declared owner, and each owner's enabling declaration is re-proven against the
working tree on every gate run, including the negative conditions that defeat an
enablement without deleting it. `up{job="X"}` is audited on its own axis, because
`up` always exists: where this repo authors a Service and a ServiceMonitor in the
same file, the monitor's endpoint port must be a port that Service defines.
It DECLARES conformance.home.k3s/scope: "whole-repo" and adds NO ci/l0.sh stage —
the whole-repo-invariants-wired meta-test fails the gate if one reappears.
PROVEN ABLE TO FAIL, five ways, each reverted:
1. an alert on a fabricated metric -> FAIL "claimed by NO entry"
2. longhorn metrics.serviceMonitor removed -> FAIL "no longer matches"
3. networkpolicy-metrics.yaml unwired -> FAIL "the monitor alone is NOT enough"
4. cert-manager podmonitor also enabled -> FAIL "FORBIDDEN pattern"
5. a register entry made stale / removed -> FAIL in both directions
TWO SHRINK-ONLY REGISTERS, because a blanket fail would make the gate red until
work that is not this change lands, and a disabled gate protects nothing:
* KNOWN_UNWIRED_METRICS (1): hubble_drop_total. See below.
* KNOWN_UNWIRED_JOBS (12): every apps/* `<app>-down` SLI rule is ALSO inert, and
this was found by the re-audit, not previously known. The rendered
ServiceMonitors name `port: metrics` and NOT ONE of the twelve app Services
defines a port of that name (zero occurrences of `name: metrics` across all
twelve manifests); prod confirms none of the twelve jobs appears in `up`. This
is the nastiest variant of the class: `up` exists in abundance so nothing looks
wrong, and PrometheusTargetsMissing cannot see it either, because a target that
was never discovered has no up series to be 0. Closing it needs an exporter per
app (Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr speak no
Prometheus) — except keycloak, where the live jobs are keycloak-service /
keycloak-discovery and the rule is a rename away from being live.
NOT DONE HERE, DELIBERATELY: Hubble. hubble.metrics.enabled RECONFIGURES THE CNI —
it opens 9965 on every cilium-agent and rolls the DaemonSet across all three nodes,
serialised by this repo's own maxUnavailable: 1 (set after the 2026-09-01
k8sServiceHost outage, where a CNI roll deadlocked 2 of 3 agents). It also needs a
matching edit to roles/cilium/templates/cilium-values.yaml.j2 in the same commit,
because platform/cilium/helmrelease.yaml carries a [parity] contract against it and
a rebuilt node would otherwise bootstrap a CNI config that disagrees with the
reconciled one. Recommended as its own change on the cilium Kustomization, with the
Ansible parity edit, a narrow labelsContext, and a one-node canary — not smuggled in
here.
ALSO FOUND, NOT FIXED HERE: kube-state-metrics is OOMKilled every ~10 minutes on
prod (90 restarts in 16h against a 128Mi limit) while its logs show the
custom-resource-state factory reloading every 3 seconds. Thirteen alert rules read
kube_* metrics, so those rules are INTERMITTENTLY inert — the same defect class,
arriving through a crashloop rather than a missing monitor. Needs its own change.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
377fa205ea |
WIP: catch envsubst collapsing a doubled dollar in substituted paths
INCOMPLETE — preserved by the PM after the authoring agent hit the session
rate limit mid-task. Do not merge without finishing and gating it.
Closes the inverse of what this check already covered. It asserted that every
${...} in a substituted path is defined or escaped; it did NOT catch content
containing a literal doubled dollar that the author needs delivered verbatim
and that envsubst silently halves. That shipped a broken CronJob (fixed in
|
||
|
jakman85
|
5fa893095b |
merge: the placeholder guard test was hiding a RED, not a pass
It had been permanently SKIPPED because it expected bootstrap/standup.sh, which never landed on main — planned, never merged, still living only on the unmerged prod-standup branches along with the 2026-09-05 incident it came from. Dropping the real script in and running the old check produced FAIL 2/34: it would have false-blocked the very merge that landed its own input. Now split into the half that needs no guard (no overlay ships a placeholder-shaped effective cluster-vars value; an empty value must be STATED by the overlay, not inherited in silence; a loud base sentinel must be explicitly overridden) and the half that self-activates. The guard is discovered by CONTENT — any tracked *.sh defining <name>_is_placeholder() — so landing one anywhere activates the second half with no edit here, and two definitions is a failure. The check never exits 3 again. Unscoped L0 goes 13 PASS / 1 SKIP -> 14 PASS / 0 SKIP. |
||
|
jakman85
|
e49b86eb41 |
merge: 35 runbooks, served, with a guard that they resolve
All 46 runbook_url annotations pointed at https://<apex>/runbooks/<name> and every one returned 404 — there was no runbooks/ directory and no route. Every alert that reached a human handed them a dead link at the moment they needed it. Runbooks are now served from their own non-gating tier at the apex prefix, PUBLIC on purpose: gating them behind Keycloak would make keycloak-down unreadable exactly when Keycloak is down, which is a loop with no exit. The new L0 guard checks both directions — a runbook_url with no file, and a file not wired into the generator — and it caught a live one on contact with main: CNPGPrimariesCoLocated had shipped hours earlier pointing at a runbook that did not exist. |
||
|
jakman85andClaude Opus 5
|
87c3d83dff |
fix(conformance): placeholder-guard-coupling actually RUNS — it was skipping forever
`conformance/tests/l0-static/placeholder-guard-coupling` (R-ENV-02, R-STATE-03) exited 3 SKIPPED on every gate run since it landed, because its check keyed its whole existence on `bootstrap/standup.sh`. Measured, not assumed: that file has NEVER been on `main` — there is no deletion commit for it anywhere; it landed only on unmerged prod-standup branches (`work/prod-standup-campaign`, `prod/*`, `feat/prod-hardening`). It was planned and never merged, not removed. SKIPPED is not green, and a permanently-skipped incident-derived test is the quiet version of the failure it was written to prevent. The skip was also hiding a RED. Run against the real `standup.sh` from `work/prod-standup-campaign`, the old check FAILED 2 of 34 assertions — false positives from treating the empty string as placeholder-shaped for knobs cluster-vars authors empty on purpose (`BACKUP_CREDENTIALS_SUFFIX`, `VELERO_BACKUP_PREFIX`, both `""` on dev and both documented there as load-bearing). It would have false-blocked the merge landing its own input. NOT deleted — the hazard is not gone. Split into two halves, one of which needs no guard: HALF A (always runs): no environment overlay ships a placeholder-shaped EFFECTIVE cluster-vars value; an empty effective value is STATED by the overlay rather than inherited in silence; a LOUD base sentinel is explicitly overridden by every overlay. This is the 2026-09-05 outcome asserted at the source instead of at the guard — if prod's S3_BACKUP_ENDPOINT is never a placeholder, velero cannot be pointed at a hostname that does not exist. Input surface is `clusters/`, all of it on `main`. HALF B (self-activating): the real predicate extracted and executed against those same values, the consumer trace, and the mid-string anchoring property. The check NEVER exits 3 again, and the guard is DISCOVERED by content rather than hardcoded: any tracked `*.sh` defining `<name>_is_placeholder()` is found, so a guard landing at `bootstrap/standup.sh` — or moved into a sourceable library, the remedy the check already recommends — activates HALF B with no edit to the test. Two files defining it is a FAIL. When no guard exists the gap is printed loudly every run instead of swallowed as a skip. LOUD vs EMPTY is the correction that makes it honest in both modes: a loud placeholder is always a defect in an overlay, while `""` is held to the explicitness rule instead — and still counts as a placeholder for a knob a guard covers, because a `""` case-arm is the guard author's declaration that empty is not a usable target. Proof, all reverted: 84 assertions and exit 0 on `main`; prod S3_BACKUP_ENDPOINT set to the literal `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` FAILS (A1); deleting dev's explicit VELERO_BACKUP_PREFIX op so it inherits `""` FAILS (A2); a LOUD base BULK_NFS_SERVER with prod's override deleted FAILS 2/86 (A1+A3); and with the real `standup.sh` present, reverting `s3_is_placeholder` to the `TBD-*` PREFIX glob FAILS 12/110 (B3 anchoring). With the pristine `standup.sh` the check PASSES at 110 assertions where the old one failed 2 of 34. L0 layer: 14 checks, 14 PASS, 0 SKIP, 0 FAIL (was 13 PASS / 1 SKIP). Coverage unchanged — R-ENV-02 and R-STATE-03 both still list this test, 49/51 with the same 2 gaps. Also: * `ci/touched-requirements.sh` — `bootstrap/*` -> R-STATE-03 R-GITOPS-01 is KEPT, its stale comment corrected. It claimed standup.sh "carries the placeholder guards", describing a file that is not here; R-STATE-03 there is ANTICIPATORY. Deleting the entry would leave the change that lands a placeholder guard selected by nothing. * `ci/tests/touched-requirements.test.sh` — new path-map section (12 assertions, was 8) locking `bootstrap/*` and `clusters/*`, so removing either entry is a test failure rather than a silent loss of selection. All three ci/tests pass (12 + 31 + 15). * `docs/BULK-NFS-STORAGE.md` §7 — its "nothing in the gate currently refuses a placeholder-shaped overlay value" is no longer true; corrected. * Scope re-audited: still deliberately requirement-scoped, not whole-repo. Every path that can break it attributes a requirement it declares. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
78d0f1b0f6 |
feat(ci): promtool in the pinned toolchain, one alert-rule harness, and the gaps declared
The CNPG alert unit tests (324 lines, mutation-checked) were run by NOTHING. promtool was
not in the baked toolchain, no gate stage evaluated any *.promtool-test.yaml, and the only
runner `curl | tar -xz`'d a promtool release from GitHub at TEST-RUN time if one was
missing. Hours later a second, independent harness landed for the Velero rules
(platform/monitoring/tests/) with its own fallback — a `podman run` of
quay.io/prometheus/prometheus. So the repo had two harnesses, two acquisition paths, and
coverage that looked present and was enforced nowhere: the exact failure mode that had just
produced a permanently-firing CNPG alert and a structurally inert Velero alert.
PROMTOOL IS NOW PINNED LIKE EVERY OTHER TOOL. PROMTOOL_VERSION=3.14.0 in
infra/devcontainer/versions.env and ci/versions.env (the two stay in lockstep, per the
repo's rule), baked by the Containerfile from the prometheus release tarball (promtool ships
inside it), recorded in /etc/dev-toolchain-versions, passed by dev.sh, and installed for the
l0 stage by .gitea/workflows/scripts/install-tools.sh so the CI runner is not permanently
SKIPPED. The pin is not free-floating: promtool IS the rule engine, so it tracks the
Prometheus prod actually runs — verified against the live cluster, not assumed:
kubectl --context k3s-prod get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'
-> v3.14.0-distroless (image quay.io/prometheus/prometheus:v3.14.0-distroless)
ONE HARNESS. platform/cnpg/run-alert-tests.sh is folded into
platform/monitoring/tests/run.sh, which now covers every PrometheusRule in the repo that has
tests, via two conventions it discovers automatically: `<manifest>.promtool-test.yaml`
beside its manifest (cnpg), and `<component>/tests/*.test.yaml` over that component's
manifests (monitoring). It writes the flattened rules under whatever name each test file's
own `rule_files:` declares, so no test file needs to know the harness exists, and it
`promtool check rules` first — a rules file that does not parse would make every unit test
pass vacuously by finding no rules at all. The cnpg script's reasoning is preserved: why a
CR must be flattened (`.spec.groups`, never re-typed), and why generated files go to a temp
dir. Its `${BASE_DOMAIN}` substitution is dropped in favour of extract_rules.py's verbatim
placeholders, with the trade-off recorded there — the hazard it guarded against is addressed
by the temp dir, and verbatim leaves one transformation fewer between the shipped rule and
the rule under test.
NO ENGINE IS ACQUIRED AT RUN TIME any more — not the GitHub release, not the container
image. Not because a pinned download is unusual here (the Containerfile downloads every tool
it bakes) but because those two decided the version, the network and the machine SILENTLY,
at test time, on whatever host ran the tests; and each quietly turned "the toolchain has no
promtool" into a green run, which is how the toolchain stayed without one. Missing promtool
is exit 3 (UNWIRED, never green); a PROMTOOL= that does not resolve is exit 2.
ONE WIRING: conformance/tests/l0-static/alert-rules-unit-tested, NOT a gate stage. The
`prom-rules` stage (ci/prom-rules.sh + its Makefile target + ci/gate.sh entry) is deleted:
it is the bespoke-per-check pattern
|
||
|
jakman85andClaude Opus 5
|
b7341b329b |
fix(runbooks): declare whole-repo scope, drop the bespoke l0 stage, +cnpg-primary-spread
Three corrections after rebasing onto main. 1. THE GUARD CAUGHT A REAL ONE ON ITS FIRST CONTACT WITH main. main had gained `CNPGPrimariesCoLocated` (platform/cnpg/alerts.yaml) pointing at /runbooks/cnpg-primary-spread, with no runbook. The check failed the rebase exactly as designed — 46 annotations, 35 names, 34 present — naming both the missing file AND the missing configMapGenerator entry. runbooks/cnpg-primary-spread.md now exists: what co-located primaries actually cost here (blast radius, not an outage, because instances: 3 + required anti-affinity), why the drift is uncorrectable-by-declaration (CNPG has no primary-placement field), the readiness gate that keeps the rule quiet during a real node outage, and the ~9s-per-database switchover rebalance, one cluster at a time. The 2026-09-07 outage in which all three primaries sat on one node is the reason the alert exists at all. 2. WHOLE-REPO SCOPE, NOT A HAND-WRITTEN STAGE. main's refactor replaced ci/l0.sh's per-check "UNCONDITIONAL" stages with a declaration: `conformance.home.k3s/scope: "whole-repo"`, unioned into every gate run by Stage 3b's `--include-scope`. My +38-line Stage 3f is removed (ci/l0.sh is now byte-identical to main) and the annotation is declared instead. Assertion (7) of the check was rewritten to match: it no longer asserts its own path appears in ci/l0.sh — it asserts the test DECLARES the scope and that ci/l0.sh still passes --include-scope, since neither half alone makes the check load-bearing. That matters here because ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02", not R-OBS-04, and maps platform/kubevirt/* and runbooks/* to nothing at all. 3. A FACTUAL CORRECTION IN ALL 35 RUNBOOKS. Every page claimed `dev.sh` defaults its current-context to k3s-dev. main's `9b4744d fix(devcontainer): dev.sh defaults to prod` made that false, and a runbook that misstates which cluster a command talks to is worse than one that says nothing. The pages now say what is durably true: the default has already changed once, so naming `--context k3s-prod` is what makes a command mean the same thing next month, and the per-invocation banner is there to be read. The check's own failure message was corrected the same way. Re-verified after all three: check green at 374 assertions (46 annotations, 35/35 runbooks present, 35/35 projected), re-applied to prod, and re-probed in-cluster through the Gateway — node-down, keycloak-down, keycloak-client-secrets, cnpg-archiving, longhorn-faulted, vpn-down, node-io and cnpg-primary-spread all HTTP 200 with text/plain; charset=utf-8 and 8.9-12.1 KB of real body; /runbooks/ index 200; an unknown name 404s with a pointer rather than a blank page; and homepage still answers the apex `/`, confirming Gateway API longest-path precedence. Full merge gate: PASS (spec-immutability, lint, l0, render-golden, l1 all green; prom-rules SKIPPED for want of promtool in the image; L2/L3 SKIPPED, no cluster). Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
44983095e1 |
feat(runbooks): make the 42 alert runbook_url annotations resolve
Every platform-health alert rule carries `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations over 34 distinct names, across platform/monitoring/alerts.yaml, platform/monitoring/pvc-capacity-alerts.yaml, platform/cnpg/alerts.yaml and platform/kubevirt/alerts.yaml. There was no runbooks/ directory and no HTTPRoute serving that path: an in-cluster probe of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404. Every alert that reached a human handed them a dead link at the exact moment the documentation is read, and the annotation made it look solved — the same green-surface-over-absent-mechanism class as the import Job that reported success while storing placeholder secrets. No alerts.yaml is touched and no runbook_url value changes. The existing URL scheme is made to resolve instead. WHAT LANDS - runbooks/<34>.md — one per name, each grounded in this cluster's real topology and, where one exists, its real incident history: the 2026-09-07 k3s-prod-1 etcd I/O starvation (measured PSI 11.36 vs 0.75/1.29, the exact journal lines, the four mechanisms proven inert), the Longhorn reboot tax and the freeze/thaw of concurrent-replica-rebuild-per-node-limit, Cilium's once-resolved egress interface cache (Ifindex 3 against a live 1780) and the absence of OSS failover, the CNPG 38-hour ContinuousArchiving=False with its measured 6.8 MiB/h disk-full clock and the 2026-09-12 aggregation-scope fix, the 2026-09-06 Keycloak lockout in which all 12 clients held literal placeholder secrets behind an entirely green cluster, the Velero R2 x-amz-tagging NotImplemented incompatibility and its v1.14.3-rc.2 pin, and swap re-enabling itself across reboots via systemd-gpt-auto-generator. Each page states what the alert means HERE, the first diagnostic commands, likely causes ordered by probability with a remediation each, and when to escalate or stop. - Every page uses FULLY-QUALIFIED resource names and says why: three CRDs register the kind `Backup` here, the bare short name resolves to Longhorn's (zero objects), so `kubectl get backup -A` prints "No resources found" on a cluster holding 6 Velero and 27 CNPG backups. Acting on that false negative is how a CLI quirk becomes data loss. - Every page states that kubectl only works via `infra/devcontainer/dev.sh run bash -c '...'` with an explicit `--context k3s-prod`, because dev.sh defaults its current-context to k3s-dev. - Pages also record which alert groups are INERT today (measured 2026-09-13: no Longhorn, cert-manager, Flux-controller or Hubble scrape job exists, so 13 rules evaluate against nothing) so nobody reads their silence as health. SERVING — runbooks/serving/ plus clusters/base/runbooks.yaml A two-replica nginx-unprivileged serving a ConfigMap generated from runbooks/*.md, at the apex `/runbooks` prefix, in its own non-gating post-platform Flux tier (dependsOn platform-config, wait:false) — the platform/homepage pattern, file for file. - The service catalog cannot express this route: ingressRoutes[].host is pattern-bound to a single DNS label plus the apex, so the bare apex is unrepresentable, and there is no path field. - Rejected: redirecting to the Gitea web UI. A runbook is read when the platform is broken, and Gitea is an app-tier workload whose own dependencies several of these runbooks document; Gateway API's RequestRedirect also has no "append .md" modifier, so it would need 34 hand-maintained ReplaceFullPath rules. - configMapGenerator maps each file to an EXTENSIONLESS key (node-down=node-down.md), so /runbooks/node-down is a plain static lookup with no rewrite rule. Served as text/plain: the namespace has DNS-only egress, so a CDN markdown renderer could not load anyway. - The tier declares NO postBuild.substituteFrom, deliberately: envsubst would run over the prose and silently empty every ${...} shell example (it fails open — the 2026-09-01 emptied Cilium k8sServiceHost). The HTTPRoute omits `hostnames:` and inherits the https-apex listener's, so nothing needs substituting. AUTH — PUBLIC, deliberately auth-mode=none. Gating these pages behind oauth2-proxy/Keycloak would make runbooks/keycloak-down unreadable when Keycloak is down and runbooks/keycloak-client-secrets unreadable during the exact 2026-09-06 failure mode. An operator who cannot log in is this route's primary reader. Content is operational procedure only, no secrets. The rationale is recorded in an annotation on the HTTPRoute so the route is left OUT of the Gateway ext-authz attachment when forward-auth enforcement lands. PROVEN — in-cluster probe through the Gateway on prod /runbooks/node-down HTTP 200 8613 bytes text/plain; charset=utf-8 /runbooks/keycloak-down HTTP 200 9378 bytes text/plain; charset=utf-8 /runbooks/cnpg-archiving HTTP 200 11729 bytes text/plain; charset=utf-8 /runbooks/longhorn-faulted HTTP 200 9135 bytes text/plain; charset=utf-8 /runbooks/vpn-down HTTP 200 11092 bytes text/plain; charset=utf-8 /runbooks/backup-failure HTTP 200 9604 bytes text/plain; charset=utf-8 HTTPRoute Accepted by io.cilium/gateway-controller on sectionName https-apex; the `/runbooks` prefix wins over homepage's apex `/` by Gateway API longest-path precedence. GUARD — conformance/tests/l0-static/alert-runbooks-resolvable (R-OBS-04) 363 offline assertions. It checks BOTH halves, because the regression is silent in both directions: an alert with a new runbook_url and no runbook links to a 404, AND a runbook committed without its configMapGenerator entry is in git, reviewed, merged, and still 404s in the cluster. Also asserts a quality floor so a stub cannot make the link look fixed, and that route/listener/nginx prefix agree with the annotations. Proven to fail, then reverted: a bogus runbook_url (2 FAIL), a runbook removed from the generator (1 FAIL), and a 31-byte stub (7 FAIL). Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
5fa278f61c |
fix(ci): declare whole-repo scope instead of hand-writing a gate stage per invariant
R-TEST-03 runs the conformance subset a change TOUCHES. That has one blind spot, and it
is the worst-shaped kind: an invariant that quantifies over the whole tracked tree — or
over subtrees ci/touched-requirements.sh's coarse path map keys to NOTHING (apps/, infra/,
most of platform/) — cannot be selected BY the requirements a change touches, because the
change most likely to break it attributes no requirement at all. Requirement-scoped
selection then filters the check out of exactly that change.
The repo's answer had been a hand-written "UNCONDITIONAL" stage per such check in
ci/l0.sh: 3c flux-postbuild-substitution, 3d keycloak-client-secret-reconciler, 3e
bulk-tier-distinct-from-replicated. Three near-identical 12-line rationales, three
hardcoded check.sh paths, three copies of the same rc 0/3/* case block — and worst, a
mechanism that failed SILENTLY, because whoever added the NEXT whole-repo invariant had
to REMEMBER to write a stage, and if they forgot the check ran only when some unrelated
requirement happened to be touched.
A test now DECLARES the scope of its invariant and the runner enforces it:
metadata:
annotations:
conformance.home.k3s/scope: "whole-repo"
* conformance/lib/common.sh conf_dirs_for_scope(s) resolves the annotation. Carries the
general rationale, once, instead of per stage.
* conformance/run.sh --include-scope SCOPE unions declaring tests into select_dirs,
still bounded by --layer. New --print-selection resolves the
selection and exits, running nothing.
* lib/run_l1_offline.py --unwired-scope: a SKIP of a declaring test exits 3, not 0.
--summary-out: one step-summary line per executed check.
* ci/l0.sh Stage 3b gains --include-scope whole-repo and an rc-3 arm.
Stages 3c/3d/3e DELETED (-150 lines).
* ci/l1.sh, ci/conformance.sh pass --include-scope too, so the next whole-repo invariant
at ANY layer needs only the annotation.
rc 3 SEMANTICS PRESERVED EXACTLY. A whole-repo check reporting "nothing to audit" is
reporting that the tree it audits has VANISHED. The deleted stages called that "not a
pass" and set the gate's rc=3; the runner now returns 3 and ci/l0.sh maps it to rc=3.
A missing check file still degrades to not-green, never to silence.
THE REASONING WAS RELOCATED, NOT DELETED. Each stage's incident narrative moved into the
header of the test it describes, beside its scope annotation: the Cilium k8sServiceHost
empty-substitution CNI outage (2026-09-01), 12 unusable Keycloak clients behind a green
cluster (2026-09-06), and a multi-TB bulk volume landing on the 3-replica tier.
AUDITED ALL 13 EXISTING l0-static TESTS, individually. 10 declare whole-repo scope; 3 are
DELIBERATELY left requirement-scoped and say so in their own headers:
* placeholder-guard-coupling — inputs are bootstrap/ + clusters/, both already mapped to
a requirement it declares. Scoping cannot hide a breaking change.
* dev-nodes-provisioned-as-code — inputs are infra/ and nothing else.
* spec-immutability-enforced — inputs are ci/ and the read-only spec repo.
The latter two were nonetheless under-gated because the PATH MAP was incomplete, which is
the right place to fix a path-scoped invariant: infra/* -> R-ENV-01 added, and R-TEST-06
added to .gitea/*|ci/* (it is no longer an uncovered ID — spec-immutability-enforced
carries the label, so attributing it no longer false-blocks the coverage gate).
NEW conformance/tests/l0-static/whole-repo-invariants-wired (R-TEST-03) keeps the
declarative mechanism honest for ALL declaring tests at once, replacing the three tests'
individual "ci/l0.sh greps my own check.sh path" self-assertions. It proves the union
EMPIRICALLY: with a requirement filter matching no test the L0 selection is empty, and
with --include-scope it is every declaring test; a scope-declared synthetic SKIP exits 3
while an undeclared one exits 0; and a ci/*.sh that hardcodes a check.sh path must belong
to a test that also declares the scope, so the bespoke pattern cannot creep back.
TASK 2 — a changed test attributes the requirements it CERTIFIES.
ci/touched-requirements.sh gains detection rule 3: a change anywhere under
conformance/tests/<test>/ attributes that test's own conformance.home.k3s/R-* labels, read
from its chainsaw-test.yaml. The content scan could not close this — it reads only
added/removed lines, so it sees a change to the LABEL BLOCK but not to the test BODY,
which is exactly the case that matters. Labels are read from HEAD or, failing that, from
the BASE, so DELETING a test still attributes what it used to certify; otherwise removing
the only verifier of a requirement is the cheapest way to make the gate stop checking it.
Whole-line `#` comments are dropped first and *.md is untouched, honouring the existing
"reference, not declaration" rules.
Both changes are fail-closed: every one of them can only ADD tests or requirements to a
run, never remove one.
TESTS: new ci/tests/touched-requirements-test-attribution.test.sh (15 assertions) covers
rule 3 — body-only edit, nested-fixture walk-up, test deletion, header-comment R-IDs not
attributed, non-test paths unaffected, and both new path-map entries. The two existing
suites still pass (39 assertions). `make -C ci test` runs all three.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
6202d92c27 |
test(conformance): gate the URL desired state, and prove the selftest can fail
L0 (212 assertions, was 175):
* runs ci/render-keycloak-client-urls.py --check, so client-urls.yaml must be exactly
what the realm import renders. A hand edit that makes the two disagree is
unmergeable -- this is what keeps the realm import the single source of truth for
hostnames rather than just the place they were first typed.
* asserts client-urls.yaml and the realm import are rendered by the SAME Flux
Kustomization. postBuild.substituteFrom is per-Kustomization, so a split would let
one ${BASE_DOMAIN} resolve to two different values and the realm would converge on
a hostname the HTTPRoutes do not serve.
* asserts the desired state uses no substitution token the realm definition lacks,
names only clients the import declares, and declares no field outside the
five-field write surface; that the CronJob mounts it readOnly at the path the
script defaults to; and that the script, renderer and check agree on which fields
are writable.
* NEW mutations.py: injects 12 plausible regressions into the EXTRACTED script and
requires --selftest to catch every one. A passing suite proves nothing if it cannot
fail, and "reported success while the realm was broken" is precisely what this
component exists to end -- so the same standard applies to its own test. ~20s.
L2: the audit Job now mounts keycloak-client-urls and asserts
KEYCLOAK_CLIENT_URLS_AUDIT_OK. Without that it would pass on a realm still holding
pre-migration redirect URIs -- the exact state prod was measured in.
Alerts: the existing Failing/Stale pair is EXTENDED to name redirect-URI drift, not
duplicated. One CronJob gets one pair; a second pair on the same job_name regex would
fire twice for one cause. No new critical, no new permanently-firing rule.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
23ea72d01e |
feat(keycloak): reconcile client redirect URIs, not just client secrets
The realm import is a one-shot that never re-runs, so a redirect URI changed in git reaches the realm never. Prod's stored URIs were measured still on the pre-migration hostname while git declared the new one -- after a BASE_DOMAIN flip every OIDC login fails with a redirect_uri mismatch, with the CR still reading Done=True and nothing alerting. Same failure class as the literal-placeholder secrets, same cure. EXTENDS the existing reconciler rather than adding a sibling. Decisive reason: a URL write needs NO new endpoint. It is the same minimal client PUT, and the current URLs already come back in the allow-listed client lookup -- so the allow-list is widened by nothing and the users/groups/roles/authentication guarantee is preserved unchanged rather than re-declared in a second copy. The ConfigMap also carries exactly one script key by L0 contract, so a sibling would have had to fork ~300 lines of safety-critical code that the header explicitly forbids forking. Reconciles redirectUris, webOrigins, rootUrl, baseUrl, adminUrl -- only the fields a client actually declares, and on a write only the fields that actually drifted. A field the desired state omits is UNMANAGED and left untouched; absence never means force-to-empty. THE TWO PASSES ARE INDEPENDENT. Each reads its own desired state and fails closed for its own fields only. A malformed URL ConfigMap cannot stop client-SECRET repair, which would otherwise make credential healing depend on a hostname file. List fields compare order-insensitively (sorted, not set -- duplicates are still drift), because Keycloak does not preserve the authored order of redirectUris and an order-sensitive compare would rewrite every client forever while reporting success. --selftest grows to 84 assertions covering the domain flip, a client gaining a URI, a client losing a stale one, a public client's URLs being managed while its secret is not, per-pass fail-closed, order-insensitivity, and a server that accepts a PUT without applying it. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
fa4deac877 |
fix(conformance): the dirty-tree assertion must judge HONOURED evidence
One historical `-dirty` row reddened this gate permanently, even after a clean re-run of the same test. status.py already resolves duplicates latest-wins (latest_by_test), so a superseded row is not evidence of anything — but this check iterated every row, so the poisoned row from the git_sha bug (fixed in the parent commit) would have failed the audit forever. The dirty-tree assertion is now scoped to rows status.py would actually honour, by importing latest_by_test rather than reimplementing the rule, so the two cannot disagree about which rows are live claims. Structural assertions — provenance keys, sentinel, test-still-exists — still apply to EVERY row. Nothing is erased: the row stays in the ledger and runs/20260913T014437Z-19fc8bc/ run.log stays committed. It simply stops being read as a live claim. Verified the guard is intact afterwards — the check still REFUSES both forgeries it is built to catch (a self-skip presented as a pass, and a proven row from the root commit). A dirty-then-clean sequence is also the normal shape of noticing a dirty tree, fixing it, and re-running; that should not permanently fail an audit. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
c66a7e29e9 |
fix(conformance): the harness contradicted itself in two places
1. auditability-proofs-producible rejected any proven ledger row without a sentinel, with no layer exemption — while record_run.py DELIBERATELY records L0/L1 rows sentinel-less via its own SENTINEL_EXEMPT_LAYERS, because run_l1_offline.py renders skips as JUnit <skipped/> and those layers have no pass token to attribute. So ANY `--record` run touching L0 or L1 wrote a row this gate then rejected. Consequence: a requirement whose verifier set includes an L1 check could never be both PROVEN and L0-green — R-NET-06 is exactly such a requirement, and hit it the moment it was re-proven on prod. The check now honours the same constant instead of restating the policy, so the two halves cannot drift apart again. 2. git_sha() marked the tree dirty from tracked files, but conformance/runs/ ledger.jsonl IS tracked — so the FIRST `--record` run of a session dirtied the tree and every LATER run was stamped `-dirty`, which check.sh then refuses to honour. Recording one test poisoned the next, and that cost a second live fault-injection run against prod to work around. Its docstring already stated the intent for the untracked runs/<id>/ half of the same artifact; the ledger is the tracked half. Now excluded via pathspec, mirroring status.py's IMPL_EXCLUDE_PREFIXES which already lists this path for the same reason. Neither change weakens an assertion: (1) applies an exemption that already existed and was already documented, (2) stops the recorder invalidating its own output. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
04606d48b2 |
docs(vpn): stop crediting the nftables kill-switch for a drop it never makes
Four places in the tree told the reader that the host nftables table is the authoritative fail-closed backstop for egress-class=vpn pods: platform/vpn-egress README, egress-steering.yaml, the role README, and the vpn-killswitch test's own GIVEN/THEN prose. It is not, and cannot be — Cilium's eBPF egress-gateway datapath bpf_redirect()s those skbs to the egress device without invoking netfilter POSTROUTING. Corrected with the measurement in each place rather than softened, because a guard the repo trusts and that does not fire is worse than no guard. What replaces the claim, everywhere: vpn-class fails closed because CILIUM drops the traffic when it cannot use its cached egress interface, verified by tcpdump on both the WAN and tun0 during a tunnel stop and a tunnel restart (0 packets out either). The nft table keeps a narrower, now-proven job: node-originated traffic sourced from a tunnel address (killswitch_drop moved 0 -> 3 packets on k3s-prod-3 when such a packet was routed at the WAN). Also recorded: * egress-steering.yaml — `interface: tun0` is resolved ONCE and cached, and the gateway node is what tells Cilium it changed (the egress-uplink node label). A future reader's instinct will be to have the node write to that object; the comment says not to, because it is Flux-managed, and points at the label and `cilium-dbg bpf egress list` as the debugging entry points instead. * egress-steering.yaml — why the new NFS media mounts are not affected by pod-scoped steering: an NFS PV is mounted by the node kernel in the host netns, so podSelector never matches it. The 192.168.0.0/16 exclusion covers the server too, but that is the second reason, not the first. * vpn-killswitch test — its recovery step depends on the re-steer path, and could not have passed before it existed. Its no-leak assertion was always right; only the explanation was wrong. Also states the prerequisite plainly: the test cannot run at all without a labelled gateway node. * platform README — prod is live on k3s-prod-3 with the before/after egress IPs, and the note that prod's vpn-class pods had NO internet beforehand (CEGP with no matching node -> `Gateway IP: Not Found` -> dropped), so arming the gateway restored egress rather than moving it off the WAN. * role README — the standing "add the shared fwmark to the interface contract" escalation is closed as not implementable on Cilium OSS, rather than left open for someone to attempt. DECISIONS-LOG carries the full session entry, including the drafted ADR-0015 / docs-03 amendment for the architect (specs are immutable to agents) and the five things left for the operator — chief among them that a silent gateway outage still has no alert, which is what let the dev gateway sit dead for 80 minutes. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
da8a0eac04 |
test(l0): R-STATE-04 gets a real check — bulk storage provably is not the replicated tier
The gate BLOCKED this branch, correctly, and the block was the interesting part.
Attaching the media library selects R-STATE-04 as a touched requirement ("bulk media storage has
a capacity strategy distinct from the replicated tier"), and R-STATE-04 had NO scaffolded test, so
the coverage REQUIRE stage failed both l0 and l1. There were two ways out. One was to keep
avoiding the token — the existing discipline in catalog/plex.yaml and catalog/qbittorrent.yaml,
which say in as many words that naming the requirement would block the merge. That made sense while
the tier was a stub. It does not make sense in the change that DELIVERS the tier: it would mean
shipping the requirement and claiming nothing. So: the check.
WHAT IT ASSERTS, and why each one is a real failure mode rather than a shape
The `bulk` StorageClass sat as `server: REPLACE-ME` for six roadmap phases, and the only thing
keeping a multi-terabyte library off the 3-replica `longhorn` class was four hand-written comments.
That is the class of failure being closed: a bulk volume on the replicated tier does not fail
loudly — Longhorn accepts the PVC and starts replicating 3x of data three mini-PCs cannot hold.
(1) DISTINCTNESS — no bulk PV/PVC on any class the repo declares with the Longhorn provisioner
(derived from platform/longhorn/, so adding a Longhorn class cannot create a blind spot),
and the class is explicitly EMPTY.
(2) CAPACITY STRATEGY as arithmetic — a bulk volume at least an order of magnitude above the
same service's replicated volume. The requirement is about two magnitudes of data; a "bulk"
volume sized like a config PVC means the tier was named, not applied. The 10x floor is
CALIBRATED against the real corpus, not guessed: the media library is ~460x its config
volume and the Nextcloud archive ~13x its 20Gi, so a higher floor would fire on a legitimate
volume and teach the next reader to weaken the check. My first draft used 100x and did
exactly that — the corpus caught it.
(3) PRE-BOUND FROM BOTH ENDS (claimRef + volumeName, agreeing). Guards the most deceptive
failure available in this tier: a dynamic bulk class hands out a FRESH EMPTY directory, the
pod mounts, the PVC Binds, everything is green, and the 8 TiB was never attached.
(4) Retain, always. (5) mountOptions carry `hard` and exactly one nfsvers=4.x, and never `soft`
— a soft mount hands a server hiccup to the app as a short read, which for a media library
is silent corruption; and v4 is what keeps the firewall surface at one port.
(6) ONE EXPORT, AT MOST ONE MOUNT PER WORKLOAD. This is the assertion I most wanted. Splitting
one export into two subPath mounts looks TIGHTER and breaks link() with EXDEV; Sonarr and
Radarr then degrade to copy-then-delete, rewriting every import over NFS and destroying the
seeding copy, surfacing weeks later as "imports are slow and my ratio is gone". No reviewer
sees that in a diff.
(7) READ-ONLY COUPLING — nfs.readOnly, the volumeMount flag and ReadOnlyMany must agree, so
dropping one still leaves the others; and every bulk mode is a `...Many`.
(8) The server is a BARE cluster-vars token, that knob has a base default, and EVERY overlay
supplies it — the seam that lets one authored definition serve two clusters that reach this
host by different addresses.
(9) LOAD-BEARING and NON-VACUOUS. Wired unconditionally into ci/l0.sh as Stage 3e for the same
reason as 3c/3d: Stage 3b scopes to touched requirements, and the path map keys NEITHER
apps/* NOR platform/longhorn/* to this requirement — so the change most likely to break
this could attribute nothing and be gated by nothing. An EMPTY bulk corpus FAILS rather
than passing trivially, and a floor on the assertion count keeps a pass from degenerating.
It reads the COMMITTED apps/ goldens rather than calling the renderer. Asking the renderer would
only prove it agrees with itself; the goldens are what Flux reconciles, and render==golden is
already its own gate stage.
PERTURBED, because a check that has never failed is not evidence. Each of these was applied to the
real tree and each fired with the intended message: bulk PVC moved onto `longhorn`; Retain -> Delete;
`hard` -> `soft`; the token replaced by the literal 192.168.1.9; sonarr's single mount split into
two subPath mounts (assertion 6); and the prod overlay's knob deleted.
Also hardens render.py: `size` is structurally required of a bulk entry (a PV with no capacity is
rejected by the API server) but the JSON Schema only enforces `export`, so omitting it raised a bare
KeyError three frames down. It now raises a sentence naming the service, the volume and the remedy.
Found by the same perturbation pass.
Coverage: 48/51 -> 49/51 (96.1%), 2 gaps, 0 safety-critical.
BASE_REF=main make -C ci gate -> MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85
|
89cf59ffb7 | docs(conformance): the GIVEN/WHEN/THEN header states the exception it now allows | ||
|
jakman85
|
730d12ddc4 |
fix(conformance): R-ENV-03 shared-path rule tests the harm, not the filename
The blanket regex around RENDERED_DIRS failed any tracked file whose name
carries a delimited `prod` token. That caught the four per-environment backup
credential halves (BACKUP_CREDENTIALS_SUFFIX) even though BOTH halves are
applied to BOTH clusters as base resources and the overlay selects only which
one the shared consumer REFERENCES — i.e. exactly the R-ENV-02 contract the
regex exists to protect. Renaming the files to dodge the regex was rejected as
gaming the guard, so the rule now encodes the property instead.
The fix is not an allowlist of those four paths. A prod-token file in a shared
path is excused only when the tree proves all of:
(A) it declares nothing but Secrets whose names end in the environment suffix
(1) its own directory's kustomization applies it, and EVERY environment value
of the selector resolves to a Secret that some resource of that same
kustomization declares, in the same namespace
(2) nothing names the suffixed Secret — or the suffixed file — literally;
every reference is `<base-name>${VAR}`
(3) that VAR is defined for the base and for every overlay, and prod's value
is the one that selects this file
References are read from parsed YAML values, never raw text: the mechanism is
spelled out by name in comments beside the resources, and a comment references
nothing.
No other assertion is weakened; the check still runs 96 assertions, and the
pass output now notes each excused file and why.
|
||
|
jakman85andClaude Opus 5
|
dc2d1ce17e |
fix(conformance): the R-NET-04 scan descended into nested agent worktrees
TRACKED_YAML rglob'd from the repo root excluding only .git and _artifacts, so it also walked .claude/worktrees/agent-*/ — the isolated worktrees created for agent runs. The check counted 5 CiliumEgressGatewayPolicy objects where exactly 1 is authored, failing R-NET-04 for a reason unrelated to egress steering. Now skips dot-DIRECTORIES (covering .git, .claude, .cache) while still matching dot-FILES such as .sops.yaml. .claude/ is gitignored, so nothing beneath it is authored surface. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
6a8953e288 |
feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing
Minimal prod delivery, authored directly on main (= prod's live revision
|
||
|
jakman85andClaude Opus 5
|
c8c1d2935d |
feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain
INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).
THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.
Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.
PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.
BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.
TWO THAT CANNOT BE TOKENIZED, flagged not forced:
- headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
anyway — client secrets are realm-scoped.
- kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
prod.home.
EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.
I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.
DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.
L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.
PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.
Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
8a2be824f6 |
feat(ci): substitution-safety gate + escape 6 comment hazards; strict mode is now SAFE
Adds conformance/tests/l0-static/flux-postbuild-substitution/ — an offline, stdlib-only
L0 check tagged R-ENV-02 + R-GITOPS-01 (deliberately NOT R-TEST-01, which would
overclaim). It reuses lib/status.py's `resources:` walk rather than forking it, so "what a
path renders" has exactly one definition. It asserts: every ${...} on a substituted path
is defined-or-escaped (comments included); no default-operator form on an undefined name;
no unparseable expansion; the rendering path actually substitutes cluster-vars; every
overlay-patched /data/KEY has a base default; and that the gate itself is load-bearing.
WIRED AS AN UNCONDITIONAL STAGE (3c) in ci/l0.sh, and that matters: stage 3b scopes to
touched requirements, and ci/touched-requirements.sh maps only clusters/* to R-ENV-02 —
so an edit to a substituted manifest under apps/ or platform/ would have filtered this
check out of exactly the change that introduces the bug. l0 fail -> BLOCK=1 in gate.sh.
Mutation-tested all five assertion classes (undefined token, indexed expansion, :- form,
missing substituteFrom, un-wired gate); each fails as intended, baseline green.
STRICT MODE IS PROVEN SAFE TO ENABLE — measured, not inspected. For all 11 substituted
paths, `kubectl kustomize --load-restrictor LoadRestrictionsNone <path> | flux envsubst
--strict` exits 0 against cluster-vars base defaults. Negative control confirms the
incident: with K3S_API_HOST unset, non-strict renders k8sServiceHost empty while strict
exits 1 with `variable not set (strict mode)`. The feature gate itself is NOT flipped
here — that is a live-cluster change and the operator's call.
Escapes the six comment hazards that stood in the way (apps/vms/unifi x4,
platform/longhorn x2). All were prose ABOUT the escaping convention; since the repo's own
contract is that $${...} emits a literal ${...}, escaping them is both precise and
self-consistent. kustomize strips these particular comments today, but comment survival is
an implementation detail — a body comment carrying $ WAS observed surviving into built
apps/wave-1 — and a survivor would kill the whole Kustomization.
DOCTRINE CORRECTED (measured against flux 2.4.0): clusters/base/apps-vms.yaml claimed an
indexed expansion is fatal "escaped or not". Escaping is TOTAL — $${ARR[0]} emits the
literal text safely; only the UNESCAPED indexed form is fatal. Corrected in place.
Also empties DOCUMENTED_GAPS in conformance/run.sh with rationale: R-ENV-02 and
R-GITOPS-01 now have a real verifying test, which run.sh's own comment named as the
trigger for removing them.
Advisory output (not failures): 6 dead cluster-vars knobs — BASE_DOMAIN, CLUSTER_ENV,
CLUSTER_POD_CIDR, PIA_REGION, PLATFORM_NODE_SELECTOR_{KEY,VALUE}. The last four are
documented reference-only; BASE_DOMAIN and CLUSTER_ENV look like genuine rot (hostnames
are hardcoded elsewhere) and are worth a follow-up.
Gate: MERGE GATE: PASS. 102 assertions, 0 hazards.
Serves: R-ENV-02, R-GITOPS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
14be9b68f6 |
test(conformance): dr-restore actually asserts off-cluster restore (R-STATE-03)
Closes the LAST scaffold-only safety-critical requirement. The operator has attested DR works (Velero -> Proxmox versitygw S3, restore demonstrated), but the L3 test carried the right tag and layer label over a 5-line TODO body — so coverage counted it and reported "0 safety-critical gaps" while the suite asserted nothing. That was test debt, not capability debt. This makes the suite assert what the operator already demonstrated. Three steps: 1. assert-backup-target-is-off-cluster (read-only): BSL must be phase Available (Velero itself reached the store), s3Url host is not localhost/*.svc/*.cluster.local, and the resolved IPv4 is none of — loopback, any Node Internal/ExternalIP, any Service ClusterIP, any Pod IP, inside any node podCIDR, inside any ServiceCIDR. A BSL that exists but never reaches Available FAILS (a live DR defect), it does not skip. 2. assert-backups-are-scheduled: R-STATE-03 says SCHEDULED off-cluster backups — Velero + BSL but zero Schedule in phase Enabled is a fail. 3. The drill: seed a nanosecond-unique canary onto a longhorn PVC in its own scratch namespace; Backup to Completed; assert a PodVolumeBackup for that volume is Completed WITH a non-empty status.snapshotID (the kopia snapshot of volume DATA reached the remote repo, not just resource JSON); ANNIHILATE the source (delete ns, poll until gone, force-delete a lingering PV, FAIL if the original PV survived — a survivor makes the restore prove nothing); Restore with namespaceMapping into a clean namespace; assert the restored PVC bound a DIFFERENT PV and that the canary bytes are identical. Off-cluster proof = endpoint outside every cluster address + BSL Available + kopia snapshotID present + every in-cluster copy destroyed + restore landing on a new PV. The bytes could only have come from outside. Skips GREEN on: no longhorn SC, Velero API absent, no BSL, Velero/node-agent not ready — all three steps re-derive the same gates so it can never half-run. An RBAC denial is deliberately NOT a skip: it fails loudly as a harness defect. Teardown is a step-level `finally` that ALWAYS runs — deletes the Restore, issues a DeleteBackupRequest so objects leave the bucket (2h ttl as backstop), deletes both scratch namespaces by name and by label, releases leftover PVs whose claimRef is one of its own. Everything it destroys is something it created. conformance/incluster/rbac.yaml: authored the R-STATE-03 grants (velero.io backups/ restores/deletebackuprequests create+delete+read; BSL/schedules/podvolume* read-only; persistentvolumes get/list/DELETE — the off-cluster proof requires destroying the PV). Also two pre-existing gaps other tests already depended on: storageclasses get/list (the skip gate of five tests) and apps deployments/daemonsets read. No write verbs on platform controllers. NOT ASSERTED, stated in the file header rather than implied: off-HOST (on dev the nodes are VMs on the same Proxmox host as the S3 target, so this does not survive hypervisor loss); no direct S3 client read (that would hand a privileged test identity the backup credential for weaker evidence); real service data (Nextcloud/Gitea/CNPG drills stay operator exercises); bare-metal rebuild DR; encryption-at-rest; retention/GC. status: scaffold-only 5 -> 4; safety-critical w/o executable L3 1 -> 0 (of 6); executable 35 -> 36. coverage unchanged at 40/51 (correct — coverage only asks whether a tagged test exists). PROVEN stayed 0 / not-derivable: it did NOT flip to proven just because the body is non-empty. AUTHORED, NEVER EXECUTED — carries a conformance.home.k3s/never-executed annotation. No cluster access from this workstation. To convert authored -> proven the operator must, IN A MAINTENANCE WINDOW (it deletes namespaces and PVs and writes/deletes bucket objects): apply the updated rbac.yaml, run `./conformance/run.sh --requirements R-STATE-03 --layer l3`, confirm the log shows DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK and NOT DR_RESTORE_SKIPPED_OK (green but proves nothing), confirm no leaked *-dr-src/*-dr-dst namespaces or orphan PVs or *-dr-backup objects, and record the green run. Serves: R-STATE-03, R-TEST-05. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |