2 Commits

Include renames

Author	SHA1	Message	Date
jakman85andClaude Opus 5	cf7d97e546	fix(obs): twelve app SLI rules read a signal that exists; kill the dead ServiceMonitors ... Twelve `<app>-down` SLI rules were STRUCTURALLY INERT. Each read `up{job="<app>"} == 0`, and the ServiceMonitor rendered beside it named `endpoints[].port: metrics` — a port NO rendered app Service defined. Every monitor discovered ZERO targets, so no `up{job="<app>"}` series existed and the rule could not fire. Verified on prod before this change: `count by (job) (up)` listed 24 jobs and not one of the twelve appeared; `up{job="plex"}` and `up{job="gitea"}` both returned nothing. `PrometheusTargetsMissing` could not see it either — a target that was never discovered has no `up` series to be 0. THE PORT NAME WAS NEVER THE MISSING PIECE. Ground truth per app: Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud export NO Prometheus metrics without a separate exporter sidecar; Gitea can but only with app.ini metrics enabled; Keycloak does, on the operator's management port, already scraped by platform/keycloak/servicemonitor.yaml; vm-demo is a KubeVirt VM with no Service at all, so its monitor selected nothing whatsoever. `metrics: true` was asserting a fact none of them had. There was also a THIRD layer of inertness nobody had hit yet: every app namespace ships default-deny with `ingress: []` and the renderer emitted no monitoring allow, so even a correct port would have been dropped by policy — while renderer-contract.md claimed the baseline carried such an allow. FIXED IN THE GENERATOR, not in twelve rendered files: * `slis: [up]` now renders a `<app>-down` alert against kube-state-metrics readiness of the service's own workload object — `kube_deployment_status_replicas_available{namespace,deployment} == 0 or absent(...)`, `kube_statefulset_status_replicas_ready` for a StatefulSet, and `absent(kubevirt_vmi_info{namespace,name,phase="running"})` for a VM. The same idiom KeycloakDown has used since it was fixed. kube-state-metrics IS scraped, so this needs no exporter and no new target. The `or absent(...)` half is load-bearing: the signal DISAPPEARING now fires the alert instead of silencing it. * schema 1.6.0 adds `observability.metricsEndpoint` — WHERE metrics are served. A ServiceMonitor is rendered ONLY from it, and it renders all three halves of a working scrape together: the named Service port, the monitor endpoint, and the monitoring-namespace CiliumNetworkPolicy allow. No catalog entry declares one, so twelve dead ServiceMonitors are gone. `metrics: true` keeps meaning logs-to-Loki plus kube-state-metrics liveness. * schema 1.6.0 also adds `observability.livenessWorkload`, and catalog/keycloak.yaml uses it. Keycloak's server is a StatefulSet run by the operator; the default Deployment shape would have made its `absent()` half fire FOREVER — the other failure mode this repo has shipped. * renderer selftest gains cases 9/9a/9b/9c: no metricsEndpoint => no monitor; the up SLI must not read `up{job=...}`; endpoint port must exist on the Service; the metrics port must NOT reach the Gateway allow; an unknown livenessWorkload.kind is REFUSED. PROOF, both directions, for all twelve (apps/wave-{0,1,2}/tests/app-slis.test.yaml, run by platform/monitoring/tests/run.sh over the RENDERED manifests): * quiet on the healthy state; firing on no-ready-replica; firing when the series is ABSENT; the VM asserted both gone and in a non-running phase; per-app isolation asserted, and keycloak asserted un-maskable by its neighbours (keycloak-operator Deployment, keycloak-db StatefulSet ready in the same namespace). * mutation-checked: six mutations, each turning its suite red — reverting plex to `up{job=...} == 0`; dropping seerr's `absent()` half; pointing keycloak at a Deployment; capitalising the VMI phase to `Running`; widening keycloak's matcher to the namespace; inverting plex's `== 0` to `!= 0`. Control run green. * live on prod with `promtool query instant`: all ten deployed app Deployments export `kube_deployment_status_replicas_available`, `kube_statefulset_status_replicas_ready{namespace="keycloak",statefulset="keycloak"}` = 2, `kubevirt_vmi_info` present with a lowercase `phase="running"`; and evaluating the twelve SHIPPED expressions returns no series for eleven (quiet, not permanently firing). The twelfth is vm-demo, which is not in apps/wave-0/kustomization.yaml and so is not deployed — its rule is not applied either. DELETION WAS NOT THE ANSWER, and the evidence is recorded rather than assumed: kube-prometheus-stack's `kubernetes-apps` group IS live on prod (17 rules, KubePodNotReady / KubeDeploymentReplicasMismatch among them), so app pods are not unmonitored — but that is a generic warning keyed to a namespace, not the per-service CRITICAL SLI R-OBS-01 declares. platform/monitoring/alerts.yaml carries no generic app-readiness rule at all. GUARD SO IT CANNOT RECUR SILENTLY: new whole-repo L0 invariant conformance/tests/l0-static/servicemonitor-ports-resolve. It walks the MONITORS, not the alerts, so it needs no `up{job=...}` rule to reach one, and resolves each endpoint by SELECTOR across the whole tree rather than by same-file adjacency. Fixing the alerts DELETED the only thing that made the broken monitors visible to the sibling check, which is exactly why this axis needed its own home. It declares `conformance.home.k3s/scope: whole-repo` and adds no ci/l0.sh stage. Proven to FAIL on the pre-fix tree: 15 failures — eleven port disagreements, vm-demo's monitor matching no Service at all, and the three generator/contract assertions — and to pass after. Its resolver is additionally proven on in-check fixtures (agreeing pair accepted, disagreeing pair rejected, target-less endpoint rejected), because after the fix there are zero in-repo monitors left for it to judge and a guard whose only evidence is "nothing failed" cannot be told apart from one that stopped working. It also asserts r_servicemonitor does not HARDCODE a port name — the mechanism by which one edit gave twelve services the same wrong one. Also fixed, found on the way: * platform/monitoring/tests/run.sh discovered the SUITE convention only inside platform/monitoring/tests/ while documenting it as general — a suite added anywhere else was never run and still looked like coverage. Now discovered tree-wide; a misplaced *.test.yaml fails loudly instead of vanishing. * KNOWN_UNWIRED_JOBS in alert-metrics-have-scrape-targets is now EMPTY, and the twelve apps/* entries in alert-rules-unit-tested's UNTESTED_FILES are gone — both because the gaps were CLOSED, not waived. Both registers are shrink-only and a stale entry fails. * renderer-contract.md claimed the default-deny baseline carried a standing monitoring scrape allow. It never did; the claim is corrected and the allow is now real, per declared metrics endpoint and scoped to that one port. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 17:11:30 -05:00
jakman85andClaude Opus 5	6b953cc56b	fix(arrs): the config PVC was mounted where the app never looks ... sonarr, radarr, jackett and qbittorrent all mounted their Longhorn config PVC at the renderer's default /data/config, but every one of these images runs with `-data=/config`. So the PVC held nothing but lost+found while the app wrote its entire configuration to the container's EPHEMERAL writable layer — destroyed on every pod recreation: restart, reschedule, image bump. Measured in-cluster before fixing, per app, rather than inferred: /data/config dev=2080 [lost+found] <- the Longhorn PVC /config dev=2053 [Sentry asp config.xml logs ...] <- the node's own disk and a marker file written to /config did not appear in /data/config, so they are demonstrably different storage. qbittorrent's /config sat on dev=66309 (/dev/nvme0n1p5), i.e. whichever node it happened to land on. This is the SAME failure catalog/plex.yaml:54 already records having fixed for Plex; plex was the only one of the five that was right. The live /config contents were copied onto each PVC BEFORE this landed, so jackett's API key and qbittorrent's credentials — which other apps reference — survive the move rather than being regenerated. catalog/sonarr.yaml:61 documented this as a latent bug and deliberately left it because it changes live on-disk state. It is being fixed now because all four pods had just been recreated, so their configs were freshly generated and nothing had accumulated to lose. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 16:35:18 -05:00