feat(obs): wire the gate-sidecar scrape, and fix the alert that would have gone silent doing it

PR #28 gave the five fleet oauth2-proxy sidecars --metrics-address=0.0.0.0:44180 and a
named `metrics` port. This wires a scraper to them and, in the process, found a real bug
in the alert this branch added.

PodMonitor, not ServiceMonitor. Measured: no Service exposes 44180 on any of the five
(jackett/radarr/sonarr/qbittorrent/pihole all publish only their app port + authproxy
4180), so a ServiceMonitor would select nothing. A PodMonitor targets pod ports directly
and needs no change to catalog/render.py or the apps/wave-* renders and goldens.
Selector is the pod label homelab.keithjackson.dev/auth-mode=forward-auth — present on
exactly those five gate pods, absent from crafty's, which still has no metrics listener.
The property, not an app list.

THE BUG: `up{container="oauth2-proxy"}` -> `up{container="oauth2-proxy"} == 1`.

A target that is discovered but FAILING still publishes `up`, with value 0. The series
exists, so count by (namespace) returned 1, `unless` removed the namespace, and
OAuth2ProxyGateUnobserved went QUIET while nothing was being scraped at all — this
branch's own disease reproduced inside the alert written to cure it. Not hypothetical:
the PodMonitor lands five targets whose scrape is dropped, so the bare form would have
silently cleared on this very commit. Proven by mutation: reverting the `== 1` fails the
two new unit-test scenarios (got: nil) and leaves every other assertion green.

The scrape does NOT work yet, and that is documented rather than papered over. From a
pod in ns monitoring to the jackett pod, :44180 AND :4180 both time out; the shared
proxy's :44180 answers. ns jackett's default-deny is podSelector:{} with
policyTypes:[Ingress,Egress] and NO ingress key (deny-all), and the one allowance is
fromEntities:[ingress] to 4180 — the Gateway's Envoy only. The endpoint itself is
healthy: port-forwarded, /metrics returns http=200, 10208 bytes, all three oauth2_proxy_*
families including requests_total{code="302"}. Same bug and fingerprint as
platform/longhorn/networkpolicy-metrics.yaml records for Longhorn; the fix is one narrow
rule per app, and it cannot ship from this tier because oauth2-proxy and apps-wave-0 are
Flux SIBLINGS (both dependsOn platform-config), so naming ns jackett here would apply
today and fail on a cold rebuild. Needs an owner in the renderer.

Touches conformance/** : one KNOWN_EXTERNAL_MONITORS entry, which is the mechanism the
failing check explicitly instructs ("DECLARE it ... with who creates it") and which
flux-podmonitor.yaml already uses. It records that the port name was measured on prod
instead of resolved from a Service, and that the targets are currently down. That makes
this PR require a non-author review on the sensitive-path rule.

Also: ratio alert still NOT shipped. oauth2_proxy_requests_total now exists on the
endpoint but not in the TSDB, so writing it today would make it inert for exactly the
reason it was refused on 2026-09-30. Runbook gains policy-vs-no-target triage and the
radarr /signalr/messages/negotiate worked case — one orphaned browser tab, 206 identical
302s, nobody ever re-prompted.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml

@@ -104,7 +104,67 @@ tests:
         exp_samples: []
 
   # ---------------------------------------------------------------------------------------------
-  # 3. NO GATE, NO ALERT. A namespace with no oauth2-proxy container must never appear, or the
+  # 3. THE REGRESSION GUARD FOR `== 1` — the scenario that exposed a real bug on 2026-10-01.
+  #
+  #    A target is DISCOVERED but every scrape FAILS: `up` exists with value 0. This is not
+  #    hypothetical — it is the exact state podmonitor.yaml creates today, because the app
+  #    namespaces' default-deny drops the scrape (measured; see podmonitor.yaml).
+  #
+  #    With the original `unless count by (namespace) (up{container="oauth2-proxy"})`, the `up`
+  #    SERIES EXISTS, so `count by (namespace)` returns 1, `unless` removes the namespace, and the
+  #    alert GOES QUIET while nothing is being scraped at all — a green signal standing in for an
+  #    absent one, which is the precise disease this whole rule was written to cure. Deleting the
+  #    `== 1` makes THIS test fail and every other test in this file still pass, which is the
+  #    property that makes it worth having.
+  - interval: 1m
+    name: a discovered-but-FAILING target does not count as observed (up==0 must still fire)
+    input_series:
+      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-6877fcd9f5-ht9d9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
+        values: '1x130'
+      # The PodMonitor target exists and never succeeds: NetworkPolicy drops the packets, so the
+      # scrape times out and `up` is published as 0 rather than going absent.
+      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.0.189:44180",job="oauth2-proxy-sidecars",namespace="jackett",pod="jackett-6877fcd9f5-ht9d9"}'
+        values: '0x130'
+    promql_expr_test:
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett"}'
+        eval_time: 65m
+        exp_samples:
+          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
+            value: 1
+      # And the inner clause itself yields NOTHING while the scrape is failing — which is the
+      # mechanism the alert depends on, asserted directly so a future reader can see why the
+      # `== 1` matters without re-deriving it. (Scenario 4 below covers the other direction:
+      # a succeeding scrape really does clear the alert, so `== 1` has not made the rule
+      # impossible to satisfy.)
+      - expr: 'count by (namespace) (up{container="oauth2-proxy"} == 1)'
+        eval_time: 65m
+        exp_samples: []
+
+  # ---------------------------------------------------------------------------------------------
+  # 4. A FLAPPING target: up goes 1 -> 0 and stays there. The alert must come BACK, not latch
+  #    cleared. Covers the ordering bug where a once-observed namespace is permanently excluded.
+  - interval: 1m
+    name: a target that starts working and then fails re-fires
+    input_series:
+      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="radarr",pod="radarr-7d55cbffb9-42hmh",service="kube-prometheus-stack-kube-state-metrics",uid="3333cccc-0000-0000-0000-000000000001"}'
+        values: '1x250'
+      # Healthy for the first 60m, then every scrape fails for the rest.
+      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.77:44180",job="oauth2-proxy-sidecars",namespace="radarr",pod="radarr-7d55cbffb9-42hmh"}'
+        values: '1x60 0x180'
+    promql_expr_test:
+      # Quiet while it was working.
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="radarr"}'
+        eval_time: 55m
+        exp_samples: []
+      # Firing again an hour after it broke.
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="radarr"}'
+        eval_time: 125m
+        exp_samples:
+          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="radarr",severity="warning"}'
+            value: 1
+
+  # ---------------------------------------------------------------------------------------------
+  # 5. NO GATE, NO ALERT. A namespace with no oauth2-proxy container must never appear, or the
   #    rule would page for every app on the platform the day someone widened the matcher.
   - interval: 1m
     name: namespaces without a gate sidecar are never reported