feat(obs): wire the gate-sidecar scrape, and fix the alert that would have gone silent doing it

PR #28 gave the five fleet oauth2-proxy sidecars --metrics-address=0.0.0.0:44180 and a
named `metrics` port. This wires a scraper to them and, in the process, found a real bug
in the alert this branch added.

PodMonitor, not ServiceMonitor. Measured: no Service exposes 44180 on any of the five
(jackett/radarr/sonarr/qbittorrent/pihole all publish only their app port + authproxy
4180), so a ServiceMonitor would select nothing. A PodMonitor targets pod ports directly
and needs no change to catalog/render.py or the apps/wave-* renders and goldens.
Selector is the pod label homelab.keithjackson.dev/auth-mode=forward-auth — present on
exactly those five gate pods, absent from crafty's, which still has no metrics listener.
The property, not an app list.

THE BUG: `up{container="oauth2-proxy"}` -> `up{container="oauth2-proxy"} == 1`.

A target that is discovered but FAILING still publishes `up`, with value 0. The series
exists, so count by (namespace) returned 1, `unless` removed the namespace, and
OAuth2ProxyGateUnobserved went QUIET while nothing was being scraped at all — this
branch's own disease reproduced inside the alert written to cure it. Not hypothetical:
the PodMonitor lands five targets whose scrape is dropped, so the bare form would have
silently cleared on this very commit. Proven by mutation: reverting the `== 1` fails the
two new unit-test scenarios (got: nil) and leaves every other assertion green.

The scrape does NOT work yet, and that is documented rather than papered over. From a
pod in ns monitoring to the jackett pod, :44180 AND :4180 both time out; the shared
proxy's :44180 answers. ns jackett's default-deny is podSelector:{} with
policyTypes:[Ingress,Egress] and NO ingress key (deny-all), and the one allowance is
fromEntities:[ingress] to 4180 — the Gateway's Envoy only. The endpoint itself is
healthy: port-forwarded, /metrics returns http=200, 10208 bytes, all three oauth2_proxy_*
families including requests_total{code="302"}. Same bug and fingerprint as
platform/longhorn/networkpolicy-metrics.yaml records for Longhorn; the fix is one narrow
rule per app, and it cannot ship from this tier because oauth2-proxy and apps-wave-0 are
Flux SIBLINGS (both dependsOn platform-config), so naming ns jackett here would apply
today and fail on a cold rebuild. Needs an owner in the renderer.

Touches conformance/** : one KNOWN_EXTERNAL_MONITORS entry, which is the mechanism the
failing check explicitly instructs ("DECLARE it ... with who creates it") and which
flux-podmonitor.yaml already uses. It records that the port name was measured on prod
instead of resolved from a Service, and that the targets are currently down. That makes
this PR require a non-author review on the sensitive-path rule.

Also: ratio alert still NOT shipped. oauth2_proxy_requests_total now exists on the
endpoint but not in the TSDB, so writing it today would make it inert for exactly the
reason it was refused on 2026-09-30. Runbook gains policy-vs-no-target triage and the
radarr /signalr/messages/negotiate worked case — one orphaned browser tab, 206 identical
302s, nobody ever re-prompted.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

platform/oauth2-proxy/README.md

@@ -124,3 +124,36 @@ and it reads the sidecar logs out of Loki (Alloy collects them; 72h retention).
 Service port, and a ServiceMonitor selecting it. Only then does the auth-churn rule — sustained
 `code="302"` as a share of `oauth2_proxy_requests_total`, held across a window longer than the
 session lifetime — become writable rather than decorative.
+
+### Update 2026-10-01: the listener exists; the scrape path does not
+
+PR #28 added `--metrics-address=0.0.0.0:44180` and a named `metrics` container port to the five
+fleet sidecars (jackett, radarr, sonarr, qbittorrent, pihole). `podmonitor.yaml` in this directory
+discovers them by the pod label `homelab.keithjackson.dev/auth-mode: forward-auth` — a **PodMonitor**
+rather than a ServiceMonitor because no Service exposes 44180 and adding one would mean changing the
+renderer.
+
+The endpoint is real: port-forwarded straight to the jackett pod, `/metrics` returns **http=200,
+10208 bytes**, exporting `oauth2_proxy_requests_in_flight`, `oauth2_proxy_requests_total` (with
+`code` labels) and `oauth2_proxy_response_duration_seconds`. That is verified by payload, not by the
+flag appearing in `args` — a distinction worth keeping, because the two claims came apart here.
+
+**The scrape is nevertheless dropped.** From a pod in ns `monitoring`, both `:44180` and `:4180` on
+the jackett pod time out; the shared proxy's `:44180` answers. The only difference is that ns
+`oauth2-proxy` has **no NetworkPolicy at all**, while each app namespace has a `default-deny` whose
+`policyTypes` include `Ingress` with no `ingress` rules (deny-all) plus a single CNP admitting
+`fromEntities: [ingress]` to port 4180 — the Gateway's Envoy and nothing else. Prometheus matches no
+clause.
+
+So the targets read `up == 0` and the TSDB is still empty for those namespaces. This is the same bug
+and the same fingerprint as `platform/longhorn/networkpolicy-metrics.yaml` documents for Longhorn,
+and the same narrow fix applies: ns `monitoring` → the gate pod label → TCP 44180 only. It cannot
+ship from this directory, because `oauth2-proxy` and `apps-wave-0` are **siblings** in the Flux graph
+(both `dependsOn: [platform-config]`), so a NetworkPolicy naming ns `jackett` would apply on today's
+cluster and fail on a cold rebuild. It belongs with each app's own rendered policy.
+
+Consequence for `auth-visibility-alerts.yaml`: its `unless` clause requires `up == 1`, not merely a
+present `up` series. A discovered-but-failing target publishes `up` with value 0, so the bare form
+would have cleared `OAuth2ProxyGateUnobserved` the moment the PodMonitor landed — reporting the gates
+as observed while nothing was being scraped. The unit test pins that scenario; deleting the `== 1`
+fails it and nothing else.