feat(obs): make oauth2-proxy auth failures visible — alert on the gap, not on absent metrics

Jackett failed on 2026-09-30 in the most expensive way available: HTTP 200 and a
plausible page. oauth2-proxy answers an unauthenticated XHR with a 302 to Keycloak
rather than a 401; the browser follows it silently against a live SSO session and
the call resolves 200 carrying a 44 KB HTML dashboard where the app's JavaScript
wanted JSON. The user is never re-prompted, reloading never helps, and every uptime
check stays green. `jackett-down` watches replicas_available, which was 1 throughout.

The reason nothing alerted turns out not to be a missing rule. Measured on prod:

    count by (__name__) ({__name__=~"oauth2_proxy.*"})
      oauth2_proxy_requests_in_flight => 2

Two series, both from the SHARED forward-auth Deployment. The six per-app gate
sidecars (crafty, jackett, pihole, qbittorrent, radarr, sonarr) publish nothing at
all: no --metrics-address, so the metrics listener never starts; no `metrics`
container port; no ServiceMonitor or PodMonitor selects them. The alert the incident
calls for — sustained auth redirects to an authenticated user — is therefore NOT
WRITABLE, and writing it anyway would have shipped a rule that is present,
evaluating, permanently silent and indistinguishable from health.

So alert on the condition that made the incident invisible, from families that ARE
scraped:

  OAuth2ProxyGateUnobserved
    count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
      unless
    count by (namespace) (up{container="oauth2-proxy"})

kube_pod_container_info is kube-state-metrics (a declared scrape owner) and sees the
container whether or not it exports anything; `up` is UNIVERSAL and carries a
`container` label for any target, so the rule needs no advance knowledge of what job
name a future per-app scrape picks. Nothing in it touches oauth2_proxy_*, so it
cannot be inert for the reason it is reporting. Run verbatim against live prod it
returns six namespaces and correctly excludes oauth2-proxy. It clears per namespace
as each gate gains a real scrape.

Also:
- promtool companion test pinning all four states: fires unscraped, quiet when
  scraped, honours for:1h, and STOPS once a scrape appears under a different job name.
- runbooks/oauth2-proxy-auth-failures.md — tell the gate's refusal from the group
  check from the app's own login from an XHR that silently got HTML, with the
  2026-09-30 jackett trace as the worked example and the Loki queries to use while
  there are no metrics. Registered in the configMapGenerator and README.
- runbooks/README.md gains an inert-alerts row for the unscraped sidecars.

Rejected: an alert on the shared proxy's oauth2_proxy_requests_total ratio. It needs
a conformance/ SCRAPE_OWNERS entry (human approval) and, decisively, would not have
fired on this incident — the shared instance is not in any sidecar-gated app's data
path and its current pods have proxied nothing. It would have looked like coverage
and been none.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

runbooks/README.md

@@ -51,6 +51,18 @@ rule groups still evaluate against nothing and **cannot fire**:
 | Flux (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` |
 | Hubble (`hubble_drop_total`) | `HubbleDeniedFlowSpike` |
 | etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all |
+| per-app oauth2-proxy sidecars (`oauth2_proxy_*`) — measured 2026-09-30 | no authentication alerting exists at all for crafty, jackett, pihole, qbittorrent, radarr, sonarr |
+
+The oauth2-proxy row is the newest and the least obvious, because the scrape is not merely missing —
+there is **nothing to scrape**. `count by (__name__) ({__name__=~"oauth2_proxy.*"})` on prod returns
+exactly one family, `oauth2_proxy_requests_in_flight`, from the two replicas of the **shared**
+forward-auth instance in the `oauth2-proxy` namespace. The six per-app gate sidecars run without
+`--metrics-address`, so their metrics listener never starts; they declare no `metrics` container port;
+and no ServiceMonitor or PodMonitor selects them. Any rule written against `oauth2_proxy_*` for those
+namespaces would be inert on arrival. `OAuth2ProxyGateUnobserved`
+(`platform/oauth2-proxy/auth-visibility-alerts.yaml`) alerts on that gap itself, using
+`kube_pod_container_info` and `up` — two families that *are* scraped — and clears when a gate gains a
+real scrape. See `oauth2-proxy-auth-failures`.
 
 Each affected runbook says so on its own page and gives the by-hand equivalent. See `targets-down`.
 
@@ -108,6 +120,14 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
 - **`keycloak-db`** — Keycloak's CloudNativePG database is unavailable.
 - **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
   every green signal missed.
+- **`oauth2-proxy-auth-failures`** — a gated app "is up but does not work". Linked from
+  `OAuth2ProxyGateUnobserved`. The gate answers an unauthenticated **XHR** with a 302 to Keycloak
+  rather than a 401, the browser follows it silently against a live SSO session, and the call
+  resolves **200 carrying HTML** where the app's JavaScript wanted JSON — so the user is never
+  re-prompted and every uptime check stays green. Tells the gate's refusal from the group check
+  from the app's own login, from the sidecar log and from Loki, with the 2026-09-30 jackett trace
+  as the worked example. **Read the caveat**: the per-app gate sidecars export no metrics at all,
+  so these logs (72h of them) are the only record this class of failure leaves.
 
 ### Operator procedures (no alert links here)
 - **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only