feat(obs): wire the gate-sidecar scrape, and fix the alert that would have gone silent doing it

PR #28 gave the five fleet oauth2-proxy sidecars --metrics-address=0.0.0.0:44180 and a
named `metrics` port. This wires a scraper to them and, in the process, found a real bug
in the alert this branch added.

PodMonitor, not ServiceMonitor. Measured: no Service exposes 44180 on any of the five
(jackett/radarr/sonarr/qbittorrent/pihole all publish only their app port + authproxy
4180), so a ServiceMonitor would select nothing. A PodMonitor targets pod ports directly
and needs no change to catalog/render.py or the apps/wave-* renders and goldens.
Selector is the pod label homelab.keithjackson.dev/auth-mode=forward-auth — present on
exactly those five gate pods, absent from crafty's, which still has no metrics listener.
The property, not an app list.

THE BUG: `up{container="oauth2-proxy"}` -> `up{container="oauth2-proxy"} == 1`.

A target that is discovered but FAILING still publishes `up`, with value 0. The series
exists, so count by (namespace) returned 1, `unless` removed the namespace, and
OAuth2ProxyGateUnobserved went QUIET while nothing was being scraped at all — this
branch's own disease reproduced inside the alert written to cure it. Not hypothetical:
the PodMonitor lands five targets whose scrape is dropped, so the bare form would have
silently cleared on this very commit. Proven by mutation: reverting the `== 1` fails the
two new unit-test scenarios (got: nil) and leaves every other assertion green.

The scrape does NOT work yet, and that is documented rather than papered over. From a
pod in ns monitoring to the jackett pod, :44180 AND :4180 both time out; the shared
proxy's :44180 answers. ns jackett's default-deny is podSelector:{} with
policyTypes:[Ingress,Egress] and NO ingress key (deny-all), and the one allowance is
fromEntities:[ingress] to 4180 — the Gateway's Envoy only. The endpoint itself is
healthy: port-forwarded, /metrics returns http=200, 10208 bytes, all three oauth2_proxy_*
families including requests_total{code="302"}. Same bug and fingerprint as
platform/longhorn/networkpolicy-metrics.yaml records for Longhorn; the fix is one narrow
rule per app, and it cannot ship from this tier because oauth2-proxy and apps-wave-0 are
Flux SIBLINGS (both dependsOn platform-config), so naming ns jackett here would apply
today and fail on a cold rebuild. Needs an owner in the renderer.

Touches conformance/** : one KNOWN_EXTERNAL_MONITORS entry, which is the mechanism the
failing check explicitly instructs ("DECLARE it ... with who creates it") and which
flux-podmonitor.yaml already uses. It records that the port name was measured on prod
instead of resolved from a Service, and that the targets are currently down. That makes
this PR require a non-author review on the sensitive-path rule.

Also: ratio alert still NOT shipped. oauth2_proxy_requests_total now exists on the
endpoint but not in the TSDB, so writing it today would make it inert for exactly the
reason it was refused on 2026-09-30. Runbook gains policy-vs-no-target triage and the
radarr /signalr/messages/negotiate worked case — one orphaned browser tab, 206 identical
302s, nobody ever re-prompted.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

conformance/tests/l0-static/servicemonitor-ports-resolve/check.sh

@@ -129,6 +129,22 @@ KNOWN_EXTERNAL_MONITORS: dict[str, str] = {
         "a PodMonitor, so it resolves against POD ports, not Service ports — the Flux controller "
         "Deployments (and their `http-prom` container port) are installed by the Flux bootstrap "
         "manifests, not authored here. Live on prod: job flux-system/flux-system, 6 targets up.",
+    "platform/oauth2-proxy/podmonitor.yaml:oauth2-proxy-sidecars":
+        "a PodMonitor, so it resolves against POD ports, not Service ports. Unlike the other "
+        "entries here the target is NOT chart-owned — the `metrics` (44180) container port is "
+        "authored in this repo, on the oauth2-proxy sidecar in apps/wave-0/{jackett,qbittorrent,"
+        "pihole}.yaml and apps/wave-1/{radarr,sonarr}.yaml. It is unauditable here because NO "
+        "Service exposes 44180 BY DESIGN: adding one would mean changing catalog/render.py and "
+        "every apps/wave-* render and golden, to publish an endpoint only Prometheus should dial. "
+        "A PodMonitor needs no Service, which is the whole reason it was chosen. What assertion "
+        "(2) would have checked was instead MEASURED on prod 2026-10-01, read-only: the pods "
+        "declare ports [('http', 4180), ('metrics', 44180)], the selector label "
+        "homelab.keithjackson.dev/auth-mode=forward-auth is on exactly those five gate pods, and "
+        "port-forwarding 44180 returns http=200 with 10208 bytes of oauth2_proxy_* metrics. "
+        "CAVEAT, deliberately recorded: those targets are DOWN on prod, because the app "
+        "namespaces' default-deny NetworkPolicy drops the scrape from ns monitoring (same bug and "
+        "fingerprint as platform/longhorn/networkpolicy-metrics.yaml documents). The port name "
+        "resolves; the network path does not yet.",
 }
 
 MONITOR_KINDS = ("ServiceMonitor", "PodMonitor")

decisions/2026-09-30-oauth2-proxy-auth-failure-visible.md

@@ -173,3 +173,87 @@ Everything below was measured against live prod through the pinned container, re
 3. **Consider whether `--set-xauthrequest` apps should get `--api-routes`** so API paths answer
    401 instead of 302. That is the fix that would make this failure *loud* rather than merely
    observable, and it belongs with whoever owns the sidecar args.
+
+---
+
+# Round 2 — 2026-10-01: the scrape got wired, and found a bug in this branch's own alert
+
+Added to the same PR (#29) rather than a fourth concurrent branch. PR #28 merged in the meantime
+and `main` moved three PRs ahead; `origin/main` is merged in here, and the `runbooks/kustomization.yaml`
+conflict was resolved by **keeping both** runbook entries (mine and `oauth2-proxy-session`).
+
+## What changed
+
+**`platform/oauth2-proxy/podmonitor.yaml` (new).** PR #28 gave the five fleet sidecars
+`--metrics-address=0.0.0.0:44180` and a named `metrics` port. No Service exposes 44180 on any of
+them, so a ServiceMonitor would have nothing to select — a **PodMonitor** targets pod ports directly
+and needs no renderer change. Selects on the pod label `homelab.keithjackson.dev/auth-mode:
+forward-auth`, which is on exactly the five fleet gate pods and absent from crafty's: the property,
+not an app list, so a sixth forward-auth app needs no edit.
+
+**`auth-visibility-alerts.yaml`: `up{container="oauth2-proxy"}` → `up{container="oauth2-proxy"} == 1`.
+This is a real bug fix, and it is the most important thing in this round.** A target that is
+discovered but failing still publishes `up`, with value **0**. The series exists, so
+`count by (namespace)` returned 1, `unless` removed the namespace, and `OAuth2ProxyGateUnobserved`
+would have **gone quiet while nothing was being scraped at all** — this branch's own disease,
+reproduced inside the alert written to cure it. Not hypothetical: the PodMonitor lands five targets
+whose scrape is dropped, so the bare form would have silently cleared on this very commit.
+
+**Two new unit-test scenarios** pinning that: a discovered-but-failing target must still fire, and a
+target that works and then breaks must re-fire rather than latch cleared.
+
+**Docs corrected rather than appended to.** The 2026-09-30 claim "the sidecars export no metrics at
+all" is now only half true and was rewritten in `runbooks/README.md`, `platform/oauth2-proxy/README.md`
+and the alert header, each showing the two-step history, because each step looked like the end of it.
+The runbook gains the policy-vs-no-target triage (step 1b) and the **radarr `/signalr` worked case**.
+
+**Deliberately NOT shipped: the ratio alert.** `oauth2_proxy_requests_total` now exists on the
+endpoint, but not in the TSDB, because the scrape is blocked. Writing the ratio rule today would make
+it inert for precisely the reason it was refused on 2026-09-30. It becomes writable the day a scrape
+succeeds, not the day a listener exists.
+
+## What I verified
+
+- **The endpoint serves — by payload, not by flag.** Port-forwarded to the jackett pod's 44180:
+  `http=200 bytes=10208`, exporting `oauth2_proxy_requests_in_flight`, `oauth2_proxy_requests_total`
+  (`code="200" 3`, `code="302" 2`) and `oauth2_proxy_response_duration_seconds`. `--metrics-address`
+  being in `args` and `/metrics` returning a body are different claims; both now hold.
+- **The scrape path is DROPPED, measured from the right place.** From grafana in ns `monitoring`
+  (beside Prometheus, same policy position) to the jackett pod IP: `:44180` → `wget: download timed
+  out`; `:4180` → **also** timed out; the shared proxy's `:44180` → 200 with `oauth2_proxy_*` present.
+  Cause, read from the objects: ns jackett's `default-deny` is `podSelector: {}`,
+  `policyTypes: [Ingress, Egress]`, with an `egress` rule and **no `ingress` key** (deny-all ingress);
+  the only allowance is CNP `jackett-allow-gateway-ingress`, `fromEntities: [ingress]` to port 4180.
+  The shared proxy is scrapable only because ns `oauth2-proxy` has **no NetworkPolicy at all**.
+- **No Service exposes 44180** on any of the five (enumerated).
+- **The pod label selector is exact:** `auth-mode=forward-auth` on all five fleet gate pods, absent
+  from crafty's. Crafty's sidecar still has no `--metrics-address` (args read in full).
+- **Flux ordering forbids shipping the fix from here:** `oauth2-proxy dependsOn=['platform-config']`
+  and `apps-wave-0 dependsOn=['platform-config']` — siblings, no ordering. A NetworkPolicy naming ns
+  `jackett` from this tier would apply today and fail on a cold rebuild.
+- **The `== 1` guard is non-vacuous — proven by mutation.** Reverting it to the bare `up` makes the
+  two new scenarios FAIL (`got: nil` at 1h5m and 2h5m) and leaves every other assertion passing;
+  restoring it returns the suite to SUCCESS under pinned promtool v3.14.0.
+- **Gate:** see the next section for the exit code and skips.
+
+## What I did NOT verify
+
+- **The PodMonitor has never been applied.** That it discovers the five pods is reasoned from the
+  measured pod labels and port name, not observed — no target list has been read with it in place,
+  because applying it is an operator action. **Expect `up == 0` on all five when it lands**, which is
+  the documented state, not a regression.
+- **I did not fix the NetworkPolicy, and this PR therefore does not make the gates observable.** It
+  makes them *discovered*, and makes the remaining gap loud instead of silent. The one narrow rule
+  per app (ns monitoring → `auth-mode=forward-auth` → TCP 44180, per
+  `platform/longhorn/networkpolicy-metrics.yaml`) has to be rendered with each app's own policy, in
+  the catalog/renderer that is in flight on another branch. **It needs an owner.**
+- **Whether Prometheus will actually select a PodMonitor in ns `oauth2-proxy` targeting other
+  namespaces.** `namespaceSelector.any: true` plus `podMonitorSelectorNilUsesHelmValues: false` and
+  the fact that cross-namespace ServiceMonitors/PrometheusRules already work here say yes; it is not
+  measured with this object.
+- **No session-lifecycle claim can be made from the current window.** `AuthSuccess` is **0** on both
+  jackett and radarr since the PR #28 restart, so nobody has signed in and `--cookie-refresh=2m30s`
+  has never executed a refresh. The cookie fix is unproven in the one dimension that matters most.
+- **The radarr `/signalr` finding is the coordinator's measurement, not mine** — I reproduced neither
+  the 206/209 sample nor the jackett 0-in-13,612. They are recorded as reported.
+- **L2/L3 conformance still did not run** (`CLUSTER_AVAILABLE` unset). Nothing is certified.

platform/oauth2-proxy/README.md

@@ -124,3 +124,36 @@ and it reads the sidecar logs out of Loki (Alloy collects them; 72h retention).
 Service port, and a ServiceMonitor selecting it. Only then does the auth-churn rule — sustained
 `code="302"` as a share of `oauth2_proxy_requests_total`, held across a window longer than the
 session lifetime — become writable rather than decorative.
+
+### Update 2026-10-01: the listener exists; the scrape path does not
+
+PR #28 added `--metrics-address=0.0.0.0:44180` and a named `metrics` container port to the five
+fleet sidecars (jackett, radarr, sonarr, qbittorrent, pihole). `podmonitor.yaml` in this directory
+discovers them by the pod label `homelab.keithjackson.dev/auth-mode: forward-auth` — a **PodMonitor**
+rather than a ServiceMonitor because no Service exposes 44180 and adding one would mean changing the
+renderer.
+
+The endpoint is real: port-forwarded straight to the jackett pod, `/metrics` returns **http=200,
+10208 bytes**, exporting `oauth2_proxy_requests_in_flight`, `oauth2_proxy_requests_total` (with
+`code` labels) and `oauth2_proxy_response_duration_seconds`. That is verified by payload, not by the
+flag appearing in `args` — a distinction worth keeping, because the two claims came apart here.
+
+**The scrape is nevertheless dropped.** From a pod in ns `monitoring`, both `:44180` and `:4180` on
+the jackett pod time out; the shared proxy's `:44180` answers. The only difference is that ns
+`oauth2-proxy` has **no NetworkPolicy at all**, while each app namespace has a `default-deny` whose
+`policyTypes` include `Ingress` with no `ingress` rules (deny-all) plus a single CNP admitting
+`fromEntities: [ingress]` to port 4180 — the Gateway's Envoy and nothing else. Prometheus matches no
+clause.
+
+So the targets read `up == 0` and the TSDB is still empty for those namespaces. This is the same bug
+and the same fingerprint as `platform/longhorn/networkpolicy-metrics.yaml` documents for Longhorn,
+and the same narrow fix applies: ns `monitoring` → the gate pod label → TCP 44180 only. It cannot
+ship from this directory, because `oauth2-proxy` and `apps-wave-0` are **siblings** in the Flux graph
+(both `dependsOn: [platform-config]`), so a NetworkPolicy naming ns `jackett` would apply on today's
+cluster and fail on a cold rebuild. It belongs with each app's own rendered policy.
+
+Consequence for `auth-visibility-alerts.yaml`: its `unless` clause requires `up == 1`, not merely a
+present `up` series. A discovered-but-failing target publishes `up` with value 0, so the bare form
+would have cleared `OAuth2ProxyGateUnobserved` the moment the PodMonitor landed — reporting the gates
+as observed while nothing was being scraped. The unit test pins that scenario; deleting the `== 1`
+fails it and nothing else.

platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml

@@ -104,7 +104,67 @@ tests:
         exp_samples: []
 
   # ---------------------------------------------------------------------------------------------
-  # 3. NO GATE, NO ALERT. A namespace with no oauth2-proxy container must never appear, or the
+  # 3. THE REGRESSION GUARD FOR `== 1` — the scenario that exposed a real bug on 2026-10-01.
+  #
+  #    A target is DISCOVERED but every scrape FAILS: `up` exists with value 0. This is not
+  #    hypothetical — it is the exact state podmonitor.yaml creates today, because the app
+  #    namespaces' default-deny drops the scrape (measured; see podmonitor.yaml).
+  #
+  #    With the original `unless count by (namespace) (up{container="oauth2-proxy"})`, the `up`
+  #    SERIES EXISTS, so `count by (namespace)` returns 1, `unless` removes the namespace, and the
+  #    alert GOES QUIET while nothing is being scraped at all — a green signal standing in for an
+  #    absent one, which is the precise disease this whole rule was written to cure. Deleting the
+  #    `== 1` makes THIS test fail and every other test in this file still pass, which is the
+  #    property that makes it worth having.
+  - interval: 1m
+    name: a discovered-but-FAILING target does not count as observed (up==0 must still fire)
+    input_series:
+      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-6877fcd9f5-ht9d9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
+        values: '1x130'
+      # The PodMonitor target exists and never succeeds: NetworkPolicy drops the packets, so the
+      # scrape times out and `up` is published as 0 rather than going absent.
+      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.0.189:44180",job="oauth2-proxy-sidecars",namespace="jackett",pod="jackett-6877fcd9f5-ht9d9"}'
+        values: '0x130'
+    promql_expr_test:
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett"}'
+        eval_time: 65m
+        exp_samples:
+          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
+            value: 1
+      # And the inner clause itself yields NOTHING while the scrape is failing — which is the
+      # mechanism the alert depends on, asserted directly so a future reader can see why the
+      # `== 1` matters without re-deriving it. (Scenario 4 below covers the other direction:
+      # a succeeding scrape really does clear the alert, so `== 1` has not made the rule
+      # impossible to satisfy.)
+      - expr: 'count by (namespace) (up{container="oauth2-proxy"} == 1)'
+        eval_time: 65m
+        exp_samples: []
+
+  # ---------------------------------------------------------------------------------------------
+  # 4. A FLAPPING target: up goes 1 -> 0 and stays there. The alert must come BACK, not latch
+  #    cleared. Covers the ordering bug where a once-observed namespace is permanently excluded.
+  - interval: 1m
+    name: a target that starts working and then fails re-fires
+    input_series:
+      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="radarr",pod="radarr-7d55cbffb9-42hmh",service="kube-prometheus-stack-kube-state-metrics",uid="3333cccc-0000-0000-0000-000000000001"}'
+        values: '1x250'
+      # Healthy for the first 60m, then every scrape fails for the rest.
+      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.77:44180",job="oauth2-proxy-sidecars",namespace="radarr",pod="radarr-7d55cbffb9-42hmh"}'
+        values: '1x60 0x180'
+    promql_expr_test:
+      # Quiet while it was working.
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="radarr"}'
+        eval_time: 55m
+        exp_samples: []
+      # Firing again an hour after it broke.
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="radarr"}'
+        eval_time: 125m
+        exp_samples:
+          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="radarr",severity="warning"}'
+            value: 1
+
+  # ---------------------------------------------------------------------------------------------
+  # 5. NO GATE, NO ALERT. A namespace with no oauth2-proxy container must never appear, or the
   #    rule would page for every app on the platform the day someone widened the matcher.
   - interval: 1m
     name: namespaces without a gate sidecar are never reported

platform/oauth2-proxy/auth-visibility-alerts.yaml

@@ -33,10 +33,18 @@
 #   $ promtool query instant :9090 'count by (__name__) ({__name__=~"oauth2_proxy.*"})'
 #     oauth2_proxy_requests_in_flight => 2          <- the SHARED proxy's two replicas. That is ALL.
 #
-#   Every sidecar (crafty, jackett, pihole, qbittorrent, radarr, sonarr) runs with
+#   PARTLY SUPERSEDED 2026-10-01, and the correction matters more than the original. PR #28 gave
+#   the five FLEET sidecars `--metrics-address=0.0.0.0:44180` and a named `metrics` port, and the
+#   endpoint genuinely serves (verified by payload: http=200, 10208 bytes, all three
+#   `oauth2_proxy_*` families present — not merely by the flag being in `args`). podmonitor.yaml
+#   in this directory now discovers them. But the scrape is still DROPPED by the app namespaces'
+#   default-deny ingress, so the TSDB remains empty for those namespaces; and crafty still has no
+#   metrics listener at all. The rule below is unchanged in purpose and still has work to do.
+#
+#   Every sidecar (crafty, jackett, pihole, qbittorrent, radarr, sonarr) ran with
 #   `--http-address=0.0.0.0:4180` and NO `--metrics-address`, so oauth2-proxy's metrics listener
-#   is never started; the container declares only port `http:4180`, there is no `metrics` port on
+#   was never started; the container declared only port `http:4180`, there was no `metrics` port on
-#   any of their Services, and there is no ServiceMonitor or PodMonitor for any of them anywhere
+#   any of their Services, and there was no ServiceMonitor or PodMonitor for any of them anywhere
 #   in the cluster (measured: `kubectl get servicemonitors,podmonitors -A`). Six gates, zero
 #   telemetry. An alert written against `oauth2_proxy_requests_total{namespace="jackett"}` would
 #   have been PERMANENTLY INERT — present, evaluating, silent, and indistinguishable from health.
@@ -81,15 +89,31 @@ spec:
           # Nothing in this expression touches `oauth2_proxy_*`, so the rule cannot itself be
           # inert for the reason it is reporting.
           #
-          # MEASURED ON PROD 2026-09-30 — this expression returns six namespaces right now:
+          # MEASURED ON PROD 2026-09-30 — this expression returned six namespaces:
           #   {namespace="crafty"} {namespace="jackett"} {namespace="pihole"}
           #   {namespace="qbittorrent"} {namespace="radarr"} {namespace="sonarr"}
-          # and excludes namespace="oauth2-proxy", whose two replicas ARE scraped. It fires on
+          # and excluded namespace="oauth2-proxy", whose two replicas ARE scraped. It fires on
           # the state the cluster is in; it is not a rule waiting for a hypothetical.
+          #
+          # `== 1` IS LOAD-BEARING, AND IT IS A BUG FIX (2026-10-01). This clause was originally
+          # a bare `up{container="oauth2-proxy"}`, which was WRONG in the one way that matters
+          # here: a target that is DISCOVERED BUT FAILING still publishes `up`, with value 0. So
+          # the series exists, `count by (namespace)` returns 1, `unless` removes the namespace,
+          # and the alert GOES QUIET while nothing whatsoever is being scraped. That is this
+          # branch's own disease — a green signal standing in for an absent one — reproduced
+          # inside the alert written to cure it. It was not hypothetical: podmonitor.yaml lands
+          # five targets whose scrape is DROPPED by NetworkPolicy (measured; see that file), so
+          # the bare form would have silently cleared on the very next commit. `== 1` means
+          # "observed" requires a scrape that actually succeeds.
+          #
+          # A failing target is therefore reported by BOTH this rule and kube-prometheus-stack's
+          # generic TargetDown. That duplication is deliberate: TargetDown says "a target is
+          # down", this says "an authentication gate is unobserved", and during an incident those
+          # are different sentences. The overlap is the cheap half of the trade.
           expr: |
             count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
               unless
-            count by (namespace) (up{container="oauth2-proxy"})
+            count by (namespace) (up{container="oauth2-proxy"} == 1)
           # 1h, not 5m: a rollout briefly has a running sidecar whose target has not been
           # discovered yet, and this condition is a standing architectural gap, never an
           # emergency. Long enough that no deploy can produce it.

platform/oauth2-proxy/kustomization.yaml

@@ -18,3 +18,8 @@ resources:
   # for the same reason it does: monitoring.coreos.com CRDs are registered by the `platform`
   # tier, which is Ready before this tier's platform-config dependency completes.
   - auth-visibility-alerts.yaml
+  # Discovers the PER-APP gate sidecars' :44180 metrics endpoints by pod label (PR #28 added the
+  # listener; no Service exposes it, so this is a PodMonitor, not a ServiceMonitor). The scrape is
+  # still denied by the app namespaces' default-deny ingress — read podmonitor.yaml's header
+  # before concluding the targets should be up.
+  - podmonitor.yaml

platform/oauth2-proxy/podmonitor.yaml

@@ -0,0 +1,109 @@
+---
+# PodMonitor for the PER-APP oauth2-proxy gate sidecars (R-OBS-04).
+#
+# ============================================================================================
+# WHY A PodMonitor AND NOT A ServiceMonitor
+# ============================================================================================
+# PR #28 gave all five fleet sidecars `--metrics-address=0.0.0.0:44180` and a named `metrics`
+# container port. It did NOT add a Service port, and measured on prod 2026-10-01 none of the five
+# Services exposes 44180:
+#
+#   jackett      jackett:     [('port-9117', 9117), ('authproxy', 4180)]
+#   radarr       radarr:      [('port-7878', 7878), ('authproxy', 4180)]
+#   sonarr       sonarr:      [('port-8989', 8989), ('authproxy', 4180)]
+#   qbittorrent  qbittorrent: [('port-8080', 8080), ('authproxy', 4180)]
+#   pihole       pihole:      [('port-80',     80), ('authproxy', 4180)]
+#
+# A ServiceMonitor selects Service PORTS, so it would have nothing to select. A PodMonitor targets
+# pod ports directly and bypasses Services entirely — so this needs no change to the Services, and
+# therefore no change to `catalog/render.py` or the `apps/wave-*` renders and goldens.
+#
+# SELECTOR: the `homelab.keithjackson.dev/auth-mode: forward-auth` POD LABEL, not a namespace list.
+# Measured 2026-10-01: that label is present on exactly the five fleet gate pods (jackett, radarr,
+# sonarr, qbittorrent, pihole) and absent from crafty's, which is a separately-managed deployment
+# (platform/crafty) that still has no `--metrics-address` at all. So the selector describes the
+# PROPERTY being scraped rather than enumerating today's apps, and a sixth forward-auth app is
+# picked up with no edit here. `namespaceSelector.any: true` because those five namespaces are
+# created by the apps-wave-0/1 tiers, which this tier does not and must not depend on.
+#
+# ============================================================================================
+# READ THIS BEFORE BELIEVING THE TARGETS ARE UP: THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY
+# ============================================================================================
+# This PodMonitor is NECESSARY AND NOT SUFFICIENT. The endpoint serves, and the path from ns
+# monitoring is blocked. Both halves measured on prod 2026-10-01, read-only:
+#
+#   THE ENDPOINT IS HEALTHY — port-forward straight to the jackett pod's 44180:
+#     GET /metrics -> http=200 bytes=10208
+#     # TYPE oauth2_proxy_requests_in_flight gauge
+#     # TYPE oauth2_proxy_requests_total counter
+#     # TYPE oauth2_proxy_response_duration_seconds histogram
+#     oauth2_proxy_requests_total{code="200"} 3
+#     oauth2_proxy_requests_total{code="302"} 2
+#
+#   THE SCRAPE PATH IS DROPPED — from a pod in ns monitoring (grafana, which sits beside
+#   Prometheus and so occupies the same policy position) to the jackett pod IP:
+#     http://10.0.0.189:44180/metrics   -> wget: download timed out
+#     http://10.0.0.189:4180/ping       -> wget: download timed out     (even the "allowed" port)
+#     http://<shared-proxy>:44180/metrics -> 200, oauth2_proxy_* present (control: ns oauth2-proxy
+#                                            has NO NetworkPolicy, which is the only reason the
+#                                            shared instance has ever been scrapable)
+#
+#   WHY: ns jackett's `default-deny` NetworkPolicy is `podSelector: {}` with
+#   `policyTypes: [Ingress, Egress]` and an `egress` rule ONLY — no `ingress` key, which in
+#   NetworkPolicy semantics is deny-all-ingress. The only ingress allowance is the CNP
+#   `jackett-allow-gateway-ingress`: `fromEntities: [ingress]` to port 4180 — the Gateway's Envoy
+#   and nothing else. Prometheus matches no clause and its packets are dropped, not refused.
+#
+# SO THE TARGETS BELOW WILL READ `up == 0` UNTIL AN INGRESS ALLOWANCE LANDS. That is a known,
+# measured, documented state — NOT a regression introduced here, and NOT something to silence.
+# It is deliberately left visible: `OAuth2ProxyGateUnobserved` in auth-visibility-alerts.yaml
+# requires `up == 1`, precisely so a discovered-but-failing target does not read as "observed"
+# (see that file's header — this exact trap is why the `== 1` is there).
+#
+# THIS IS THE SAME BUG LONGHORN HAD, and the platform has a documented fix idiom for it:
+# platform/longhorn/networkpolicy-metrics.yaml, whose header records the identical fingerprint
+# measured on 2026-09-13 — ServiceMonitor in place, 3 targets DISCOVERED, `up` 0/0/0, TSDB empty,
+# `lastError: context deadline exceeded` (a drop, not a refusal). The fix there is a narrow
+# additive `networking.k8s.io/v1` NetworkPolicy: ns monitoring -> the pod label, that one port.
+#
+# WHY THAT POLICY IS NOT IN THIS DIRECTORY. It would have to be five objects, one per app
+# namespace, and those namespaces are created by apps-wave-0/1. Measured Flux graph, 2026-10-01:
+#   oauth2-proxy  dependsOn=['platform-config']        <- this tier
+#   apps-wave-0   dependsOn=['platform-config']        <- SIBLING, no ordering between them
+#   apps-wave-1   dependsOn=['apps-wave-0']
+# They are siblings, so on a cold bootstrap there is no guarantee ns jackett exists when this tier
+# applies, and a NetworkPolicy naming a missing namespace fails to apply. It would work on today's
+# running cluster and break on a rebuild — the "works now, broken on restore" shape this repo
+# keeps paying for. Longhorn's precedent is safe only because longhorn-system is created by the
+# same tier that ships its policy. The allowance therefore belongs WITH THE APPS, rendered next to
+# the rest of each app's network policy — i.e. in the catalog/renderer, which is in flight on
+# another branch. Tracked in the handoff; it is one narrow rule per app, not a redesign.
+apiVersion: monitoring.coreos.com/v1
+kind: PodMonitor
+metadata:
+  name: oauth2-proxy-sidecars
+  namespace: oauth2-proxy
+  labels:
+    release: kube-prometheus-stack     # kube-prometheus-stack's podMonitorSelector
+    app.kubernetes.io/name: oauth2-proxy
+    homelab.keithjackson.dev/purpose: metrics-scrape
+spec:
+  namespaceSelector:
+    # The gate sidecars live in the app namespaces, not this one. `any` rather than a fixed list so
+    # a new forward-auth app needs no edit here — the pod label below is what scopes this.
+    any: true
+  selector:
+    matchLabels:
+      # The PROPERTY, not the app list. Present on exactly the five fleet gate pods (measured
+      # 2026-10-01); absent from crafty, which has no metrics listener yet.
+      homelab.keithjackson.dev/auth-mode: forward-auth
+  podMetricsEndpoints:
+    # By PORT NAME (`metrics`, 44180), never by number: the name is what PR #28 declared on the
+    # container, and a numeric port here would silently stop matching if the port were renumbered.
+    - port: metrics
+      path: /metrics
+      interval: 30s
+      # 10s, matching the drop signature above: a DENIED path does not refuse, it hangs until the
+      # scrape times out. Leaving the default would make each failing target hold a scrape slot for
+      # the full interval.
+      scrapeTimeout: 10s

runbooks/README.md

@@ -51,18 +51,36 @@ rule groups still evaluate against nothing and **cannot fire**:
 | Flux (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` |
 | Hubble (`hubble_drop_total`) | `HubbleDeniedFlowSpike` |
 | etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all |
-| per-app oauth2-proxy sidecars (`oauth2_proxy_*`) — measured 2026-09-30 | no authentication alerting exists at all for crafty, jackett, pihole, qbittorrent, radarr, sonarr |
+| per-app oauth2-proxy sidecars (`oauth2_proxy_*`) — updated 2026-10-01 | no authentication alerting exists at all for crafty, jackett, pihole, qbittorrent, radarr, sonarr |
 
-The oauth2-proxy row is the newest and the least obvious, because the scrape is not merely missing —
+The oauth2-proxy row is the newest, and it has **moved twice in two days** — worth reading in order,
-there is **nothing to scrape**. `count by (__name__) ({__name__=~"oauth2_proxy.*"})` on prod returns
+because each step looked like the end of it.
+
+**2026-09-30 — nothing to scrape.** `count by (__name__) ({__name__=~"oauth2_proxy.*"})` returned
 exactly one family, `oauth2_proxy_requests_in_flight`, from the two replicas of the **shared**
-forward-auth instance in the `oauth2-proxy` namespace. The six per-app gate sidecars run without
+forward-auth instance. All six per-app gate sidecars ran without `--metrics-address`, so their metrics
-`--metrics-address`, so their metrics listener never starts; they declare no `metrics` container port;
+listener never started, and no ServiceMonitor or PodMonitor selected them.
-and no ServiceMonitor or PodMonitor selects them. Any rule written against `oauth2_proxy_*` for those
+
-namespaces would be inert on arrival. `OAuth2ProxyGateUnobserved`
+**2026-10-01 — the listener exists, and the scrape is still blocked.** PR #28 added
-(`platform/oauth2-proxy/auth-visibility-alerts.yaml`) alerts on that gap itself, using
+`--metrics-address=0.0.0.0:44180` and a named `metrics` port to the five fleet sidecars, and the
-`kube_pod_container_info` and `up` — two families that *are* scraped — and clears when a gate gains a
+endpoint genuinely serves (port-forwarded: `http=200`, 10208 bytes, all three `oauth2_proxy_*`
-real scrape. See `oauth2-proxy-auth-failures`.
+families, `oauth2_proxy_requests_total{code="302"}` present). `platform/oauth2-proxy/podmonitor.yaml`
+now discovers them by pod label. **But the app namespaces' `default-deny` NetworkPolicy drops the
+scrape**: from a pod in ns `monitoring`, both `:44180` and `:4180` on the jackett pod time out, while
+the shared proxy's `:44180` answers — the only difference being that ns `oauth2-proxy` has no
+NetworkPolicy at all. So the targets are discovered and `up == 0`, and the TSDB is still empty for
+those namespaces. **This is the same bug Longhorn had**, with the same fingerprint; the fix idiom is
+`platform/longhorn/networkpolicy-metrics.yaml` (ns monitoring → one pod label → one port), and for
+these apps it has to be rendered alongside each app's own policy rather than shipped from the
+oauth2-proxy tier, which is a *sibling* of `apps-wave-0` in the Flux graph and so cannot rely on those
+namespaces existing yet.
+
+`OAuth2ProxyGateUnobserved` (`platform/oauth2-proxy/auth-visibility-alerts.yaml`) alerts on the gap
+itself from `kube_pod_container_info` and `up` — two families that *are* scraped. Note its `up == 1`:
+a discovered-but-failing target publishes `up` with value **0**, so the earlier bare form would have
+gone quiet the moment the PodMonitor landed, while nothing was being scraped. That near-miss is the
+whole reason to distrust a cleared alert here. Crafty remains outside all of this — no metrics
+listener at all. See `oauth2-proxy-auth-failures`.
 
 Each affected runbook says so on its own page and gives the by-hand equivalent. See `targets-down`.
 

runbooks/oauth2-proxy-auth-failures.md

@@ -42,12 +42,14 @@ signature of the session being **re-established over and over instead of held**
 another proxy is overwriting, or a cookie that never comes back at all — not of Keycloak being
 down and not of the app being broken.
 
-**Read this before you trust any silence:** the per-app gate sidecars export **no metrics**. They
+**Read this before you trust any silence (updated 2026-10-01).** The five fleet gate sidecars now
-run without `--metrics-address`, so oauth2-proxy's metrics listener never starts; they declare no
+*do* run a metrics listener on `:44180`, and `platform/oauth2-proxy/podmonitor.yaml` discovers
-`metrics` port; and no ServiceMonitor or PodMonitor selects them. Only the shared
+them — but **the scrape is dropped by the app namespaces' `default-deny` NetworkPolicy**, so those
-`oauth2-proxy` namespace instance is scraped, and it is not in the data path for any of these
+targets read `up == 0` and the TSDB holds nothing for them. Crafty has no listener at all. Only
-apps. So there is nothing to graph and nothing that could have alerted. The logs below are not a
+the shared `oauth2-proxy` namespace instance is genuinely scraped, and it is not in the data path
-convenience — until that changes they are the **only** record that this class of failure happened.
+for any of these apps. So there is still nothing to graph and nothing that could have alerted.
+**The logs below remain the only record that this class of failure happened.** Check which state
+you are in before you trust a quiet dashboard — "First commands" step 1 tells you in one query.
 
 ## First commands
 
@@ -60,9 +62,49 @@ with an explicit context.
 infra/devcontainer/dev.sh run bash -c \
   'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
      -c prometheus -- /bin/promtool query instant http://localhost:9090 \
-     "count by (namespace) (kube_pod_container_info{container=\"oauth2-proxy\"}) unless count by (namespace) (up{container=\"oauth2-proxy\"})"'
+     "count by (namespace) (kube_pod_container_info{container=\"oauth2-proxy\"}) unless count by (namespace) (up{container=\"oauth2-proxy\"} == 1)"'
 ```
 
+**1b. If a namespace is listed, is it "no target" or "target failing"?** The two need different
+fixes and the alert deliberately does not distinguish them:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
+     -c prometheus -- /bin/promtool query instant http://localhost:9090 \
+     "up{container=\"oauth2-proxy\"}"'
+```
+
+No series for the namespace → nothing discovers it (no PodMonitor match; check the pod carries
+`homelab.keithjackson.dev/auth-mode=forward-auth` and a named `metrics` port). A series with
+value **0** → discovered but every scrape fails; get the reason from the target's `lastError`:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
+   curl -s localhost:19090/api/v1/targets | python3 -c "
+import json,sys
+for t in json.load(sys.stdin)[\"data\"][\"activeTargets\"]:
+    if t[\"labels\"].get(\"container\")==\"oauth2-proxy\":
+        print(t[\"labels\"].get(\"namespace\"), t[\"health\"], t.get(\"lastError\",\"\")[:90])"'
+```
+
+`context deadline exceeded` is a **drop**, not a refusal — that is NetworkPolicy, not a dead
+endpoint. Confirm the endpoint itself is fine by port-forwarding straight to the pod, which
+bypasses policy entirely:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n jackett port-forward \
+     $(kubectl --context k3s-prod -n jackett get pod -l app.kubernetes.io/name=jackett -o name | head -1) \
+     14418:44180 & sleep 5; curl -s -o /dev/null -w "http=%{http_code} bytes=%{size_download}\n" \
+     http://127.0.0.1:14418/metrics'
+```
+
+`http=200` with a few KB means the listener is healthy and the path is what is broken. As of
+2026-10-01 that is the state of all five fleet apps, and the fix is an ingress allowance from ns
+`monitoring` to port 44180 — see `platform/longhorn/networkpolicy-metrics.yaml` for the idiom.
+
 **2. The sidecar log — but you must grep it.** The gate's stdout is ~11,500 lines a day per app
 and almost all of it is the kubelet health probe (`GET "/ping" ... 200`). A plain
 `kubectl logs --tail=100` shows you nothing but `/ping` and looks perfectly healthy. Filter:
@@ -105,6 +147,37 @@ ratio in the tens means sessions are not surviving. Measured on prod 2026-09-30
 
 All three of those are the failure, not the baseline.
 
+**The cleanest worked case of all — radarr, measured 2026-10-01.** After the cookie fix landed,
+jackett dropped to **0** `No valid authentication` lines in 13,612 (from 39), and `removing
+session` / `cookie signature not valid` went to **0 on both apps** (radarr had 8 and 8 before).
+The eviction fingerprint is gone. But radarr still logged **206** rejections — and sampling 209 of
+them, **every single one was the same request**:
+
+    GET "/signalr/messages/negotiate" ... 302    <- browser UA, over and over
+
+That is one left-open browser tab. Its cookie is gone, its SignalR socket keeps trying to
+reconnect, and every attempt is silently 302'd to Keycloak. Nobody is at the keyboard, so nobody
+is ever re-prompted; the tab just sits there looking like a working page that has stopped
+updating. **This is the entire failure class in one line**: a non-navigation request answered with
+a redirect, forever, with no error surfaced anywhere.
+
+So when you read a rejection count, **always group by request path before concluding anything**:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n radarr logs deploy/radarr -c oauth2-proxy --since=24h \
+     | grep -v "/ping" | grep "No valid authentication" -A 1 \
+     | grep -oE "\"[^\"]+\" HTTP" | sort | uniq -c | sort -rn | head'
+```
+
+One path dominating means an orphaned client, not a broken gate — annoying but harmless, and it
+clears when the tab is closed. Many different paths from one session means the session itself is
+not surviving, which is the real defect. A per-app `--api-route` is the structural fix for the
+first shape: it makes those paths answer **401** instead of 302, so the client fails loudly
+instead of looping. Note that a count of **0 `AuthSuccess`** alongside a high rejection count
+means nobody has signed in since the last restart — so `--cookie-refresh` has never run, and no
+session-lifecycle conclusion can be drawn from that window at all.
+
 **4. Did the gate admit the user, or did the app refuse them?** Look for the `[AuthSuccess]` line
 and what follows it. If `[AuthSuccess]` appears and the very next thing is the *app's own* login
 page or the app's 401, the gate did its job and layer 3 is your problem — that is the Crafty