docs(obs): record why CI is red — approval-policy on the one conformance entry

The gate job reported failure while the local run was rc 3 with nothing failed. Measured cause is a single sensitive path (the KNOWN_EXTERNAL_MONITORS entry) with no non-author approving review, which ci/approval-policy.sh blocks fail-closed by design. Also records the general trap: approval-policy short-circuits to N/A off a pull_request event, so a clean LOCAL gate is a false reassurance for any branch touching a sensitive path. Only the PR run tells the truth. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
feat(obs): wire the gate-sidecar scrape, and fix the alert that would have gone silent doing it
2026-10-01 22:12:19 -05:00 · 2026-10-01 21:56:07 -05:00 · 2026-10-01 21:40:37 -05:00 · 2026-10-02 02:17:48 +00:00 · 2026-10-02 02:17:23 +00:00 · 2026-09-30 10:55:19 -05:00
10 changed files with 1095 additions and 0 deletions
@@ -129,6 +129,22 @@ KNOWN_EXTERNAL_MONITORS: dict[str, str] = {
        "a PodMonitor, so it resolves against POD ports, not Service ports — the Flux controller "
        "Deployments (and their `http-prom` container port) are installed by the Flux bootstrap "
        "manifests, not authored here. Live on prod: job flux-system/flux-system, 6 targets up.",
    "platform/oauth2-proxy/podmonitor.yaml:oauth2-proxy-sidecars":
        "a PodMonitor, so it resolves against POD ports, not Service ports. Unlike the other "
        "entries here the target is NOT chart-owned — the `metrics` (44180) container port is "
        "authored in this repo, on the oauth2-proxy sidecar in apps/wave-0/{jackett,qbittorrent,"
        "pihole}.yaml and apps/wave-1/{radarr,sonarr}.yaml. It is unauditable here because NO "
        "Service exposes 44180 BY DESIGN: adding one would mean changing catalog/render.py and "
        "every apps/wave-* render and golden, to publish an endpoint only Prometheus should dial. "
        "A PodMonitor needs no Service, which is the whole reason it was chosen. What assertion "
        "(2) would have checked was instead MEASURED on prod 2026-10-01, read-only: the pods "
        "declare ports [('http', 4180), ('metrics', 44180)], the selector label "
        "homelab.keithjackson.dev/auth-mode=forward-auth is on exactly those five gate pods, and "
        "port-forwarding 44180 returns http=200 with 10208 bytes of oauth2_proxy_* metrics. "
        "CAVEAT, deliberately recorded: those targets are DOWN on prod, because the app "
        "namespaces' default-deny NetworkPolicy drops the scrape from ns monitoring (same bug and "
        "fingerprint as platform/longhorn/networkpolicy-metrics.yaml documents). The port name "
        "resolves; the network path does not yet.",
 }
 MONITOR_KINDS = ("ServiceMonitor", "PodMonitor")
@@ -0,0 +1,289 @@
 ---
 date: 2026-09-30
 branch: feat/oauth2-proxy-auth-failure-visible
 title: The Jackett auth failure was invisible because the gates emit nothing — alert on the gap, not on a metric that does not exist
 ---
 ## What changed
 Jackett's UI broke in the way that costs the most: **HTTP 200 and a plausible page**. The
 operator's report named the property that matters — *"reloading the page never fixes the issue and
 the user is never re-prompted for login"*. Two sibling branches fix the auth defects. This one is
 about the fact that **no signal existed anywhere**, and it turns out that is not an oversight in
 the alert rules. It is an absence of data.
 **The measurement that decided the whole design** (prod, live Prometheus, 2026-09-30):
    count by (__name__) ({__name__=~"oauth2_proxy.*"})
      oauth2_proxy_requests_in_flight => 2
 Two series. Both from the **shared** forward-auth Deployment in the `oauth2-proxy` namespace.
 That is every oauth2-proxy metric this cluster holds. The six per-app gate sidecars — crafty,
 jackett, pihole, qbittorrent, radarr, sonarr — publish **nothing at all**: they run with
 `--http-address=0.0.0.0:4180` and no `--metrics-address`, so oauth2-proxy's metrics listener never
 starts; they declare only a `http:4180` container port; and no ServiceMonitor or PodMonitor in the
 cluster selects them.
 So the alert the brief asked for — *"a gated app is serving a sustained rate of auth redirects to
 a user who is authenticated"* — **is not writable**, and writing it anyway would have produced
 exactly the artefact this repo keeps getting burned by: a rule that is present, evaluating,
 permanently silent, and indistinguishable from health. It was not written.
 What was written instead alerts on the condition that made the incident invisible, from families
 that **are** scraped:
 - **`platform/oauth2-proxy/auth-visibility-alerts.yaml`** — `OAuth2ProxyGateUnobserved`:
      count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
        unless
      count by (namespace) (up{container="oauth2-proxy"})
  `for: 1h`, `severity: warning`. `kube_pod_container_info` is kube-state-metrics (a declared
  owner in `alert-metrics-have-scrape-targets`) and sees the container whether or not it exports
  anything; `up` is UNIVERSAL and carries a `container` label for any scrape target, so the rule
  needs no advance knowledge of what job name a future per-app scrape will use. Nothing in it
  touches `oauth2_proxy_*`, so it cannot be inert for the reason it is reporting. **It returns six
  namespaces against live prod right now** and excludes `oauth2-proxy`, which is scraped. It
  clears per-namespace as each gate gains a real scrape.
 - **`platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml`** — four states pinned:
  fires for an unscraped namespace, never for the scraped one, honours `for: 1h` (pending at 59m),
  and **stops** once a scrape appears under a *different* job name. Series shapes copied off live
  prod, not invented.
 - **`runbooks/oauth2-proxy-auth-failures.md`** — how to tell "the gate refused me" from "the group
  check refused me" from "the app refused me" from "the XHR silently got HTML", with the
  2026-09-30 jackett trace as the worked example. Registered in `runbooks/kustomization.yaml`'s
  `configMapGenerator` and listed in `runbooks/README.md`.
 - **`runbooks/README.md`** — a new row in *"Alerts you should not trust the silence of"* for the
  per-app sidecars, plus the measured reason.
 - **`platform/oauth2-proxy/README.md`** — what this component can and cannot see, and the exact
  three-part change that would close the gap.
 **Why the obvious alternative was rejected.** An alert on the *shared* proxy's
 `oauth2_proxy_requests_total{code="302"}` ratio is the one a reader expects to see here. It would
 need a new `SCRAPE_OWNERS` entry in `conformance/**` (a sensitive path needing human approval),
 and — decisively — **it would not have fired on this incident**: the shared instance is not in the
 data path for any sidecar-gated app, and its current pods export no `oauth2_proxy_requests_total`
 at all, having proxied nothing since they started (the counter exists in the TSDB only from an
 older ReplicaSet, `oauth2-proxy-78449b5657-*`, with `code` labels `302` and `401` and no `202`).
 Shipping it would have looked like coverage of the reported failure and been none.
 ## What I verified
 Everything below was measured against live prod through the pinned container, read-only. No
 `apply`, no `edit`, no `patch`, no `flux reconcile`.
 - **The full oauth2-proxy metric inventory.** `count by (__name__) ({__name__=~"oauth2_proxy.*"})`
  → `oauth2_proxy_requests_in_flight => 2`. `count by (__name__) ({job="oauth2-proxy"})` returns
  go_*/process_*/promhttp_* plus that one family and nothing else. **Only the shared instance is
  scraped**: `up{job="oauth2-proxy"}` → 2 targets, `10.0.1.189:44180` and `10.0.2.47:44180`, both
  `1`, `namespace="oauth2-proxy"`.
 - **The per-app sidecars are unscraped, and unscrapable as configured.** Every pod carrying an
  `oauth2-proxy` container, with its declared ports: the two shared replicas have
  `[('http',4180),('metrics',44180)]`; crafty, jackett, pihole, qbittorrent, radarr and sonarr all
  have `[('http',4180)]` only. The live jackett sidecar's args contain no `--metrics-address`.
  `kubectl get servicemonitors,podmonitors -A` lists 16 ServiceMonitors and 5 PodMonitors; the
  only oauth2-proxy entry is `oauth2-proxy/oauth2-proxy`.
 - **No existing alert is inert *for this reason*, because no alert references these metrics at
  all.** `grep -rn oauth2_proxy --include=*.yaml` over the tree hits only cookie-name strings in
  `platform/crafty/`. The 39 live PrometheusRules include `platform-health-identity`, whose four
  rules are about Keycloak readiness and the client reconciler and are all keyed on `kube_*`
  (scraped, fine). **The inertness here is the absence of a rule, not a broken one** — which is
  why the runbook README's inert-alerts table gains a row phrased like the etcd one.
 - **Keycloak cannot express it either.** `count by (__name__) ({__name__=~"keycloak_.*"})` →
  empty, and empty over `[30d]`. The `keycloak-service` scrape yields JVM/Agroal/JGroups metrics
  only; there are no user-event or login counters, so "a burst of re-logins for one client" is not
  observable from the IdP side.
 - **`oauth2_proxy_requests_total` has existed historically but not on the current pods.**
  `count_over_time(oauth2_proxy_requests_total[30d])` returns three series, all from
  `oauth2-proxy-78449b5657-*` (a prior ReplicaSet): `code="302"` on both replicas and `code="401"`
  on one, ~634 samples each (~5h of a 30d window). **No `code="202"` ever.**
 - **The alert expression fires against live data.** Run verbatim on prod:
  `count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by
  (namespace) (up{container="oauth2-proxy"})` → `pihole, jackett, crafty, qbittorrent, sonarr,
  radarr`, value 1 each; `oauth2-proxy` correctly absent.
 - **The log signal exists and is collected.** Loki (`loki-0`, SingleBinary) and a 3-node Alloy
  DaemonSet are running, and Alloy labels pod logs with `namespace`/`container`. LogQL over 24h,
  `{container="oauth2-proxy"}`: crafty 12036 lines, jackett 11669, oauth2-proxy 34559, pihole
  11520, qbittorrent 11520, radarr 11763, sonarr 11520. Filtered to `|= "No valid authentication
  in request"`: **crafty 81, jackett 39, radarr 67**; to `|= "AuthSuccess"`: **crafty 1, jackett
  4, radarr 1**. Those ratios (81:1, 10:1, 67:1) are the incident's signature and are in the
  runbook as the thing to compare against.
 - **Why `kubectl logs` hid it.** The jackett sidecar's stdout is ~11.5k lines/day and almost all
  of it is `GET "/ping" ... 200 2` from the kubelet probe, twice every 15s. A `--tail=40` shows
  forty `/ping` lines and looks perfectly healthy. The runbook's commands all filter.
 - **Runbook registration, both halves.** `- oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md`
  is in `runbooks/kustomization.yaml`'s `configMapGenerator.files`, and the page is listed under
  *Identity* in `runbooks/README.md`. The page clears the L0 quality floor (the four operator
  sections, >1500 bytes, mentions `dev.sh` and `--context k3s-prod`), and the alert's
  `runbook_url` points at exactly that name.
 - **Gate:** run in the pinned toolbox from this worktree with the `ENGINE` shim. Exit code and
  per-stage outcomes in the next section — read them there, not here.
 ## What I did NOT verify
 - **The alert has never fired in Alertmanager.** Nothing is applied; the PrometheusRule exists
  only in git on this branch. What is proven is that its expression returns the expected vector
  against live prod data and that promtool agrees about its behaviour over synthetic series.
  **Merging is what deploys it.**
 - **`OAuth2ProxyGateUnobserved` would NOT have fired *during* the 2026-09-30 jackett incident in
  any useful sense** — it describes a standing condition that has been true since the sidecars
  were deployed, so it would have been firing before, during and after. That is deliberate and it
  is the honest claim: it is a **gap tracker**, not an incident detector. It would have meant that
  on 2026-09-30 an operator asking "why did nothing alert?" had the answer already on the board
  instead of discovering it during the investigation. **No alert on this platform could have
  detected the incident itself, and none can today.**
 - **The alert that the brief actually asked for does not exist, and cannot be written yet.** The
  sustained-redirect-ratio rule needs `oauth2_proxy_requests_total` from the *per-app* sidecars.
  **Enabling that requires editing files owned by the two sibling branches** — the sidecar args in
  `apps/wave-0/jackett.yaml` (Mira Vance) and `apps/wave-0/{qbittorrent,pihole}.yaml`,
  `apps/wave-1/{radarr,sonarr}.yaml` (Theo Bright) — so it was **not attempted here**. The change
  is three things per app, together: `--metrics-address=0.0.0.0:44180`, a `metrics` containerPort
  plus Service port, and a ServiceMonitor. Once any one app has them, the rule belongs beside
  `OAuth2ProxyGateUnobserved`, and that app's namespace drops out of it automatically.
 - **The Loki alerting path is inert and I did not fix it.** Loki *has* a ruler stanza and the
  chart's `loki-sc-rules` sidecar is running with `LABEL=loki_rule`, `FOLDER=/rules` — so a
  ConfigMap-shipped LogQL rule would be delivered. But `GET /loki/api/v1/rules` answers **"unable
  to read rule dir /var/loki/rules/fake: no such file or directory"** (the ruler reads a different
  directory from the one the sidecar writes), and the rendered config contains **no
  `alertmanager_url`** — `grep -i 'alertmanager\|rule_path'` over the `loki` ConfigMap returns
  nothing. So a LogQL alert on `|= "No valid authentication in request"` — which *would* have
  caught this incident, and is the single highest-value follow-up here — **would evaluate and
  notify nobody**. Wiring it means `rulerConfig` (alertmanager_url + a matching rules directory)
  in `platform/monitoring/loki.yaml`, which is not a file this brief gave me. **I did not ship an
  inert rule ConfigMap to look productive.**
 - **Nothing was reproduced in a browser.** The 302-to-200-with-HTML mechanism is read off the
  sidecar access log and the response sizes (`/UI/Dashboard 200 44379`), not observed in devtools.
 - **Loki retention is 72h** (`platform/monitoring/loki.yaml`), so the 24h ratios above are a
  snapshot; anything older than three days is already gone. This is stated in the runbook because
  it bounds what an operator can still investigate.
 - **The gate's L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), so no runtime
  requirement touched by this branch is certified. See the exit code below — **rc 3 is
  UNDETERMINED, not green.**
 ### Follow-ups, in order of value
 1. **Wire the Loki ruler to Alertmanager** (`platform/monitoring/loki.yaml`). This is the only
   route to an alert that would have caught the actual 2026-09-30 failure, and the data is
   already being collected. The rule is one expression:
   `sum by (namespace) (rate({container="oauth2-proxy"} |= "No valid authentication in request" [15m]))
   > 5 * sum by (namespace) (rate({container="oauth2-proxy"} |= "AuthSuccess" [15m]))`.
 2. **Give one gate sidecar a metrics endpoint** and watch its namespace drop out of
   `OAuth2ProxyGateUnobserved`. That is the cheapest proof the alert is a tracker and not a nag,
   and it unlocks the ratio rule in Prometheus.
 3. **Consider whether `--set-xauthrequest` apps should get `--api-routes`** so API paths answer
   401 instead of 302. That is the fix that would make this failure *loud* rather than merely
   observable, and it belongs with whoever owns the sidecar args.
 ---
 # Round 2 — 2026-10-01: the scrape got wired, and found a bug in this branch's own alert
 Added to the same PR (#29) rather than a fourth concurrent branch. PR #28 merged in the meantime
 and `main` moved three PRs ahead; `origin/main` is merged in here, and the `runbooks/kustomization.yaml`
 conflict was resolved by **keeping both** runbook entries (mine and `oauth2-proxy-session`).
 ## What changed
 **`platform/oauth2-proxy/podmonitor.yaml` (new).** PR #28 gave the five fleet sidecars
 `--metrics-address=0.0.0.0:44180` and a named `metrics` port. No Service exposes 44180 on any of
 them, so a ServiceMonitor would have nothing to select — a **PodMonitor** targets pod ports directly
 and needs no renderer change. Selects on the pod label `homelab.keithjackson.dev/auth-mode:
 forward-auth`, which is on exactly the five fleet gate pods and absent from crafty's: the property,
 not an app list, so a sixth forward-auth app needs no edit.
 **`auth-visibility-alerts.yaml`: `up{container="oauth2-proxy"}` → `up{container="oauth2-proxy"} == 1`.
 This is a real bug fix, and it is the most important thing in this round.** A target that is
 discovered but failing still publishes `up`, with value **0**. The series exists, so
 `count by (namespace)` returned 1, `unless` removed the namespace, and `OAuth2ProxyGateUnobserved`
 would have **gone quiet while nothing was being scraped at all** — this branch's own disease,
 reproduced inside the alert written to cure it. Not hypothetical: the PodMonitor lands five targets
 whose scrape is dropped, so the bare form would have silently cleared on this very commit.
 **Two new unit-test scenarios** pinning that: a discovered-but-failing target must still fire, and a
 target that works and then breaks must re-fire rather than latch cleared.
 **Docs corrected rather than appended to.** The 2026-09-30 claim "the sidecars export no metrics at
 all" is now only half true and was rewritten in `runbooks/README.md`, `platform/oauth2-proxy/README.md`
 and the alert header, each showing the two-step history, because each step looked like the end of it.
 The runbook gains the policy-vs-no-target triage (step 1b) and the **radarr `/signalr` worked case**.
 **Deliberately NOT shipped: the ratio alert.** `oauth2_proxy_requests_total` now exists on the
 endpoint, but not in the TSDB, because the scrape is blocked. Writing the ratio rule today would make
 it inert for precisely the reason it was refused on 2026-09-30. It becomes writable the day a scrape
 succeeds, not the day a listener exists.
 ## What I verified
 - **The endpoint serves — by payload, not by flag.** Port-forwarded to the jackett pod's 44180:
  `http=200 bytes=10208`, exporting `oauth2_proxy_requests_in_flight`, `oauth2_proxy_requests_total`
  (`code="200" 3`, `code="302" 2`) and `oauth2_proxy_response_duration_seconds`. `--metrics-address`
  being in `args` and `/metrics` returning a body are different claims; both now hold.
 - **The scrape path is DROPPED, measured from the right place.** From grafana in ns `monitoring`
  (beside Prometheus, same policy position) to the jackett pod IP: `:44180` → `wget: download timed
  out`; `:4180` → **also** timed out; the shared proxy's `:44180` → 200 with `oauth2_proxy_*` present.
  Cause, read from the objects: ns jackett's `default-deny` is `podSelector: {}`,
  `policyTypes: [Ingress, Egress]`, with an `egress` rule and **no `ingress` key** (deny-all ingress);
  the only allowance is CNP `jackett-allow-gateway-ingress`, `fromEntities: [ingress]` to port 4180.
  The shared proxy is scrapable only because ns `oauth2-proxy` has **no NetworkPolicy at all**.
 - **No Service exposes 44180** on any of the five (enumerated).
 - **The pod label selector is exact:** `auth-mode=forward-auth` on all five fleet gate pods, absent
  from crafty's. Crafty's sidecar still has no `--metrics-address` (args read in full).
 - **Flux ordering forbids shipping the fix from here:** `oauth2-proxy dependsOn=['platform-config']`
  and `apps-wave-0 dependsOn=['platform-config']` — siblings, no ordering. A NetworkPolicy naming ns
  `jackett` from this tier would apply today and fail on a cold rebuild.
 - **The `== 1` guard is non-vacuous — proven by mutation.** Reverting it to the bare `up` makes the
  two new scenarios FAIL (`got: nil` at 1h5m and 2h5m) and leaves every other assertion passing;
  restoring it returns the suite to SUCCESS under pinned promtool v3.14.0.
 - **Gate:** see the next section for the exit code and skips.
 ## What I did NOT verify
 - **The PodMonitor has never been applied.** That it discovers the five pods is reasoned from the
  measured pod labels and port name, not observed — no target list has been read with it in place,
  because applying it is an operator action. **Expect `up == 0` on all five when it lands**, which is
  the documented state, not a regression.
 - **I did not fix the NetworkPolicy, and this PR therefore does not make the gates observable.** It
  makes them *discovered*, and makes the remaining gap loud instead of silent. The one narrow rule
  per app (ns monitoring → `auth-mode=forward-auth` → TCP 44180, per
  `platform/longhorn/networkpolicy-metrics.yaml`) has to be rendered with each app's own policy, in
  the catalog/renderer that is in flight on another branch. **It needs an owner.**
 - **Whether Prometheus will actually select a PodMonitor in ns `oauth2-proxy` targeting other
  namespaces.** `namespaceSelector.any: true` plus `podMonitorSelectorNilUsesHelmValues: false` and
  the fact that cross-namespace ServiceMonitors/PrometheusRules already work here say yes; it is not
  measured with this object.
 - **No session-lifecycle claim can be made from the current window.** `AuthSuccess` is **0** on both
  jackett and radarr since the PR #28 restart, so nobody has signed in and `--cookie-refresh=2m30s`
  has never executed a refresh. The cookie fix is unproven in the one dimension that matters most.
 - **The radarr `/signalr` finding is the coordinator's measurement, not mine** — I reproduced neither
  the 206/209 sample nor the jackett 0-in-13,612. They are recorded as reported.
 - **L2/L3 conformance still did not run** (`CLUSTER_AVAILABLE` unset). Nothing is certified.
 ### Round 2 addendum — CI is RED, by design, and the local gate cannot see why
 After pushing round 2, `gate / gate (pull_request)` reported **failure after 1m42s** while the local
 toolbox run was rc 3 with nothing failed. Measured cause, singular:
    sensitive paths touched:
      conformance/tests/l0-static/servicemonitor-ports-resolve/check.sh   (matched conformance/**)
    approving reviews: (no reviews at all)        author=gitea-admin
 `ci/approval-policy.sh` is fail-closed and exits 1 when a PR touches a `sensitive_paths` glob with no
 `APPROVED` review from a non-author — "a change to the gate cannot be gated by the gate". The single
 `KNOWN_EXTERNAL_MONITORS` entry is what trips it. **Nothing in L0/L1 failed.**
 **The local gate structurally cannot catch this, and said `approval-policy: green (N/A)`.** The stage
 short-circuits to N/A when `GITHUB_EVENT_NAME` is unset, which it always is off-CI. So for any branch
 touching a sensitive path, a clean local gate is a FALSE REASSURANCE — the PR run is the only honest
 signal. That is a general trap, not specific to this change, and it is worth knowing before someone
 concludes a red PR means broken code.
 To clear: a non-author approving review, then **re-run the gate** — approval does not re-trigger the
 status check, so the required context stays red until the job runs again. `required_approvals=1`
 already applied to every PR here, so the human was needed regardless; the conformance entry only
 turns the tick red while that is pending.
 The alternative is to drop the register entry, which means dropping `podmonitor.yaml` with it (the
 check fails without the declaration). I did not take it: the entry is the "declare the limit"
 mechanism the check explicitly asks for, it is shrink-only so it cannot rot quietly, and
 `flux-podmonitor.yaml` is the same case already in the register. But it is a real trade and the
 reviewer should make it knowingly.
@@ -88,3 +88,72 @@ This proves the R-IDP-02 property (unauthenticated cannot reach the upstream) on
 3. Log in as a **non-platform-admins** user -> expect **403** (authenticated but not authorized).
 4. Confirm the session cookie is `.home.keithjackson.dev`-scoped (single sign-on across the other
   protected UIs).
 ## Observability: what this component can and cannot see (measured 2026-09-30)
 `servicemonitor.yaml` scrapes **this namespace's** deployment and nothing else. On prod that is
 two targets, `job="oauth2-proxy"`, both `up=1`. The complete set of oauth2-proxy-specific series
 in the TSDB is:
    count by (__name__) ({__name__=~"oauth2_proxy.*"})
      oauth2_proxy_requests_in_flight => 2
 **The per-app gate sidecars are not scraped, and there is nothing to scrape.** Six apps run an
 `oauth2-proxy` container of their own — crafty, jackett, pihole, qbittorrent, radarr, sonarr —
 each with `--http-address=0.0.0.0:4180` and **no `--metrics-address`**, so oauth2-proxy's metrics
 listener never starts. They declare only a `http:4180` container port, no `metrics` port exists on
 their Services, and `kubectl get servicemonitors,podmonitors -A` selects none of them. Any alert
 written against `oauth2_proxy_requests_total{namespace="jackett"}` would be structurally inert.
 This matters because the gate's characteristic failure is invisible by construction: an
 unauthenticated **XHR** gets a `302` to Keycloak rather than a `401`, the browser follows it
 against a live SSO session with no prompt, and the call resolves **200 carrying an HTML page**
 where the app expected JSON. Nothing in the request path reports an error. On 2026-09-30 jackett
 spent a burst of requests in exactly that state, re-established a session, and lost it 26 seconds
 into a 5-minute session, while `jackett-down` sat at 1 replica available the whole time.
 `auth-visibility-alerts.yaml` therefore alerts on **the gap**, not on the incident:
 `OAuth2ProxyGateUnobserved` compares `kube_pod_container_info{container="oauth2-proxy"}` against
 `up{container="oauth2-proxy"}` — two already-scraped families — and names every namespace running
 a gate that nothing observes. It fires for six namespaces today and clears per-namespace as each
 one gains a scrape. `runbooks/oauth2-proxy-auth-failures.md` is the by-hand substitute meanwhile,
 and it reads the sidecar logs out of Loki (Alloy collects them; 72h retention).
 **To close the gap for an app**, three things are needed together, on that app's own manifest:
 `--metrics-address=0.0.0.0:44180` on the sidecar, a `metrics` containerPort plus the matching
 Service port, and a ServiceMonitor selecting it. Only then does the auth-churn rule — sustained
 `code="302"` as a share of `oauth2_proxy_requests_total`, held across a window longer than the
 session lifetime — become writable rather than decorative.
 ### Update 2026-10-01: the listener exists; the scrape path does not
 PR #28 added `--metrics-address=0.0.0.0:44180` and a named `metrics` container port to the five
 fleet sidecars (jackett, radarr, sonarr, qbittorrent, pihole). `podmonitor.yaml` in this directory
 discovers them by the pod label `homelab.keithjackson.dev/auth-mode: forward-auth` — a **PodMonitor**
 rather than a ServiceMonitor because no Service exposes 44180 and adding one would mean changing the
 renderer.
 The endpoint is real: port-forwarded straight to the jackett pod, `/metrics` returns **http=200,
 10208 bytes**, exporting `oauth2_proxy_requests_in_flight`, `oauth2_proxy_requests_total` (with
 `code` labels) and `oauth2_proxy_response_duration_seconds`. That is verified by payload, not by the
 flag appearing in `args` — a distinction worth keeping, because the two claims came apart here.
 **The scrape is nevertheless dropped.** From a pod in ns `monitoring`, both `:44180` and `:4180` on
 the jackett pod time out; the shared proxy's `:44180` answers. The only difference is that ns
 `oauth2-proxy` has **no NetworkPolicy at all**, while each app namespace has a `default-deny` whose
 `policyTypes` include `Ingress` with no `ingress` rules (deny-all) plus a single CNP admitting
 `fromEntities: [ingress]` to port 4180 — the Gateway's Envoy and nothing else. Prometheus matches no
 clause.
 So the targets read `up == 0` and the TSDB is still empty for those namespaces. This is the same bug
 and the same fingerprint as `platform/longhorn/networkpolicy-metrics.yaml` documents for Longhorn,
 and the same narrow fix applies: ns `monitoring` → the gate pod label → TCP 44180 only. It cannot
 ship from this directory, because `oauth2-proxy` and `apps-wave-0` are **siblings** in the Flux graph
 (both `dependsOn: [platform-config]`), so a NetworkPolicy naming ns `jackett` would apply on today's
 cluster and fail on a cold rebuild. It belongs with each app's own rendered policy.
 Consequence for `auth-visibility-alerts.yaml`: its `unless` clause requires `up == 1`, not merely a
 present `up` series. A discovered-but-failing target publishes `up` with value 0, so the bare form
 would have cleared `OAuth2ProxyGateUnobserved` the moment the PodMonitor landed — reporting the gates
 as observed while nothing was being scraped. The unit test pins that scenario; deleting the `== 1`
 fails it and nothing else.
@@ -0,0 +1,177 @@
 ---
 # promtool unit tests for OAuth2ProxyGateUnobserved (platform/oauth2-proxy/auth-visibility-alerts.yaml).
 #
 # WHY. The rule is an `unless` between two aggregations over DIFFERENT metric families, and the
 # failure modes of that shape are both silent: a label that does not survive `count by` makes the
 # right-hand side match nothing (the alert fires for the scraped namespace too, i.e. permanently),
 # and a right-hand side that over-matches makes it fire for nobody (i.e. inert). Neither is
 # visible by reading the expression, and neither is visible on a live cluster where the answer
 # happens to look plausible. So all four states are pinned: fires for an unscraped namespace,
 # stays quiet for a scraped one, honours `for: 1h`, and STOPS once a scrape appears.
 #
 # HOW TO RUN. The one repo-wide harness, by the companion convention:
 #   ./infra/devcontainer/dev.sh run bash -c 'platform/monitoring/tests/run.sh platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml'
 # promtool is pinned in infra/devcontainer/versions.env; never run it off the host PATH.
 #
 # THE SERIES BELOW ARE REAL SHAPES read off live prod Prometheus on 2026-09-30 before being
 # copied here — kube_pod_container_info as kube-state-metrics emits it (job="kube-state-metrics",
 # its own instance/pod labels, image/image_id/container_id) and `up` as the oauth2-proxy
 # ServiceMonitor emits it (endpoint="metrics", port 44180). The point of using the real shapes is
 # that `count by (namespace)` has to discard all of them; a test with tidy two-label series would
 # pass while the real expression collapsed differently.
 #
 # ASSERTIONS USE `ALERTS` RATHER THAN exp_alerts: the description annotation is long prose and
 # comparing it verbatim would couple this file to every wording change while proving nothing the
 # ALERTS series does not already prove.
 rule_files:
  - auth-visibility-alerts.rules.yaml
 evaluation_interval: 1m
 tests:
  # ---------------------------------------------------------------------------------------------
  # 1. THE STATE PROD IS ACTUALLY IN (2026-09-30): jackett runs a gate sidecar with no scrape;
  #    the shared oauth2-proxy namespace runs two that ARE scraped. Asserts the alert fires for
  #    jackett and NOT for oauth2-proxy, and that `for: 1h` is honoured (pending at 59m).
  - interval: 1m
    name: an unscraped gate namespace fires after 1h; the scraped one never does
    input_series:
      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
        values: '1x130'
      # The shared instance: kube-state-metrics sees both replicas...
      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://aa11bb22",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000001"}'
        values: '1x130'
      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://cc33dd44",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000002"}'
        values: '1x130'
      # ...and both replicas are genuinely scraped, so `up` carries container="oauth2-proxy" there.
      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.189:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="oauth2-proxy"}'
        values: '1x130'
      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.2.47:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="oauth2-proxy"}'
        values: '1x130'
    promql_expr_test:
      # Not yet firing at 59m — `for: 1h` has not elapsed. Guards against someone "simplifying"
      # the hold away and turning every rollout into a page.
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
        eval_time: 59m
        exp_samples: []
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett"}'
        eval_time: 59m
        exp_samples:
          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett",severity="warning"}'
            value: 1
      # Firing for jackett past the hold...
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
        eval_time: 65m
        exp_samples:
          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
            value: 1
      # ...and never for the namespace that IS scraped. This is the assertion that catches an
      # `unless` whose right-hand side stopped matching: without it, a broken rule that fires for
      # every gate namespace would pass the test above.
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="oauth2-proxy"}'
        eval_time: 65m
        exp_samples: []
      # The count must be 1 per namespace-with-a-gate, not 1 overall: a `count by (namespace)`
      # that lost its `by` would collapse to a single unlabelled series.
      - expr: 'count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by (namespace) (up{container="oauth2-proxy"})'
        eval_time: 65m
        exp_samples:
          - labels: '{namespace="jackett"}'
            value: 1
  # ---------------------------------------------------------------------------------------------
  # 2. THE FIX LANDING. jackett's sidecar gains --metrics-address and a ServiceMonitor 70m in, so
  #    `up` starts carrying container="oauth2-proxy" for namespace="jackett" under a DIFFERENT job
  #    name than the shared proxy's. The alert must stop. This is what makes the rule a gap
  #    tracker rather than a permanent nag, and it pins that the match is on `container`, not on
  #    a hardcoded job="oauth2-proxy" (which would keep firing forever after a correct fix).
  - interval: 1m
    name: the alert clears once the sidecar is actually scraped
    input_series:
      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
        values: '1x130'
      # Absent for the first 70 minutes, then present. `_x` is "no sample"; the scrape did not
      # exist yet.
      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.213:44180",job="jackett-auth",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="jackett-auth-metrics"}'
        values: '_x70 1x60'
    promql_expr_test:
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett"}'
        eval_time: 65m
        exp_samples:
          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
            value: 1
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="jackett"}'
        eval_time: 80m
        exp_samples: []
  # ---------------------------------------------------------------------------------------------
  # 3. THE REGRESSION GUARD FOR `== 1` — the scenario that exposed a real bug on 2026-10-01.
  #
  #    A target is DISCOVERED but every scrape FAILS: `up` exists with value 0. This is not
  #    hypothetical — it is the exact state podmonitor.yaml creates today, because the app
  #    namespaces' default-deny drops the scrape (measured; see podmonitor.yaml).
  #
  #    With the original `unless count by (namespace) (up{container="oauth2-proxy"})`, the `up`
  #    SERIES EXISTS, so `count by (namespace)` returns 1, `unless` removes the namespace, and the
  #    alert GOES QUIET while nothing is being scraped at all — a green signal standing in for an
  #    absent one, which is the precise disease this whole rule was written to cure. Deleting the
  #    `== 1` makes THIS test fail and every other test in this file still pass, which is the
  #    property that makes it worth having.
  - interval: 1m
    name: a discovered-but-FAILING target does not count as observed (up==0 must still fire)
    input_series:
      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-6877fcd9f5-ht9d9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
        values: '1x130'
      # The PodMonitor target exists and never succeeds: NetworkPolicy drops the packets, so the
      # scrape times out and `up` is published as 0 rather than going absent.
      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.0.189:44180",job="oauth2-proxy-sidecars",namespace="jackett",pod="jackett-6877fcd9f5-ht9d9"}'
        values: '0x130'
    promql_expr_test:
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett"}'
        eval_time: 65m
        exp_samples:
          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
            value: 1
      # And the inner clause itself yields NOTHING while the scrape is failing — which is the
      # mechanism the alert depends on, asserted directly so a future reader can see why the
      # `== 1` matters without re-deriving it. (Scenario 4 below covers the other direction:
      # a succeeding scrape really does clear the alert, so `== 1` has not made the rule
      # impossible to satisfy.)
      - expr: 'count by (namespace) (up{container="oauth2-proxy"} == 1)'
        eval_time: 65m
        exp_samples: []
  # ---------------------------------------------------------------------------------------------
  # 4. A FLAPPING target: up goes 1 -> 0 and stays there. The alert must come BACK, not latch
  #    cleared. Covers the ordering bug where a once-observed namespace is permanently excluded.
  - interval: 1m
    name: a target that starts working and then fails re-fires
    input_series:
      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="radarr",pod="radarr-7d55cbffb9-42hmh",service="kube-prometheus-stack-kube-state-metrics",uid="3333cccc-0000-0000-0000-000000000001"}'
        values: '1x250'
      # Healthy for the first 60m, then every scrape fails for the rest.
      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.77:44180",job="oauth2-proxy-sidecars",namespace="radarr",pod="radarr-7d55cbffb9-42hmh"}'
        values: '1x60 0x180'
    promql_expr_test:
      # Quiet while it was working.
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="radarr"}'
        eval_time: 55m
        exp_samples: []
      # Firing again an hour after it broke.
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="radarr"}'
        eval_time: 125m
        exp_samples:
          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="radarr",severity="warning"}'
            value: 1
  # ---------------------------------------------------------------------------------------------
  # 5. NO GATE, NO ALERT. A namespace with no oauth2-proxy container must never appear, or the
  #    rule would page for every app on the platform the day someone widened the matcher.
  - interval: 1m
    name: namespaces without a gate sidecar are never reported
    input_series:
      - series: 'kube_pod_container_info{container="plex",container_id="containerd://feed0001",endpoint="http",image="ghcr.io/example/plex:1",image_id="ghcr.io/example/plex@sha256:feed0001",image_spec="ghcr.io/example/plex:1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="plex",pod="plex-0",service="kube-prometheus-stack-kube-state-metrics",uid="2222bbbb-0000-0000-0000-000000000001"}'
        values: '1x130'
    promql_expr_test:
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved"}'
        eval_time: 65m
        exp_samples: []
@@ -0,0 +1,137 @@
 ---
 # oauth2-proxy AUTH-VISIBILITY alerting (R-OBS-04).
 #
 # ============================================================================================
 # WHY THIS FILE EXISTS — THE 2026-09-30 JACKETT FAILURE THAT NOTHING OBSERVED
 # ============================================================================================
 # Jackett's UI stopped working. Reloading never fixed it and the user was never re-prompted to
 # log in. The sidecar's own access log had the whole story:
 #
 #   [oauthproxy.go:1026] No valid authentication in request. Initiating login.
 #   GET "/api/v2.0/indexers/yts/config?_=..."          HTTP/1.1 ... 302 400
 #   GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
 #   ... ten of these at 15:30:53, then a fresh [AuthSuccess] at 15:31:06,
 #   then "No valid authentication in request" again 26 SECONDS LATER, inside the same session.
 #
 # NOTHING ANYWHERE ELSE RECORDED IT. oauth2-proxy answers an unauthenticated XHR with a 302 to
 # Keycloak rather than a 401; the browser follows it silently, the live SSO session satisfies it
 # without a prompt, and the XHR resolves 200 carrying a 44 KB HTML dashboard where the app's
 # JavaScript wanted JSON. From the browser, from the user, and from any uptime check, the service
 # is healthy. `jackett-down` (apps/wave-0/jackett.yaml) watches
 # kube_deployment_status_replicas_available, which was 1 throughout and would have been 1 if the
 # gate had rejected every request ever made.
 #
 # ============================================================================================
 # WHY THIS FILE DOES NOT CONTAIN THE OBVIOUS ALERT — MEASURED, prod, 2026-09-30
 # ============================================================================================
 # The alert you would want is "a gated app is serving a sustained rate of auth redirects to a
 # user who is already authenticated". It is NOT WRITABLE against this cluster's TSDB, and the
 # reason is this file's real subject:
 #
 #   THE PER-APP GATE SIDECARS EXPORT NO METRICS AT ALL.
 #
 #   $ promtool query instant :9090 'count by (__name__) ({__name__=~"oauth2_proxy.*"})'
 #     oauth2_proxy_requests_in_flight => 2          <- the SHARED proxy's two replicas. That is ALL.
 #
 #   PARTLY SUPERSEDED 2026-10-01, and the correction matters more than the original. PR #28 gave
 #   the five FLEET sidecars `--metrics-address=0.0.0.0:44180` and a named `metrics` port, and the
 #   endpoint genuinely serves (verified by payload: http=200, 10208 bytes, all three
 #   `oauth2_proxy_*` families present — not merely by the flag being in `args`). podmonitor.yaml
 #   in this directory now discovers them. But the scrape is still DROPPED by the app namespaces'
 #   default-deny ingress, so the TSDB remains empty for those namespaces; and crafty still has no
 #   metrics listener at all. The rule below is unchanged in purpose and still has work to do.
 #
 #   Every sidecar (crafty, jackett, pihole, qbittorrent, radarr, sonarr) ran with
 #   `--http-address=0.0.0.0:4180` and NO `--metrics-address`, so oauth2-proxy's metrics listener
 #   was never started; the container declared only port `http:4180`, there was no `metrics` port on
 #   any of their Services, and there was no ServiceMonitor or PodMonitor for any of them anywhere
 #   in the cluster (measured: `kubectl get servicemonitors,podmonitors -A`). Six gates, zero
 #   telemetry. An alert written against `oauth2_proxy_requests_total{namespace="jackett"}` would
 #   have been PERMANENTLY INERT — present, evaluating, silent, and indistinguishable from health.
 #   This repo has shipped that mistake before (VeleroNoRecentBackup, the Longhorn rules), which is
 #   why `conformance/tests/l0-static/alert-metrics-have-scrape-targets` exists, and why the
 #   temptation was refused here rather than ratcheted into a register.
 #
 #   The SHARED proxy IS scraped (job="oauth2-proxy", 2 targets, up=1) but it is not in the data
 #   path for any sidecar-gated app — each app gates itself — and its current pods export no
 #   `oauth2_proxy_requests_total` at all, having proxied nothing since they started. An alert on
 #   the shared instance would not have fired on this incident either. Saying so is the point.
 #
 # So this file alerts on the CONDITION THAT MADE THE INCIDENT INVISIBLE, not on the incident.
 # That condition is measurable today, from two already-scraped metric families, and it is the
 # precondition for ever being able to alert on the incident itself. It self-clears the moment a
 # sidecar gains a metrics endpoint and a scrape — at which point the auth-churn rule becomes
 # writable and belongs here beside it. See runbooks/oauth2-proxy-auth-failures.md, which is the
 # by-hand equivalent an operator uses in the meantime.
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
 metadata:
  name: oauth2-proxy-auth-visibility
  namespace: oauth2-proxy
  labels:
    release: kube-prometheus-stack     # kube-prometheus-stack's ruleSelector
    app.kubernetes.io/name: oauth2-proxy
 spec:
  groups:
    - name: oauth2-proxy.auth-visibility
      rules:
        - alert: OAuth2ProxyGateUnobserved
          # A namespace runs an oauth2-proxy gate container, and Prometheus holds no scrape
          # target for that container in that namespace.
          #
          # BOTH SIDES ARE ALREADY-SCRAPED FAMILIES, deliberately:
          #   * kube_pod_container_info  — kube-state-metrics, the declared owner in
          #     alert-metrics-have-scrape-targets. It sees the container whether or not the
          #     container exports anything, which is exactly the property needed here.
          #   * up                       — UNIVERSAL. `up` carries the `container` label for any
          #     ServiceMonitor/PodMonitor target, so this works for a future per-app scrape
          #     without knowing what job name it will pick.
          # Nothing in this expression touches `oauth2_proxy_*`, so the rule cannot itself be
          # inert for the reason it is reporting.
          #
          # MEASURED ON PROD 2026-09-30 — this expression returned six namespaces:
          #   {namespace="crafty"} {namespace="jackett"} {namespace="pihole"}
          #   {namespace="qbittorrent"} {namespace="radarr"} {namespace="sonarr"}
          # and excluded namespace="oauth2-proxy", whose two replicas ARE scraped. It fires on
          # the state the cluster is in; it is not a rule waiting for a hypothetical.
          #
          # `== 1` IS LOAD-BEARING, AND IT IS A BUG FIX (2026-10-01). This clause was originally
          # a bare `up{container="oauth2-proxy"}`, which was WRONG in the one way that matters
          # here: a target that is DISCOVERED BUT FAILING still publishes `up`, with value 0. So
          # the series exists, `count by (namespace)` returns 1, `unless` removes the namespace,
          # and the alert GOES QUIET while nothing whatsoever is being scraped. That is this
          # branch's own disease — a green signal standing in for an absent one — reproduced
          # inside the alert written to cure it. It was not hypothetical: podmonitor.yaml lands
          # five targets whose scrape is DROPPED by NetworkPolicy (measured; see that file), so
          # the bare form would have silently cleared on the very next commit. `== 1` means
          # "observed" requires a scrape that actually succeeds.
          #
          # A failing target is therefore reported by BOTH this rule and kube-prometheus-stack's
          # generic TargetDown. That duplication is deliberate: TargetDown says "a target is
          # down", this says "an authentication gate is unobserved", and during an incident those
          # are different sentences. The overlap is the cheap half of the trade.
          expr: |
            count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
              unless
            count by (namespace) (up{container="oauth2-proxy"} == 1)
          # 1h, not 5m: a rollout briefly has a running sidecar whose target has not been
          # discovered yet, and this condition is a standing architectural gap, never an
          # emergency. Long enough that no deploy can produce it.
          for: 1h
          labels:
            severity: warning
          annotations:
            summary: "oauth2-proxy gate in {{ $labels.namespace }} exports no metrics"
            description: >-
              The {{ $labels.namespace }} namespace runs an oauth2-proxy gate container and
              Prometheus has NO scrape target for it, so every authentication decision that gate
              makes — every redirect to Keycloak, every 403 on a group check, every session that
              silently fails to hold — is invisible to metrics and to alerting. This is not a
              theoretical gap: on 2026-09-30 the jackett gate answered a burst of the app's own
              XHR calls with 302s to Keycloak, re-established a session, and lost it again 26
              seconds later, and the only record anywhere was a line in the sidecar's stdout.
              No alert could have fired, because there is nothing to alert on. Until a scrape
              exists the runbook's LogQL queries are the substitute; the fix is
              `--metrics-address=0.0.0.0:44180` on the sidecar plus a `metrics` port and a
              ServiceMonitor, after which the auth-churn rule becomes writable.
            runbook_url: "https://${BASE_DOMAIN}/runbooks/oauth2-proxy-auth-failures"
@@ -14,3 +14,12 @@ resources:
  - oauth2-proxy-credentials.sops.yaml
  - httproute.yaml       # forward-auth sign-in/callback ingress (was referenced in-place by platform-config)
  - servicemonitor.yaml  # Prometheus scrape (was referenced in-place by platform-config)
  # Alerting on whether the gates are OBSERVABLE at all. Travels with the servicemonitor above
  # for the same reason it does: monitoring.coreos.com CRDs are registered by the `platform`
  # tier, which is Ready before this tier's platform-config dependency completes.
  - auth-visibility-alerts.yaml
  # Discovers the PER-APP gate sidecars' :44180 metrics endpoints by pod label (PR #28 added the
  # listener; no Service exposes it, so this is a PodMonitor, not a ServiceMonitor). The scrape is
  # still denied by the app namespaces' default-deny ingress — read podmonitor.yaml's header
  # before concluding the targets should be up.
  - podmonitor.yaml
@@ -0,0 +1,109 @@
 ---
 # PodMonitor for the PER-APP oauth2-proxy gate sidecars (R-OBS-04).
 #
 # ============================================================================================
 # WHY A PodMonitor AND NOT A ServiceMonitor
 # ============================================================================================
 # PR #28 gave all five fleet sidecars `--metrics-address=0.0.0.0:44180` and a named `metrics`
 # container port. It did NOT add a Service port, and measured on prod 2026-10-01 none of the five
 # Services exposes 44180:
 #
 #   jackett      jackett:     [('port-9117', 9117), ('authproxy', 4180)]
 #   radarr       radarr:      [('port-7878', 7878), ('authproxy', 4180)]
 #   sonarr       sonarr:      [('port-8989', 8989), ('authproxy', 4180)]
 #   qbittorrent  qbittorrent: [('port-8080', 8080), ('authproxy', 4180)]
 #   pihole       pihole:      [('port-80',     80), ('authproxy', 4180)]
 #
 # A ServiceMonitor selects Service PORTS, so it would have nothing to select. A PodMonitor targets
 # pod ports directly and bypasses Services entirely — so this needs no change to the Services, and
 # therefore no change to `catalog/render.py` or the `apps/wave-*` renders and goldens.
 #
 # SELECTOR: the `homelab.keithjackson.dev/auth-mode: forward-auth` POD LABEL, not a namespace list.
 # Measured 2026-10-01: that label is present on exactly the five fleet gate pods (jackett, radarr,
 # sonarr, qbittorrent, pihole) and absent from crafty's, which is a separately-managed deployment
 # (platform/crafty) that still has no `--metrics-address` at all. So the selector describes the
 # PROPERTY being scraped rather than enumerating today's apps, and a sixth forward-auth app is
 # picked up with no edit here. `namespaceSelector.any: true` because those five namespaces are
 # created by the apps-wave-0/1 tiers, which this tier does not and must not depend on.
 #
 # ============================================================================================
 # READ THIS BEFORE BELIEVING THE TARGETS ARE UP: THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY
 # ============================================================================================
 # This PodMonitor is NECESSARY AND NOT SUFFICIENT. The endpoint serves, and the path from ns
 # monitoring is blocked. Both halves measured on prod 2026-10-01, read-only:
 #
 #   THE ENDPOINT IS HEALTHY — port-forward straight to the jackett pod's 44180:
 #     GET /metrics -> http=200 bytes=10208
 #     # TYPE oauth2_proxy_requests_in_flight gauge
 #     # TYPE oauth2_proxy_requests_total counter
 #     # TYPE oauth2_proxy_response_duration_seconds histogram
 #     oauth2_proxy_requests_total{code="200"} 3
 #     oauth2_proxy_requests_total{code="302"} 2
 #
 #   THE SCRAPE PATH IS DROPPED — from a pod in ns monitoring (grafana, which sits beside
 #   Prometheus and so occupies the same policy position) to the jackett pod IP:
 #     http://10.0.0.189:44180/metrics   -> wget: download timed out
 #     http://10.0.0.189:4180/ping       -> wget: download timed out     (even the "allowed" port)
 #     http://<shared-proxy>:44180/metrics -> 200, oauth2_proxy_* present (control: ns oauth2-proxy
 #                                            has NO NetworkPolicy, which is the only reason the
 #                                            shared instance has ever been scrapable)
 #
 #   WHY: ns jackett's `default-deny` NetworkPolicy is `podSelector: {}` with
 #   `policyTypes: [Ingress, Egress]` and an `egress` rule ONLY — no `ingress` key, which in
 #   NetworkPolicy semantics is deny-all-ingress. The only ingress allowance is the CNP
 #   `jackett-allow-gateway-ingress`: `fromEntities: [ingress]` to port 4180 — the Gateway's Envoy
 #   and nothing else. Prometheus matches no clause and its packets are dropped, not refused.
 #
 # SO THE TARGETS BELOW WILL READ `up == 0` UNTIL AN INGRESS ALLOWANCE LANDS. That is a known,
 # measured, documented state — NOT a regression introduced here, and NOT something to silence.
 # It is deliberately left visible: `OAuth2ProxyGateUnobserved` in auth-visibility-alerts.yaml
 # requires `up == 1`, precisely so a discovered-but-failing target does not read as "observed"
 # (see that file's header — this exact trap is why the `== 1` is there).
 #
 # THIS IS THE SAME BUG LONGHORN HAD, and the platform has a documented fix idiom for it:
 # platform/longhorn/networkpolicy-metrics.yaml, whose header records the identical fingerprint
 # measured on 2026-09-13 — ServiceMonitor in place, 3 targets DISCOVERED, `up` 0/0/0, TSDB empty,
 # `lastError: context deadline exceeded` (a drop, not a refusal). The fix there is a narrow
 # additive `networking.k8s.io/v1` NetworkPolicy: ns monitoring -> the pod label, that one port.
 #
 # WHY THAT POLICY IS NOT IN THIS DIRECTORY. It would have to be five objects, one per app
 # namespace, and those namespaces are created by apps-wave-0/1. Measured Flux graph, 2026-10-01:
 #   oauth2-proxy  dependsOn=['platform-config']        <- this tier
 #   apps-wave-0   dependsOn=['platform-config']        <- SIBLING, no ordering between them
 #   apps-wave-1   dependsOn=['apps-wave-0']
 # They are siblings, so on a cold bootstrap there is no guarantee ns jackett exists when this tier
 # applies, and a NetworkPolicy naming a missing namespace fails to apply. It would work on today's
 # running cluster and break on a rebuild — the "works now, broken on restore" shape this repo
 # keeps paying for. Longhorn's precedent is safe only because longhorn-system is created by the
 # same tier that ships its policy. The allowance therefore belongs WITH THE APPS, rendered next to
 # the rest of each app's network policy — i.e. in the catalog/renderer, which is in flight on
 # another branch. Tracked in the handoff; it is one narrow rule per app, not a redesign.
 apiVersion: monitoring.coreos.com/v1
 kind: PodMonitor
 metadata:
  name: oauth2-proxy-sidecars
  namespace: oauth2-proxy
  labels:
    release: kube-prometheus-stack     # kube-prometheus-stack's podMonitorSelector
    app.kubernetes.io/name: oauth2-proxy
    homelab.keithjackson.dev/purpose: metrics-scrape
 spec:
  namespaceSelector:
    # The gate sidecars live in the app namespaces, not this one. `any` rather than a fixed list so
    # a new forward-auth app needs no edit here — the pod label below is what scopes this.
    any: true
  selector:
    matchLabels:
      # The PROPERTY, not the app list. Present on exactly the five fleet gate pods (measured
      # 2026-10-01); absent from crafty, which has no metrics listener yet.
      homelab.keithjackson.dev/auth-mode: forward-auth
  podMetricsEndpoints:
    # By PORT NAME (`metrics`, 44180), never by number: the name is what PR #28 declared on the
    # container, and a numeric port here would silently stop matching if the port were renumbered.
    - port: metrics
      path: /metrics
      interval: 30s
      # 10s, matching the drop signature above: a DENIED path does not refuse, it hangs until the
      # scrape times out. Leaving the default would make each failing target hold a scrape slot for
      # the full interval.
      scrapeTimeout: 10s
@@ -51,6 +51,36 @@ rule groups still evaluate against nothing and **cannot fire**:
 | Flux (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` |
 | Hubble (`hubble_drop_total`) | `HubbleDeniedFlowSpike` |
 | etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all |
 | per-app oauth2-proxy sidecars (`oauth2_proxy_*`) — updated 2026-10-01 | no authentication alerting exists at all for crafty, jackett, pihole, qbittorrent, radarr, sonarr |
 The oauth2-proxy row is the newest, and it has **moved twice in two days** — worth reading in order,
 because each step looked like the end of it.
 **2026-09-30 — nothing to scrape.** `count by (__name__) ({__name__=~"oauth2_proxy.*"})` returned
 exactly one family, `oauth2_proxy_requests_in_flight`, from the two replicas of the **shared**
 forward-auth instance. All six per-app gate sidecars ran without `--metrics-address`, so their metrics
 listener never started, and no ServiceMonitor or PodMonitor selected them.
 **2026-10-01 — the listener exists, and the scrape is still blocked.** PR #28 added
 `--metrics-address=0.0.0.0:44180` and a named `metrics` port to the five fleet sidecars, and the
 endpoint genuinely serves (port-forwarded: `http=200`, 10208 bytes, all three `oauth2_proxy_*`
 families, `oauth2_proxy_requests_total{code="302"}` present). `platform/oauth2-proxy/podmonitor.yaml`
 now discovers them by pod label. **But the app namespaces' `default-deny` NetworkPolicy drops the
 scrape**: from a pod in ns `monitoring`, both `:44180` and `:4180` on the jackett pod time out, while
 the shared proxy's `:44180` answers — the only difference being that ns `oauth2-proxy` has no
 NetworkPolicy at all. So the targets are discovered and `up == 0`, and the TSDB is still empty for
 those namespaces. **This is the same bug Longhorn had**, with the same fingerprint; the fix idiom is
 `platform/longhorn/networkpolicy-metrics.yaml` (ns monitoring → one pod label → one port), and for
 these apps it has to be rendered alongside each app's own policy rather than shipped from the
 oauth2-proxy tier, which is a *sibling* of `apps-wave-0` in the Flux graph and so cannot rely on those
 namespaces existing yet.
 `OAuth2ProxyGateUnobserved` (`platform/oauth2-proxy/auth-visibility-alerts.yaml`) alerts on the gap
 itself from `kube_pod_container_info` and `up` — two families that *are* scraped. Note its `up == 1`:
 a discovered-but-failing target publishes `up` with value **0**, so the earlier bare form would have
 gone quiet the moment the PodMonitor landed, while nothing was being scraped. That near-miss is the
 whole reason to distrust a cleared alert here. Crafty remains outside all of this — no metrics
 listener at all. See `oauth2-proxy-auth-failures`.
 Each affected runbook says so on its own page and gives the by-hand equivalent. See `targets-down`.
@@ -108,6 +138,14 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
 - **`keycloak-db`** — Keycloak's CloudNativePG database is unavailable.
 - **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
  every green signal missed.
 - **`oauth2-proxy-auth-failures`** — a gated app "is up but does not work". Linked from
  `OAuth2ProxyGateUnobserved`. The gate answers an unauthenticated **XHR** with a 302 to Keycloak
  rather than a 401, the browser follows it silently against a live SSO session, and the call
  resolves **200 carrying HTML** where the app's JavaScript wanted JSON — so the user is never
  re-prompted and every uptime check stays green. Tells the gate's refusal from the group check
  from the app's own login, from the sidecar log and from Loki, with the 2026-09-30 jackett trace
  as the worked example. **Read the caveat**: the per-app gate sidecars export no metrics at all,
  so these logs (72h of them) are the only record this class of failure leaves.
 ### Operator procedures (no alert links here)
 - **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only
@@ -85,10 +85,17 @@ configMapGenerator:
      - node-io=node-io.md
      - node-memory=node-memory.md
      - node-memory-swap=node-memory-swap.md
      # Linked from OAuth2ProxyGateUnobserved, and the diagnostic procedure for any gated app
      # that "is up but does not work" — the gate answers an unauthenticated XHR with a 302,
      # not a 401, so the failure arrives as an HTTP 200 carrying the wrong body.
      - oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md
      # Operator procedure, not alert-linked. Reading the per-app oauth2-proxy sidecar log to tell
      # Keycloak, the sidecar and the app's own login apart. Cookie collision between two sidecars
      # presents as random session loss in ONE app's log and is only legible with both side by side,
      # so the page has to be reachable at 3am without going looking for it in git.
      # SIBLING PAGE to oauth2-proxy-auth-failures above, and they are deliberately both here: this
      # one is the SESSION-LIFECYCLE procedure (why a session died), that one is the DIAGNOSTIC
      # TRIAGE (which of the three layers refused, and how to tell a 302-to-200 from a real success).
      - oauth2-proxy-session=oauth2-proxy-session.md
      # Operator procedure, not an alert page (no alert links to it — the
      # alert-runbooks-resolvable check treats that as advisory). It is served
@@ -0,0 +1,244 @@
 # oauth2-proxy auth failures — telling the three layers apart
 Linked from `OAuth2ProxyGateUnobserved`, and the page to open whenever a gated app "is up but
 does not work": a UI that loads and then shows nothing, a page that never finishes, a list that
 stays empty, a save that silently does not save. The alert is about the *visibility* gap; this
 page is the diagnostic procedure, and it works whether or not the alert ever fired.
 ## What this means here
 Every app behind Keycloak on this platform has **three** places a request can be refused, and
 from a browser they look identical — because two of the three answer with **HTTP 200**.
 | Layer | What refuses | What the browser sees |
 |---|---|---|
 | 1. The gate (oauth2-proxy sidecar) | no valid session cookie | **302 to Keycloak**, then usually **200** |
 | 2. The gate's group check | session valid, wrong group | 403, or a redirect loop |
 | 3. The app itself | the app's own login / API key | the app's own 401 or login page |
 **Layer 1 is the one that hides.** oauth2-proxy answers an unauthenticated request with a
 redirect, not a 401 — correct for a browser navigation, disastrous for an **XHR**. The app's
 JavaScript asks for JSON, the gate answers `302`, the browser follows it silently, the live
 Keycloak SSO session satisfies it with no prompt, and the XHR resolves **200 with an HTML
 dashboard in the body**. `fetch()` sees a success. The user sees an empty page. Reloading does
 nothing, and nobody is ever asked to log in, because from the browser's point of view nothing
 failed.
 The worked example, jackett on 2026-09-30, straight from the sidecar log:
    [oauthproxy.go:1026] No valid authentication in request. Initiating login.
    GET "/api/v2.0/indexers/yts/config?_=..."          HTTP/1.1 ... 302 400
    GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
    ... ten of these at 15:30:53
    [AuthSuccess] Authenticated via OAuth2: Session{email:... created:2026-09-30 15:31:06
                  expires:2026-09-30 15:36:06 refresh_token:true groups:[platform-admins]}
    GET "/UI/Dashboard"        200 44379
    GET "/api/v2.0/indexers?_=..." 200 1501229
    ... and 26 seconds later, inside that same 5-minute session:
    [oauthproxy.go:1026] No valid authentication in request. Initiating login.
 A session created at 15:31:06, valid until 15:36:06, not honoured at 15:31:32. That is the
 signature of the session being **re-established over and over instead of held** — a cookie that
 another proxy is overwriting, or a cookie that never comes back at all — not of Keycloak being
 down and not of the app being broken.
 **Read this before you trust any silence (updated 2026-10-01).** The five fleet gate sidecars now
 *do* run a metrics listener on `:44180`, and `platform/oauth2-proxy/podmonitor.yaml` discovers
 them — but **the scrape is dropped by the app namespaces' `default-deny` NetworkPolicy**, so those
 targets read `up == 0` and the TSDB holds nothing for them. Crafty has no listener at all. Only
 the shared `oauth2-proxy` namespace instance is genuinely scraped, and it is not in the data path
 for any of these apps. So there is still nothing to graph and nothing that could have alerted.
 **The logs below remain the only record that this class of failure happened.** Check which state
 you are in before you trust a quiet dashboard — "First commands" step 1 tells you in one query.
 ## First commands
 `kubectl` does not exist on the workstation; everything runs in the pinned container, and always
 with an explicit context.
 **1. Which namespaces have an unobserved gate** (this is what the alert measures):
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
     -c prometheus -- /bin/promtool query instant http://localhost:9090 \
     "count by (namespace) (kube_pod_container_info{container=\"oauth2-proxy\"}) unless count by (namespace) (up{container=\"oauth2-proxy\"} == 1)"'
 ```
 **1b. If a namespace is listed, is it "no target" or "target failing"?** The two need different
 fixes and the alert deliberately does not distinguish them:
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
     -c prometheus -- /bin/promtool query instant http://localhost:9090 \
     "up{container=\"oauth2-proxy\"}"'
 ```
 No series for the namespace → nothing discovers it (no PodMonitor match; check the pod carries
 `homelab.keithjackson.dev/auth-mode=forward-auth` and a named `metrics` port). A series with
 value **0** → discovered but every scrape fails; get the reason from the target's `lastError`:
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   curl -s localhost:19090/api/v1/targets | python3 -c "
 import json,sys
 for t in json.load(sys.stdin)[\"data\"][\"activeTargets\"]:
    if t[\"labels\"].get(\"container\")==\"oauth2-proxy\":
        print(t[\"labels\"].get(\"namespace\"), t[\"health\"], t.get(\"lastError\",\"\")[:90])"'
 ```
 `context deadline exceeded` is a **drop**, not a refusal — that is NetworkPolicy, not a dead
 endpoint. Confirm the endpoint itself is fine by port-forwarding straight to the pod, which
 bypasses policy entirely:
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n jackett port-forward \
     $(kubectl --context k3s-prod -n jackett get pod -l app.kubernetes.io/name=jackett -o name | head -1) \
     14418:44180 & sleep 5; curl -s -o /dev/null -w "http=%{http_code} bytes=%{size_download}\n" \
     http://127.0.0.1:14418/metrics'
 ```
 `http=200` with a few KB means the listener is healthy and the path is what is broken. As of
 2026-10-01 that is the state of all five fleet apps, and the fix is an ingress allowance from ns
 `monitoring` to port 44180 — see `platform/longhorn/networkpolicy-metrics.yaml` for the idiom.
 **2. The sidecar log — but you must grep it.** The gate's stdout is ~11,500 lines a day per app
 and almost all of it is the kubelet health probe (`GET "/ping" ... 200`). A plain
 `kubectl logs --tail=100` shows you nothing but `/ping` and looks perfectly healthy. Filter:
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n jackett logs deploy/jackett -c oauth2-proxy --since=2h \
     | grep -v "/ping" | grep -E "No valid authentication|AuthSuccess|403|401|callback"'
 ```
 Substitute the namespace: `crafty`, `jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`.
 **3. The same question across every gate at once, from Loki.** Alloy collects all of these, so
 you can ask about the whole fleet and about a window that has already scrolled out of
 `kubectl logs`. Port-forward Loki and query it:
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/loki 13100:3100 & sleep 5; \
   curl -s -G --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"No valid authentication in request\" [24h]))" \
     http://127.0.0.1:13100/loki/api/v1/query | python3 -m json.tool | grep -E "namespace|value" '
 ```
 and the denominator, which is what makes the number mean something:
 ```
   ... --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"AuthSuccess\" [24h]))" ...
 ```
 **How to read the ratio.** A healthy sign-in is roughly **one or two** `No valid authentication`
 lines per `AuthSuccess` — the first request of a session is unauthenticated by definition. A
 ratio in the tens means sessions are not surviving. Measured on prod 2026-09-30 (24h):
 | namespace | `No valid authentication` | `AuthSuccess` | ratio |
 |---|---|---|---|
 | crafty | 81 | 1 | 81:1 |
 | radarr | 67 | 1 | 67:1 |
 | jackett | 39 | 4 | ~10:1 |
 | pihole, qbittorrent, sonarr | 0 | 0 | (unused in the window) |
 All three of those are the failure, not the baseline.
 **The cleanest worked case of all — radarr, measured 2026-10-01.** After the cookie fix landed,
 jackett dropped to **0** `No valid authentication` lines in 13,612 (from 39), and `removing
 session` / `cookie signature not valid` went to **0 on both apps** (radarr had 8 and 8 before).
 The eviction fingerprint is gone. But radarr still logged **206** rejections — and sampling 209 of
 them, **every single one was the same request**:
    GET "/signalr/messages/negotiate" ... 302    <- browser UA, over and over
 That is one left-open browser tab. Its cookie is gone, its SignalR socket keeps trying to
 reconnect, and every attempt is silently 302'd to Keycloak. Nobody is at the keyboard, so nobody
 is ever re-prompted; the tab just sits there looking like a working page that has stopped
 updating. **This is the entire failure class in one line**: a non-navigation request answered with
 a redirect, forever, with no error surfaced anywhere.
 So when you read a rejection count, **always group by request path before concluding anything**:
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n radarr logs deploy/radarr -c oauth2-proxy --since=24h \
     | grep -v "/ping" | grep "No valid authentication" -A 1 \
     | grep -oE "\"[^\"]+\" HTTP" | sort | uniq -c | sort -rn | head'
 ```
 One path dominating means an orphaned client, not a broken gate — annoying but harmless, and it
 clears when the tab is closed. Many different paths from one session means the session itself is
 not surviving, which is the real defect. A per-app `--api-route` is the structural fix for the
 first shape: it makes those paths answer **401** instead of 302, so the client fails loudly
 instead of looping. Note that a count of **0 `AuthSuccess`** alongside a high rejection count
 means nobody has signed in since the last restart — so `--cookie-refresh` has never run, and no
 session-lifecycle conclusion can be drawn from that window at all.
 **4. Did the gate admit the user, or did the app refuse them?** Look for the `[AuthSuccess]` line
 and what follows it. If `[AuthSuccess]` appears and the very next thing is the *app's own* login
 page or the app's 401, the gate did its job and layer 3 is your problem — that is the Crafty
 shape, and `runbooks/crafty-admin-credential.md` covers it.
 **5. Cookie names, when sessions do not survive.** Two oauth2-proxy instances emitting the
 **same** cookie name on the **same** domain hold different cookie-secrets, so each overwrites the
 other's cookie and the loser logs exactly the `No valid authentication` line above. List what
 each gate is configured to emit (arguments only — never print a Secret's data):
 ```
 infra/devcontainer/dev.sh run bash -c \
  'for ns in crafty jackett pihole qbittorrent radarr sonarr; do printf "%-12s " "$ns"; \
     kubectl --context k3s-prod -n $ns get deploy $ns \
       -o jsonpath="{range .spec.template.spec.containers[?(@.name==\"oauth2-proxy\")]}{.args}{end}" \
     | tr "," "\n" | grep -E "cookie-name|cookie-domain" | tr "\n" " "; echo; done'
 ```
 A gate with no `--cookie-name` is using oauth2-proxy's default `_oauth2_proxy`. If two of them
 share that default **and** a zone-wide `--cookie-domain=.home.keithjackson.dev`, they are
 evicting each other, and the symptom is intermittent and depends on which app you visited last.
 ## Likely causes, most likely first
 1. **Cookie collision between two gates on one domain.** Default cookie names plus a zone-wide
   cookie domain. Signature: high `No valid authentication`-to-`AuthSuccess` ratio, sessions
   dying seconds after being created, symptom depends on browsing order, and *nothing* wrong at
   Keycloak. Fix is a unique `--cookie-name` and a host-scoped `--cookie-domain` per gate — see
   `decisions/2026-09-29-crafty-oauth2-proxy-signin.md`, which found and fixed exactly this.
 2. **The cookie is too large to come back.** With `--pass-access-token` the session carries the
   access token; a fat Keycloak token pushes the cookie past 4 KB and it gets split into
   `_oauth2_proxy_0`, `_oauth2_proxy_1`… If a chunk is dropped the session is unreadable and the
   gate logs the same line. Look for numbered cookie names in the log, and for a session that
   never survives a single navigation.
 3. **The XHR path is not the navigation path.** The app's API is under a prefix the gate handles
   differently, or the app issues cross-origin XHRs the cookie does not accompany. Signature:
   `/UI/...` returns 200 and `/api/...` returns 302 in the *same second*, which is precisely the
   jackett trace above.
 4. **Session lifetime shorter than the app's polling interval,** with no refresh. The jackett
   session above was a 5-minute session. If `refresh_token:true` is present and sessions still
   die early, the refresh is failing, not missing.
 5. **Group check, not authentication.** `--allowed-group` is a floor: a valid Keycloak session in
   the wrong group is refused *after* a successful login, which reads as a loop. Look for 403 and
   for the `groups:[...]` list on the `[AuthSuccess]` line.
 6. **The app, not the gate.** `[AuthSuccess]` followed by the app's own login page or 401.
 ## When to escalate or stop
 - **Stop and hand over if the fix is a manifest change.** Everything above is read-only
  diagnosis. Changing a `--cookie-name`, adding `--metrics-address`, or editing an alert goes
  through a branch and a PR; `kubectl edit` on a gate is drift that the next Flux reconcile
  reverts, and in the meantime you have an unreproducible cluster.
 - **Stop if you find yourself about to print a Secret.** The cookie-secret and client-secret live
  in `oauth2-proxy-credentials` and the per-app equivalents. Compare *key names* and whether two
  Secrets differ; never dump `.data` or `.stringData`. A password has been leaked into a
  transcript here exactly this way.
 - **Escalate to the platform owner if Keycloak itself is the suspect** — check `KeycloakDown` and
  `KeycloakDatabaseDown` first (`runbooks/keycloak-down.md`, `runbooks/keycloak-db.md`). In the
  2026-09-30 incident Keycloak was healthy throughout and blaming it would have cost days.
 - **The honest limit of this page.** With no metrics from the sidecars there is no history beyond
  Loki's **72h** retention (`platform/monitoring/loki.yaml`, `retention_period: 72h`). If the
  report is older than three days, the evidence is gone and the only remaining move is to
  reproduce it. That is the cost of the gap `OAuth2ProxyGateUnobserved` is tracking, and it is
  the reason to close it rather than silence it.
Author	SHA1	Message	Date
jakman85andClaude Opus 5	1e71b578d5	docs(obs): record why CI is red — approval-policy on the one conformance entry gate / gate (pull_request) Successful in 1m41s Details gate / conformance (pull_request) Has been skipped Details The gate job reported failure while the local run was rc 3 with nothing failed. Measured cause is a single sensitive path (the KNOWN_EXTERNAL_MONITORS entry) with no non-author approving review, which ci/approval-policy.sh blocks fail-closed by design. Also records the general trap: approval-policy short-circuits to N/A off a pull_request event, so a clean LOCAL gate is a false reassurance for any branch touching a sensitive path. Only the PR run tells the truth. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-01 22:12:19 -05:00
jakman85andClaude Opus 5	bce101d03f	feat(obs): wire the gate-sidecar scrape, and fix the alert that would have gone silent doing it gate / gate (pull_request) Failing after 1m42s Details gate / conformance (pull_request) Has been skipped Details PR #28 gave the five fleet oauth2-proxy sidecars --metrics-address=0.0.0.0:44180 and a named `metrics` port. This wires a scraper to them and, in the process, found a real bug in the alert this branch added. PodMonitor, not ServiceMonitor. Measured: no Service exposes 44180 on any of the five (jackett/radarr/sonarr/qbittorrent/pihole all publish only their app port + authproxy 4180), so a ServiceMonitor would select nothing. A PodMonitor targets pod ports directly and needs no change to catalog/render.py or the apps/wave-* renders and goldens. Selector is the pod label homelab.keithjackson.dev/auth-mode=forward-auth — present on exactly those five gate pods, absent from crafty's, which still has no metrics listener. The property, not an app list. THE BUG: `up{container="oauth2-proxy"}` -> `up{container="oauth2-proxy"} == 1`. A target that is discovered but FAILING still publishes `up`, with value 0. The series exists, so count by (namespace) returned 1, `unless` removed the namespace, and OAuth2ProxyGateUnobserved went QUIET while nothing was being scraped at all — this branch's own disease reproduced inside the alert written to cure it. Not hypothetical: the PodMonitor lands five targets whose scrape is dropped, so the bare form would have silently cleared on this very commit. Proven by mutation: reverting the `== 1` fails the two new unit-test scenarios (got: nil) and leaves every other assertion green. The scrape does NOT work yet, and that is documented rather than papered over. From a pod in ns monitoring to the jackett pod, :44180 AND :4180 both time out; the shared proxy's :44180 answers. ns jackett's default-deny is podSelector:{} with policyTypes:[Ingress,Egress] and NO ingress key (deny-all), and the one allowance is fromEntities:[ingress] to 4180 — the Gateway's Envoy only. The endpoint itself is healthy: port-forwarded, /metrics returns http=200, 10208 bytes, all three oauth2_proxy_* families including requests_total{code="302"}. Same bug and fingerprint as platform/longhorn/networkpolicy-metrics.yaml records for Longhorn; the fix is one narrow rule per app, and it cannot ship from this tier because oauth2-proxy and apps-wave-0 are Flux SIBLINGS (both dependsOn platform-config), so naming ns jackett here would apply today and fail on a cold rebuild. Needs an owner in the renderer. Touches conformance/** : one KNOWN_EXTERNAL_MONITORS entry, which is the mechanism the failing check explicitly instructs ("DECLARE it ... with who creates it") and which flux-podmonitor.yaml already uses. It records that the port name was measured on prod instead of resolved from a Service, and that the targets are currently down. That makes this PR require a non-author review on the sensitive-path rule. Also: ratio alert still NOT shipped. oauth2_proxy_requests_total now exists on the endpoint but not in the TSDB, so writing it today would make it inert for exactly the reason it was refused on 2026-09-30. Runbook gains policy-vs-no-target triage and the radarr /signalr/messages/negotiate worked case — one orphaned browser tab, 206 identical 302s, nobody ever re-prompted. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-01 21:56:07 -05:00
jakman85	ba79a1b59b	Merge remote-tracking branch 'origin/main' into feat/oauth2-proxy-auth-failure-visible # Conflicts: # runbooks/kustomization.yaml	2026-10-01 21:40:37 -05:00
jakman85	cdf3e14e27	Merge pull request 'feat(plex): hardware transcoding via `hardware.gpu: intel-quicksync`, plus the runbook for the manual last mile' (#32 ) from feat/plex-quicksync into main gate / gate (push) Successful in 1m42s Details gate / conformance (push) Has been skipped Details Reviewed-on: #32 Reviewed-by: Keith Jackson <[email protected]>	2026-10-02 02:17:48 +00:00
jakman85	41a20dfd20	Merge pull request 'feat(schema): declare a host hardware capability, `hardware.gpu: intel-quicksync`' (#27 ) from feat/sd-hardware-capability into main gate / conformance (push) Has been cancelled Details gate / gate (push) Has been cancelled Details Reviewed-on: #27 Reviewed-by: Keith Jackson <[email protected]>	2026-10-02 02:17:23 +00:00
jakman85andClaude Opus 5	fc7ec01628	feat(obs): make oauth2-proxy auth failures visible — alert on the gap, not on absent metrics gate / gate (pull_request) Successful in 1m42s Details gate / conformance (pull_request) Has been skipped Details Jackett failed on 2026-09-30 in the most expensive way available: HTTP 200 and a plausible page. oauth2-proxy answers an unauthenticated XHR with a 302 to Keycloak rather than a 401; the browser follows it silently against a live SSO session and the call resolves 200 carrying a 44 KB HTML dashboard where the app's JavaScript wanted JSON. The user is never re-prompted, reloading never helps, and every uptime check stays green. `jackett-down` watches replicas_available, which was 1 throughout. The reason nothing alerted turns out not to be a missing rule. Measured on prod: count by (__name__) ({__name__=~"oauth2_proxy."}) oauth2_proxy_requests_in_flight => 2 Two series, both from the SHARED forward-auth Deployment. The six per-app gate sidecars (crafty, jackett, pihole, qbittorrent, radarr, sonarr) publish nothing at all: no --metrics-address, so the metrics listener never starts; no `metrics` container port; no ServiceMonitor or PodMonitor selects them. The alert the incident calls for — sustained auth redirects to an authenticated user — is therefore NOT WRITABLE, and writing it anyway would have shipped a rule that is present, evaluating, permanently silent and indistinguishable from health. So alert on the condition that made the incident invisible, from families that ARE scraped: OAuth2ProxyGateUnobserved count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by (namespace) (up{container="oauth2-proxy"}) kube_pod_container_info is kube-state-metrics (a declared scrape owner) and sees the container whether or not it exports anything; `up` is UNIVERSAL and carries a `container` label for any target, so the rule needs no advance knowledge of what job name a future per-app scrape picks. Nothing in it touches oauth2_proxy_, so it cannot be inert for the reason it is reporting. Run verbatim against live prod it returns six namespaces and correctly excludes oauth2-proxy. It clears per namespace as each gate gains a real scrape. Also: - promtool companion test pinning all four states: fires unscraped, quiet when scraped, honours for:1h, and STOPS once a scrape appears under a different job name. - runbooks/oauth2-proxy-auth-failures.md — tell the gate's refusal from the group check from the app's own login from an XHR that silently got HTML, with the 2026-09-30 jackett trace as the worked example and the Loki queries to use while there are no metrics. Registered in the configMapGenerator and README. - runbooks/README.md gains an inert-alerts row for the unscraped sidecars. Rejected: an alert on the shared proxy's oauth2_proxy_requests_total ratio. It needs a conformance/ SCRAPE_OWNERS entry (human approval) and, decisively, would not have fired on this incident — the shared instance is not in any sidecar-gated app's data path and its current pods have proxied nothing. It would have looked like coverage and been none. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-30 10:55:19 -05:00