From fc7ec01628e016c2fc8540d7cbbd178cafdc5067 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Wed, 30 Sep 2026 10:55:19 -0500 Subject: [PATCH 1/3] =?UTF-8?q?feat(obs):=20make=20oauth2-proxy=20auth=20f?= =?UTF-8?q?ailures=20visible=20=E2=80=94=20alert=20on=20the=20gap,=20not?= =?UTF-8?q?=20on=20absent=20metrics?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Jackett failed on 2026-09-30 in the most expensive way available: HTTP 200 and a plausible page. oauth2-proxy answers an unauthenticated XHR with a 302 to Keycloak rather than a 401; the browser follows it silently against a live SSO session and the call resolves 200 carrying a 44 KB HTML dashboard where the app's JavaScript wanted JSON. The user is never re-prompted, reloading never helps, and every uptime check stays green. `jackett-down` watches replicas_available, which was 1 throughout. The reason nothing alerted turns out not to be a missing rule. Measured on prod: count by (__name__) ({__name__=~"oauth2_proxy.*"}) oauth2_proxy_requests_in_flight => 2 Two series, both from the SHARED forward-auth Deployment. The six per-app gate sidecars (crafty, jackett, pihole, qbittorrent, radarr, sonarr) publish nothing at all: no --metrics-address, so the metrics listener never starts; no `metrics` container port; no ServiceMonitor or PodMonitor selects them. The alert the incident calls for — sustained auth redirects to an authenticated user — is therefore NOT WRITABLE, and writing it anyway would have shipped a rule that is present, evaluating, permanently silent and indistinguishable from health. So alert on the condition that made the incident invisible, from families that ARE scraped: OAuth2ProxyGateUnobserved count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by (namespace) (up{container="oauth2-proxy"}) kube_pod_container_info is kube-state-metrics (a declared scrape owner) and sees the container whether or not it exports anything; `up` is UNIVERSAL and carries a `container` label for any target, so the rule needs no advance knowledge of what job name a future per-app scrape picks. Nothing in it touches oauth2_proxy_*, so it cannot be inert for the reason it is reporting. Run verbatim against live prod it returns six namespaces and correctly excludes oauth2-proxy. It clears per namespace as each gate gains a real scrape. Also: - promtool companion test pinning all four states: fires unscraped, quiet when scraped, honours for:1h, and STOPS once a scrape appears under a different job name. - runbooks/oauth2-proxy-auth-failures.md — tell the gate's refusal from the group check from the app's own login from an XHR that silently got HTML, with the 2026-09-30 jackett trace as the worked example and the Loki queries to use while there are no metrics. Registered in the configMapGenerator and README. - runbooks/README.md gains an inert-alerts row for the unscraped sidecars. Rejected: an alert on the shared proxy's oauth2_proxy_requests_total ratio. It needs a conformance/ SCRAPE_OWNERS entry (human approval) and, decisively, would not have fired on this incident — the shared instance is not in any sidecar-gated app's data path and its current pods have proxied nothing. It would have looked like coverage and been none. Co-Authored-By: Claude Opus 5 (1M context) --- ...09-30-oauth2-proxy-auth-failure-visible.md | 175 ++++++++++++++++++ platform/oauth2-proxy/README.md | 36 ++++ .../auth-visibility-alerts.promtool-test.yaml | 117 ++++++++++++ .../oauth2-proxy/auth-visibility-alerts.yaml | 113 +++++++++++ platform/oauth2-proxy/kustomization.yaml | 4 + runbooks/README.md | 20 ++ runbooks/kustomization.yaml | 4 + runbooks/oauth2-proxy-auth-failures.md | 171 +++++++++++++++++ 8 files changed, 640 insertions(+) create mode 100644 decisions/2026-09-30-oauth2-proxy-auth-failure-visible.md create mode 100644 platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml create mode 100644 platform/oauth2-proxy/auth-visibility-alerts.yaml create mode 100644 runbooks/oauth2-proxy-auth-failures.md diff --git a/decisions/2026-09-30-oauth2-proxy-auth-failure-visible.md b/decisions/2026-09-30-oauth2-proxy-auth-failure-visible.md new file mode 100644 index 0000000..edf73ca --- /dev/null +++ b/decisions/2026-09-30-oauth2-proxy-auth-failure-visible.md @@ -0,0 +1,175 @@ +--- +date: 2026-09-30 +branch: feat/oauth2-proxy-auth-failure-visible +title: The Jackett auth failure was invisible because the gates emit nothing — alert on the gap, not on a metric that does not exist +--- + +## What changed + +Jackett's UI broke in the way that costs the most: **HTTP 200 and a plausible page**. The +operator's report named the property that matters — *"reloading the page never fixes the issue and +the user is never re-prompted for login"*. Two sibling branches fix the auth defects. This one is +about the fact that **no signal existed anywhere**, and it turns out that is not an oversight in +the alert rules. It is an absence of data. + +**The measurement that decided the whole design** (prod, live Prometheus, 2026-09-30): + + count by (__name__) ({__name__=~"oauth2_proxy.*"}) + oauth2_proxy_requests_in_flight => 2 + +Two series. Both from the **shared** forward-auth Deployment in the `oauth2-proxy` namespace. +That is every oauth2-proxy metric this cluster holds. The six per-app gate sidecars — crafty, +jackett, pihole, qbittorrent, radarr, sonarr — publish **nothing at all**: they run with +`--http-address=0.0.0.0:4180` and no `--metrics-address`, so oauth2-proxy's metrics listener never +starts; they declare only a `http:4180` container port; and no ServiceMonitor or PodMonitor in the +cluster selects them. + +So the alert the brief asked for — *"a gated app is serving a sustained rate of auth redirects to +a user who is authenticated"* — **is not writable**, and writing it anyway would have produced +exactly the artefact this repo keeps getting burned by: a rule that is present, evaluating, +permanently silent, and indistinguishable from health. It was not written. + +What was written instead alerts on the condition that made the incident invisible, from families +that **are** scraped: + +- **`platform/oauth2-proxy/auth-visibility-alerts.yaml`** — `OAuth2ProxyGateUnobserved`: + + count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) + unless + count by (namespace) (up{container="oauth2-proxy"}) + + `for: 1h`, `severity: warning`. `kube_pod_container_info` is kube-state-metrics (a declared + owner in `alert-metrics-have-scrape-targets`) and sees the container whether or not it exports + anything; `up` is UNIVERSAL and carries a `container` label for any scrape target, so the rule + needs no advance knowledge of what job name a future per-app scrape will use. Nothing in it + touches `oauth2_proxy_*`, so it cannot be inert for the reason it is reporting. **It returns six + namespaces against live prod right now** and excludes `oauth2-proxy`, which is scraped. It + clears per-namespace as each gate gains a real scrape. +- **`platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml`** — four states pinned: + fires for an unscraped namespace, never for the scraped one, honours `for: 1h` (pending at 59m), + and **stops** once a scrape appears under a *different* job name. Series shapes copied off live + prod, not invented. +- **`runbooks/oauth2-proxy-auth-failures.md`** — how to tell "the gate refused me" from "the group + check refused me" from "the app refused me" from "the XHR silently got HTML", with the + 2026-09-30 jackett trace as the worked example. Registered in `runbooks/kustomization.yaml`'s + `configMapGenerator` and listed in `runbooks/README.md`. +- **`runbooks/README.md`** — a new row in *"Alerts you should not trust the silence of"* for the + per-app sidecars, plus the measured reason. +- **`platform/oauth2-proxy/README.md`** — what this component can and cannot see, and the exact + three-part change that would close the gap. + +**Why the obvious alternative was rejected.** An alert on the *shared* proxy's +`oauth2_proxy_requests_total{code="302"}` ratio is the one a reader expects to see here. It would +need a new `SCRAPE_OWNERS` entry in `conformance/**` (a sensitive path needing human approval), +and — decisively — **it would not have fired on this incident**: the shared instance is not in the +data path for any sidecar-gated app, and its current pods export no `oauth2_proxy_requests_total` +at all, having proxied nothing since they started (the counter exists in the TSDB only from an +older ReplicaSet, `oauth2-proxy-78449b5657-*`, with `code` labels `302` and `401` and no `202`). +Shipping it would have looked like coverage of the reported failure and been none. + +## What I verified + +Everything below was measured against live prod through the pinned container, read-only. No +`apply`, no `edit`, no `patch`, no `flux reconcile`. + +- **The full oauth2-proxy metric inventory.** `count by (__name__) ({__name__=~"oauth2_proxy.*"})` + → `oauth2_proxy_requests_in_flight => 2`. `count by (__name__) ({job="oauth2-proxy"})` returns + go_*/process_*/promhttp_* plus that one family and nothing else. **Only the shared instance is + scraped**: `up{job="oauth2-proxy"}` → 2 targets, `10.0.1.189:44180` and `10.0.2.47:44180`, both + `1`, `namespace="oauth2-proxy"`. +- **The per-app sidecars are unscraped, and unscrapable as configured.** Every pod carrying an + `oauth2-proxy` container, with its declared ports: the two shared replicas have + `[('http',4180),('metrics',44180)]`; crafty, jackett, pihole, qbittorrent, radarr and sonarr all + have `[('http',4180)]` only. The live jackett sidecar's args contain no `--metrics-address`. + `kubectl get servicemonitors,podmonitors -A` lists 16 ServiceMonitors and 5 PodMonitors; the + only oauth2-proxy entry is `oauth2-proxy/oauth2-proxy`. +- **No existing alert is inert *for this reason*, because no alert references these metrics at + all.** `grep -rn oauth2_proxy --include=*.yaml` over the tree hits only cookie-name strings in + `platform/crafty/`. The 39 live PrometheusRules include `platform-health-identity`, whose four + rules are about Keycloak readiness and the client reconciler and are all keyed on `kube_*` + (scraped, fine). **The inertness here is the absence of a rule, not a broken one** — which is + why the runbook README's inert-alerts table gains a row phrased like the etcd one. +- **Keycloak cannot express it either.** `count by (__name__) ({__name__=~"keycloak_.*"})` → + empty, and empty over `[30d]`. The `keycloak-service` scrape yields JVM/Agroal/JGroups metrics + only; there are no user-event or login counters, so "a burst of re-logins for one client" is not + observable from the IdP side. +- **`oauth2_proxy_requests_total` has existed historically but not on the current pods.** + `count_over_time(oauth2_proxy_requests_total[30d])` returns three series, all from + `oauth2-proxy-78449b5657-*` (a prior ReplicaSet): `code="302"` on both replicas and `code="401"` + on one, ~634 samples each (~5h of a 30d window). **No `code="202"` ever.** +- **The alert expression fires against live data.** Run verbatim on prod: + `count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by + (namespace) (up{container="oauth2-proxy"})` → `pihole, jackett, crafty, qbittorrent, sonarr, + radarr`, value 1 each; `oauth2-proxy` correctly absent. +- **The log signal exists and is collected.** Loki (`loki-0`, SingleBinary) and a 3-node Alloy + DaemonSet are running, and Alloy labels pod logs with `namespace`/`container`. LogQL over 24h, + `{container="oauth2-proxy"}`: crafty 12036 lines, jackett 11669, oauth2-proxy 34559, pihole + 11520, qbittorrent 11520, radarr 11763, sonarr 11520. Filtered to `|= "No valid authentication + in request"`: **crafty 81, jackett 39, radarr 67**; to `|= "AuthSuccess"`: **crafty 1, jackett + 4, radarr 1**. Those ratios (81:1, 10:1, 67:1) are the incident's signature and are in the + runbook as the thing to compare against. +- **Why `kubectl logs` hid it.** The jackett sidecar's stdout is ~11.5k lines/day and almost all + of it is `GET "/ping" ... 200 2` from the kubelet probe, twice every 15s. A `--tail=40` shows + forty `/ping` lines and looks perfectly healthy. The runbook's commands all filter. +- **Runbook registration, both halves.** `- oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md` + is in `runbooks/kustomization.yaml`'s `configMapGenerator.files`, and the page is listed under + *Identity* in `runbooks/README.md`. The page clears the L0 quality floor (the four operator + sections, >1500 bytes, mentions `dev.sh` and `--context k3s-prod`), and the alert's + `runbook_url` points at exactly that name. +- **Gate:** run in the pinned toolbox from this worktree with the `ENGINE` shim. Exit code and + per-stage outcomes in the next section — read them there, not here. + +## What I did NOT verify + +- **The alert has never fired in Alertmanager.** Nothing is applied; the PrometheusRule exists + only in git on this branch. What is proven is that its expression returns the expected vector + against live prod data and that promtool agrees about its behaviour over synthetic series. + **Merging is what deploys it.** +- **`OAuth2ProxyGateUnobserved` would NOT have fired *during* the 2026-09-30 jackett incident in + any useful sense** — it describes a standing condition that has been true since the sidecars + were deployed, so it would have been firing before, during and after. That is deliberate and it + is the honest claim: it is a **gap tracker**, not an incident detector. It would have meant that + on 2026-09-30 an operator asking "why did nothing alert?" had the answer already on the board + instead of discovering it during the investigation. **No alert on this platform could have + detected the incident itself, and none can today.** +- **The alert that the brief actually asked for does not exist, and cannot be written yet.** The + sustained-redirect-ratio rule needs `oauth2_proxy_requests_total` from the *per-app* sidecars. + **Enabling that requires editing files owned by the two sibling branches** — the sidecar args in + `apps/wave-0/jackett.yaml` (Mira Vance) and `apps/wave-0/{qbittorrent,pihole}.yaml`, + `apps/wave-1/{radarr,sonarr}.yaml` (Theo Bright) — so it was **not attempted here**. The change + is three things per app, together: `--metrics-address=0.0.0.0:44180`, a `metrics` containerPort + plus Service port, and a ServiceMonitor. Once any one app has them, the rule belongs beside + `OAuth2ProxyGateUnobserved`, and that app's namespace drops out of it automatically. +- **The Loki alerting path is inert and I did not fix it.** Loki *has* a ruler stanza and the + chart's `loki-sc-rules` sidecar is running with `LABEL=loki_rule`, `FOLDER=/rules` — so a + ConfigMap-shipped LogQL rule would be delivered. But `GET /loki/api/v1/rules` answers **"unable + to read rule dir /var/loki/rules/fake: no such file or directory"** (the ruler reads a different + directory from the one the sidecar writes), and the rendered config contains **no + `alertmanager_url`** — `grep -i 'alertmanager\|rule_path'` over the `loki` ConfigMap returns + nothing. So a LogQL alert on `|= "No valid authentication in request"` — which *would* have + caught this incident, and is the single highest-value follow-up here — **would evaluate and + notify nobody**. Wiring it means `rulerConfig` (alertmanager_url + a matching rules directory) + in `platform/monitoring/loki.yaml`, which is not a file this brief gave me. **I did not ship an + inert rule ConfigMap to look productive.** +- **Nothing was reproduced in a browser.** The 302-to-200-with-HTML mechanism is read off the + sidecar access log and the response sizes (`/UI/Dashboard 200 44379`), not observed in devtools. +- **Loki retention is 72h** (`platform/monitoring/loki.yaml`), so the 24h ratios above are a + snapshot; anything older than three days is already gone. This is stated in the runbook because + it bounds what an operator can still investigate. +- **The gate's L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), so no runtime + requirement touched by this branch is certified. See the exit code below — **rc 3 is + UNDETERMINED, not green.** + +### Follow-ups, in order of value + +1. **Wire the Loki ruler to Alertmanager** (`platform/monitoring/loki.yaml`). This is the only + route to an alert that would have caught the actual 2026-09-30 failure, and the data is + already being collected. The rule is one expression: + `sum by (namespace) (rate({container="oauth2-proxy"} |= "No valid authentication in request" [15m])) + > 5 * sum by (namespace) (rate({container="oauth2-proxy"} |= "AuthSuccess" [15m]))`. +2. **Give one gate sidecar a metrics endpoint** and watch its namespace drop out of + `OAuth2ProxyGateUnobserved`. That is the cheapest proof the alert is a tracker and not a nag, + and it unlocks the ratio rule in Prometheus. +3. **Consider whether `--set-xauthrequest` apps should get `--api-routes`** so API paths answer + 401 instead of 302. That is the fix that would make this failure *loud* rather than merely + observable, and it belongs with whoever owns the sidecar args. diff --git a/platform/oauth2-proxy/README.md b/platform/oauth2-proxy/README.md index dff6816..c2e2cf2 100644 --- a/platform/oauth2-proxy/README.md +++ b/platform/oauth2-proxy/README.md @@ -88,3 +88,39 @@ This proves the R-IDP-02 property (unauthenticated cannot reach the upstream) on 3. Log in as a **non-platform-admins** user -> expect **403** (authenticated but not authorized). 4. Confirm the session cookie is `.home.keithjackson.dev`-scoped (single sign-on across the other protected UIs). + +## Observability: what this component can and cannot see (measured 2026-09-30) + +`servicemonitor.yaml` scrapes **this namespace's** deployment and nothing else. On prod that is +two targets, `job="oauth2-proxy"`, both `up=1`. The complete set of oauth2-proxy-specific series +in the TSDB is: + + count by (__name__) ({__name__=~"oauth2_proxy.*"}) + oauth2_proxy_requests_in_flight => 2 + +**The per-app gate sidecars are not scraped, and there is nothing to scrape.** Six apps run an +`oauth2-proxy` container of their own — crafty, jackett, pihole, qbittorrent, radarr, sonarr — +each with `--http-address=0.0.0.0:4180` and **no `--metrics-address`**, so oauth2-proxy's metrics +listener never starts. They declare only a `http:4180` container port, no `metrics` port exists on +their Services, and `kubectl get servicemonitors,podmonitors -A` selects none of them. Any alert +written against `oauth2_proxy_requests_total{namespace="jackett"}` would be structurally inert. + +This matters because the gate's characteristic failure is invisible by construction: an +unauthenticated **XHR** gets a `302` to Keycloak rather than a `401`, the browser follows it +against a live SSO session with no prompt, and the call resolves **200 carrying an HTML page** +where the app expected JSON. Nothing in the request path reports an error. On 2026-09-30 jackett +spent a burst of requests in exactly that state, re-established a session, and lost it 26 seconds +into a 5-minute session, while `jackett-down` sat at 1 replica available the whole time. + +`auth-visibility-alerts.yaml` therefore alerts on **the gap**, not on the incident: +`OAuth2ProxyGateUnobserved` compares `kube_pod_container_info{container="oauth2-proxy"}` against +`up{container="oauth2-proxy"}` — two already-scraped families — and names every namespace running +a gate that nothing observes. It fires for six namespaces today and clears per-namespace as each +one gains a scrape. `runbooks/oauth2-proxy-auth-failures.md` is the by-hand substitute meanwhile, +and it reads the sidecar logs out of Loki (Alloy collects them; 72h retention). + +**To close the gap for an app**, three things are needed together, on that app's own manifest: +`--metrics-address=0.0.0.0:44180` on the sidecar, a `metrics` containerPort plus the matching +Service port, and a ServiceMonitor selecting it. Only then does the auth-churn rule — sustained +`code="302"` as a share of `oauth2_proxy_requests_total`, held across a window longer than the +session lifetime — become writable rather than decorative. diff --git a/platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml b/platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml new file mode 100644 index 0000000..3fb9270 --- /dev/null +++ b/platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml @@ -0,0 +1,117 @@ +--- +# promtool unit tests for OAuth2ProxyGateUnobserved (platform/oauth2-proxy/auth-visibility-alerts.yaml). +# +# WHY. The rule is an `unless` between two aggregations over DIFFERENT metric families, and the +# failure modes of that shape are both silent: a label that does not survive `count by` makes the +# right-hand side match nothing (the alert fires for the scraped namespace too, i.e. permanently), +# and a right-hand side that over-matches makes it fire for nobody (i.e. inert). Neither is +# visible by reading the expression, and neither is visible on a live cluster where the answer +# happens to look plausible. So all four states are pinned: fires for an unscraped namespace, +# stays quiet for a scraped one, honours `for: 1h`, and STOPS once a scrape appears. +# +# HOW TO RUN. The one repo-wide harness, by the companion convention: +# ./infra/devcontainer/dev.sh run bash -c 'platform/monitoring/tests/run.sh platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml' +# promtool is pinned in infra/devcontainer/versions.env; never run it off the host PATH. +# +# THE SERIES BELOW ARE REAL SHAPES read off live prod Prometheus on 2026-09-30 before being +# copied here — kube_pod_container_info as kube-state-metrics emits it (job="kube-state-metrics", +# its own instance/pod labels, image/image_id/container_id) and `up` as the oauth2-proxy +# ServiceMonitor emits it (endpoint="metrics", port 44180). The point of using the real shapes is +# that `count by (namespace)` has to discard all of them; a test with tidy two-label series would +# pass while the real expression collapsed differently. +# +# ASSERTIONS USE `ALERTS` RATHER THAN exp_alerts: the description annotation is long prose and +# comparing it verbatim would couple this file to every wording change while proving nothing the +# ALERTS series does not already prove. +rule_files: + - auth-visibility-alerts.rules.yaml +evaluation_interval: 1m + +tests: + # --------------------------------------------------------------------------------------------- + # 1. THE STATE PROD IS ACTUALLY IN (2026-09-30): jackett runs a gate sidecar with no scrape; + # the shared oauth2-proxy namespace runs two that ARE scraped. Asserts the alert fires for + # jackett and NOT for oauth2-proxy, and that `for: 1h` is honoured (pending at 59m). + - interval: 1m + name: an unscraped gate namespace fires after 1h; the scraped one never does + input_series: + - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}' + values: '1x130' + # The shared instance: kube-state-metrics sees both replicas... + - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://aa11bb22",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000001"}' + values: '1x130' + - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://cc33dd44",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000002"}' + values: '1x130' + # ...and both replicas are genuinely scraped, so `up` carries container="oauth2-proxy" there. + - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.189:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="oauth2-proxy"}' + values: '1x130' + - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.2.47:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="oauth2-proxy"}' + values: '1x130' + promql_expr_test: + # Not yet firing at 59m — `for: 1h` has not elapsed. Guards against someone "simplifying" + # the hold away and turning every rollout into a page. + - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}' + eval_time: 59m + exp_samples: [] + - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett"}' + eval_time: 59m + exp_samples: + - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett",severity="warning"}' + value: 1 + # Firing for jackett past the hold... + - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}' + eval_time: 65m + exp_samples: + - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}' + value: 1 + # ...and never for the namespace that IS scraped. This is the assertion that catches an + # `unless` whose right-hand side stopped matching: without it, a broken rule that fires for + # every gate namespace would pass the test above. + - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="oauth2-proxy"}' + eval_time: 65m + exp_samples: [] + # The count must be 1 per namespace-with-a-gate, not 1 overall: a `count by (namespace)` + # that lost its `by` would collapse to a single unlabelled series. + - expr: 'count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by (namespace) (up{container="oauth2-proxy"})' + eval_time: 65m + exp_samples: + - labels: '{namespace="jackett"}' + value: 1 + + # --------------------------------------------------------------------------------------------- + # 2. THE FIX LANDING. jackett's sidecar gains --metrics-address and a ServiceMonitor 70m in, so + # `up` starts carrying container="oauth2-proxy" for namespace="jackett" under a DIFFERENT job + # name than the shared proxy's. The alert must stop. This is what makes the rule a gap + # tracker rather than a permanent nag, and it pins that the match is on `container`, not on + # a hardcoded job="oauth2-proxy" (which would keep firing forever after a correct fix). + - interval: 1m + name: the alert clears once the sidecar is actually scraped + input_series: + - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}' + values: '1x130' + # Absent for the first 70 minutes, then present. `_x` is "no sample"; the scrape did not + # exist yet. + - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.213:44180",job="jackett-auth",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="jackett-auth-metrics"}' + values: '_x70 1x60' + promql_expr_test: + - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett"}' + eval_time: 65m + exp_samples: + - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}' + value: 1 + - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="jackett"}' + eval_time: 80m + exp_samples: [] + + # --------------------------------------------------------------------------------------------- + # 3. NO GATE, NO ALERT. A namespace with no oauth2-proxy container must never appear, or the + # rule would page for every app on the platform the day someone widened the matcher. + - interval: 1m + name: namespaces without a gate sidecar are never reported + input_series: + - series: 'kube_pod_container_info{container="plex",container_id="containerd://feed0001",endpoint="http",image="ghcr.io/example/plex:1",image_id="ghcr.io/example/plex@sha256:feed0001",image_spec="ghcr.io/example/plex:1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="plex",pod="plex-0",service="kube-prometheus-stack-kube-state-metrics",uid="2222bbbb-0000-0000-0000-000000000001"}' + values: '1x130' + promql_expr_test: + - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved"}' + eval_time: 65m + exp_samples: [] diff --git a/platform/oauth2-proxy/auth-visibility-alerts.yaml b/platform/oauth2-proxy/auth-visibility-alerts.yaml new file mode 100644 index 0000000..bd1d73b --- /dev/null +++ b/platform/oauth2-proxy/auth-visibility-alerts.yaml @@ -0,0 +1,113 @@ +--- +# oauth2-proxy AUTH-VISIBILITY alerting (R-OBS-04). +# +# ============================================================================================ +# WHY THIS FILE EXISTS — THE 2026-09-30 JACKETT FAILURE THAT NOTHING OBSERVED +# ============================================================================================ +# Jackett's UI stopped working. Reloading never fixed it and the user was never re-prompted to +# log in. The sidecar's own access log had the whole story: +# +# [oauthproxy.go:1026] No valid authentication in request. Initiating login. +# GET "/api/v2.0/indexers/yts/config?_=..." HTTP/1.1 ... 302 400 +# GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409 +# ... ten of these at 15:30:53, then a fresh [AuthSuccess] at 15:31:06, +# then "No valid authentication in request" again 26 SECONDS LATER, inside the same session. +# +# NOTHING ANYWHERE ELSE RECORDED IT. oauth2-proxy answers an unauthenticated XHR with a 302 to +# Keycloak rather than a 401; the browser follows it silently, the live SSO session satisfies it +# without a prompt, and the XHR resolves 200 carrying a 44 KB HTML dashboard where the app's +# JavaScript wanted JSON. From the browser, from the user, and from any uptime check, the service +# is healthy. `jackett-down` (apps/wave-0/jackett.yaml) watches +# kube_deployment_status_replicas_available, which was 1 throughout and would have been 1 if the +# gate had rejected every request ever made. +# +# ============================================================================================ +# WHY THIS FILE DOES NOT CONTAIN THE OBVIOUS ALERT — MEASURED, prod, 2026-09-30 +# ============================================================================================ +# The alert you would want is "a gated app is serving a sustained rate of auth redirects to a +# user who is already authenticated". It is NOT WRITABLE against this cluster's TSDB, and the +# reason is this file's real subject: +# +# THE PER-APP GATE SIDECARS EXPORT NO METRICS AT ALL. +# +# $ promtool query instant :9090 'count by (__name__) ({__name__=~"oauth2_proxy.*"})' +# oauth2_proxy_requests_in_flight => 2 <- the SHARED proxy's two replicas. That is ALL. +# +# Every sidecar (crafty, jackett, pihole, qbittorrent, radarr, sonarr) runs with +# `--http-address=0.0.0.0:4180` and NO `--metrics-address`, so oauth2-proxy's metrics listener +# is never started; the container declares only port `http:4180`, there is no `metrics` port on +# any of their Services, and there is no ServiceMonitor or PodMonitor for any of them anywhere +# in the cluster (measured: `kubectl get servicemonitors,podmonitors -A`). Six gates, zero +# telemetry. An alert written against `oauth2_proxy_requests_total{namespace="jackett"}` would +# have been PERMANENTLY INERT — present, evaluating, silent, and indistinguishable from health. +# This repo has shipped that mistake before (VeleroNoRecentBackup, the Longhorn rules), which is +# why `conformance/tests/l0-static/alert-metrics-have-scrape-targets` exists, and why the +# temptation was refused here rather than ratcheted into a register. +# +# The SHARED proxy IS scraped (job="oauth2-proxy", 2 targets, up=1) but it is not in the data +# path for any sidecar-gated app — each app gates itself — and its current pods export no +# `oauth2_proxy_requests_total` at all, having proxied nothing since they started. An alert on +# the shared instance would not have fired on this incident either. Saying so is the point. +# +# So this file alerts on the CONDITION THAT MADE THE INCIDENT INVISIBLE, not on the incident. +# That condition is measurable today, from two already-scraped metric families, and it is the +# precondition for ever being able to alert on the incident itself. It self-clears the moment a +# sidecar gains a metrics endpoint and a scrape — at which point the auth-churn rule becomes +# writable and belongs here beside it. See runbooks/oauth2-proxy-auth-failures.md, which is the +# by-hand equivalent an operator uses in the meantime. +apiVersion: monitoring.coreos.com/v1 +kind: PrometheusRule +metadata: + name: oauth2-proxy-auth-visibility + namespace: oauth2-proxy + labels: + release: kube-prometheus-stack # kube-prometheus-stack's ruleSelector + app.kubernetes.io/name: oauth2-proxy +spec: + groups: + - name: oauth2-proxy.auth-visibility + rules: + - alert: OAuth2ProxyGateUnobserved + # A namespace runs an oauth2-proxy gate container, and Prometheus holds no scrape + # target for that container in that namespace. + # + # BOTH SIDES ARE ALREADY-SCRAPED FAMILIES, deliberately: + # * kube_pod_container_info — kube-state-metrics, the declared owner in + # alert-metrics-have-scrape-targets. It sees the container whether or not the + # container exports anything, which is exactly the property needed here. + # * up — UNIVERSAL. `up` carries the `container` label for any + # ServiceMonitor/PodMonitor target, so this works for a future per-app scrape + # without knowing what job name it will pick. + # Nothing in this expression touches `oauth2_proxy_*`, so the rule cannot itself be + # inert for the reason it is reporting. + # + # MEASURED ON PROD 2026-09-30 — this expression returns six namespaces right now: + # {namespace="crafty"} {namespace="jackett"} {namespace="pihole"} + # {namespace="qbittorrent"} {namespace="radarr"} {namespace="sonarr"} + # and excludes namespace="oauth2-proxy", whose two replicas ARE scraped. It fires on + # the state the cluster is in; it is not a rule waiting for a hypothetical. + expr: | + count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) + unless + count by (namespace) (up{container="oauth2-proxy"}) + # 1h, not 5m: a rollout briefly has a running sidecar whose target has not been + # discovered yet, and this condition is a standing architectural gap, never an + # emergency. Long enough that no deploy can produce it. + for: 1h + labels: + severity: warning + annotations: + summary: "oauth2-proxy gate in {{ $labels.namespace }} exports no metrics" + description: >- + The {{ $labels.namespace }} namespace runs an oauth2-proxy gate container and + Prometheus has NO scrape target for it, so every authentication decision that gate + makes — every redirect to Keycloak, every 403 on a group check, every session that + silently fails to hold — is invisible to metrics and to alerting. This is not a + theoretical gap: on 2026-09-30 the jackett gate answered a burst of the app's own + XHR calls with 302s to Keycloak, re-established a session, and lost it again 26 + seconds later, and the only record anywhere was a line in the sidecar's stdout. + No alert could have fired, because there is nothing to alert on. Until a scrape + exists the runbook's LogQL queries are the substitute; the fix is + `--metrics-address=0.0.0.0:44180` on the sidecar plus a `metrics` port and a + ServiceMonitor, after which the auth-churn rule becomes writable. + runbook_url: "https://${BASE_DOMAIN}/runbooks/oauth2-proxy-auth-failures" diff --git a/platform/oauth2-proxy/kustomization.yaml b/platform/oauth2-proxy/kustomization.yaml index 3c80fe0..d35ecdf 100644 --- a/platform/oauth2-proxy/kustomization.yaml +++ b/platform/oauth2-proxy/kustomization.yaml @@ -14,3 +14,7 @@ resources: - oauth2-proxy-credentials.sops.yaml - httproute.yaml # forward-auth sign-in/callback ingress (was referenced in-place by platform-config) - servicemonitor.yaml # Prometheus scrape (was referenced in-place by platform-config) + # Alerting on whether the gates are OBSERVABLE at all. Travels with the servicemonitor above + # for the same reason it does: monitoring.coreos.com CRDs are registered by the `platform` + # tier, which is Ready before this tier's platform-config dependency completes. + - auth-visibility-alerts.yaml diff --git a/runbooks/README.md b/runbooks/README.md index 9c859fd..53436b9 100644 --- a/runbooks/README.md +++ b/runbooks/README.md @@ -51,6 +51,18 @@ rule groups still evaluate against nothing and **cannot fire**: | Flux (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` | | Hubble (`hubble_drop_total`) | `HubbleDeniedFlowSpike` | | etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all | +| per-app oauth2-proxy sidecars (`oauth2_proxy_*`) — measured 2026-09-30 | no authentication alerting exists at all for crafty, jackett, pihole, qbittorrent, radarr, sonarr | + +The oauth2-proxy row is the newest and the least obvious, because the scrape is not merely missing — +there is **nothing to scrape**. `count by (__name__) ({__name__=~"oauth2_proxy.*"})` on prod returns +exactly one family, `oauth2_proxy_requests_in_flight`, from the two replicas of the **shared** +forward-auth instance in the `oauth2-proxy` namespace. The six per-app gate sidecars run without +`--metrics-address`, so their metrics listener never starts; they declare no `metrics` container port; +and no ServiceMonitor or PodMonitor selects them. Any rule written against `oauth2_proxy_*` for those +namespaces would be inert on arrival. `OAuth2ProxyGateUnobserved` +(`platform/oauth2-proxy/auth-visibility-alerts.yaml`) alerts on that gap itself, using +`kube_pod_container_info` and `up` — two families that *are* scraped — and clears when a gate gains a +real scrape. See `oauth2-proxy-auth-failures`. Each affected runbook says so on its own page and gives the by-hand equivalent. See `targets-down`. @@ -108,6 +120,14 @@ Each affected runbook says so on its own page and gives the by-hand equivalent. - **`keycloak-db`** — Keycloak's CloudNativePG database is unavailable. - **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that every green signal missed. +- **`oauth2-proxy-auth-failures`** — a gated app "is up but does not work". Linked from + `OAuth2ProxyGateUnobserved`. The gate answers an unauthenticated **XHR** with a 302 to Keycloak + rather than a 401, the browser follows it silently against a live SSO session, and the call + resolves **200 carrying HTML** where the app's JavaScript wanted JSON — so the user is never + re-prompted and every uptime check stays green. Tells the gate's refusal from the group check + from the app's own login, from the sidecar log and from Loki, with the 2026-09-30 jackett trace + as the worked example. **Read the caveat**: the per-app gate sidecars export no metrics at all, + so these logs (72h of them) are the only record this class of failure leaves. ### Operator procedures (no alert links here) - **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only diff --git a/runbooks/kustomization.yaml b/runbooks/kustomization.yaml index 08793d7..e016017 100644 --- a/runbooks/kustomization.yaml +++ b/runbooks/kustomization.yaml @@ -85,6 +85,10 @@ configMapGenerator: - node-io=node-io.md - node-memory=node-memory.md - node-memory-swap=node-memory-swap.md + # Linked from OAuth2ProxyGateUnobserved, and the diagnostic procedure for any gated app + # that "is up but does not work" — the gate answers an unauthenticated XHR with a 302, + # not a 401, so the failure arrives as an HTTP 200 carrying the wrong body. + - oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md # Operator procedure, not an alert page (no alert links to it — the # alert-runbooks-resolvable check treats that as advisory). It is served # because the plex.tv claim window is short and is not a thing to go looking diff --git a/runbooks/oauth2-proxy-auth-failures.md b/runbooks/oauth2-proxy-auth-failures.md new file mode 100644 index 0000000..e667a02 --- /dev/null +++ b/runbooks/oauth2-proxy-auth-failures.md @@ -0,0 +1,171 @@ +# oauth2-proxy auth failures — telling the three layers apart + +Linked from `OAuth2ProxyGateUnobserved`, and the page to open whenever a gated app "is up but +does not work": a UI that loads and then shows nothing, a page that never finishes, a list that +stays empty, a save that silently does not save. The alert is about the *visibility* gap; this +page is the diagnostic procedure, and it works whether or not the alert ever fired. + +## What this means here + +Every app behind Keycloak on this platform has **three** places a request can be refused, and +from a browser they look identical — because two of the three answer with **HTTP 200**. + +| Layer | What refuses | What the browser sees | +|---|---|---| +| 1. The gate (oauth2-proxy sidecar) | no valid session cookie | **302 to Keycloak**, then usually **200** | +| 2. The gate's group check | session valid, wrong group | 403, or a redirect loop | +| 3. The app itself | the app's own login / API key | the app's own 401 or login page | + +**Layer 1 is the one that hides.** oauth2-proxy answers an unauthenticated request with a +redirect, not a 401 — correct for a browser navigation, disastrous for an **XHR**. The app's +JavaScript asks for JSON, the gate answers `302`, the browser follows it silently, the live +Keycloak SSO session satisfies it with no prompt, and the XHR resolves **200 with an HTML +dashboard in the body**. `fetch()` sees a success. The user sees an empty page. Reloading does +nothing, and nobody is ever asked to log in, because from the browser's point of view nothing +failed. + +The worked example, jackett on 2026-09-30, straight from the sidecar log: + + [oauthproxy.go:1026] No valid authentication in request. Initiating login. + GET "/api/v2.0/indexers/yts/config?_=..." HTTP/1.1 ... 302 400 + GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409 + ... ten of these at 15:30:53 + [AuthSuccess] Authenticated via OAuth2: Session{email:... created:2026-09-30 15:31:06 + expires:2026-09-30 15:36:06 refresh_token:true groups:[platform-admins]} + GET "/UI/Dashboard" 200 44379 + GET "/api/v2.0/indexers?_=..." 200 1501229 + ... and 26 seconds later, inside that same 5-minute session: + [oauthproxy.go:1026] No valid authentication in request. Initiating login. + +A session created at 15:31:06, valid until 15:36:06, not honoured at 15:31:32. That is the +signature of the session being **re-established over and over instead of held** — a cookie that +another proxy is overwriting, or a cookie that never comes back at all — not of Keycloak being +down and not of the app being broken. + +**Read this before you trust any silence:** the per-app gate sidecars export **no metrics**. They +run without `--metrics-address`, so oauth2-proxy's metrics listener never starts; they declare no +`metrics` port; and no ServiceMonitor or PodMonitor selects them. Only the shared +`oauth2-proxy` namespace instance is scraped, and it is not in the data path for any of these +apps. So there is nothing to graph and nothing that could have alerted. The logs below are not a +convenience — until that changes they are the **only** record that this class of failure happened. + +## First commands + +`kubectl` does not exist on the workstation; everything runs in the pinned container, and always +with an explicit context. + +**1. Which namespaces have an unobserved gate** (this is what the alert measures): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \ + -c prometheus -- /bin/promtool query instant http://localhost:9090 \ + "count by (namespace) (kube_pod_container_info{container=\"oauth2-proxy\"}) unless count by (namespace) (up{container=\"oauth2-proxy\"})"' +``` + +**2. The sidecar log — but you must grep it.** The gate's stdout is ~11,500 lines a day per app +and almost all of it is the kubelet health probe (`GET "/ping" ... 200`). A plain +`kubectl logs --tail=100` shows you nothing but `/ping` and looks perfectly healthy. Filter: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n jackett logs deploy/jackett -c oauth2-proxy --since=2h \ + | grep -v "/ping" | grep -E "No valid authentication|AuthSuccess|403|401|callback"' +``` + +Substitute the namespace: `crafty`, `jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`. + +**3. The same question across every gate at once, from Loki.** Alloy collects all of these, so +you can ask about the whole fleet and about a window that has already scrolled out of +`kubectl logs`. Port-forward Loki and query it: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/loki 13100:3100 & sleep 5; \ + curl -s -G --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"No valid authentication in request\" [24h]))" \ + http://127.0.0.1:13100/loki/api/v1/query | python3 -m json.tool | grep -E "namespace|value" ' +``` + +and the denominator, which is what makes the number mean something: + +``` + ... --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"AuthSuccess\" [24h]))" ... +``` + +**How to read the ratio.** A healthy sign-in is roughly **one or two** `No valid authentication` +lines per `AuthSuccess` — the first request of a session is unauthenticated by definition. A +ratio in the tens means sessions are not surviving. Measured on prod 2026-09-30 (24h): + +| namespace | `No valid authentication` | `AuthSuccess` | ratio | +|---|---|---|---| +| crafty | 81 | 1 | 81:1 | +| radarr | 67 | 1 | 67:1 | +| jackett | 39 | 4 | ~10:1 | +| pihole, qbittorrent, sonarr | 0 | 0 | (unused in the window) | + +All three of those are the failure, not the baseline. + +**4. Did the gate admit the user, or did the app refuse them?** Look for the `[AuthSuccess]` line +and what follows it. If `[AuthSuccess]` appears and the very next thing is the *app's own* login +page or the app's 401, the gate did its job and layer 3 is your problem — that is the Crafty +shape, and `runbooks/crafty-admin-credential.md` covers it. + +**5. Cookie names, when sessions do not survive.** Two oauth2-proxy instances emitting the +**same** cookie name on the **same** domain hold different cookie-secrets, so each overwrites the +other's cookie and the loser logs exactly the `No valid authentication` line above. List what +each gate is configured to emit (arguments only — never print a Secret's data): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'for ns in crafty jackett pihole qbittorrent radarr sonarr; do printf "%-12s " "$ns"; \ + kubectl --context k3s-prod -n $ns get deploy $ns \ + -o jsonpath="{range .spec.template.spec.containers[?(@.name==\"oauth2-proxy\")]}{.args}{end}" \ + | tr "," "\n" | grep -E "cookie-name|cookie-domain" | tr "\n" " "; echo; done' +``` + +A gate with no `--cookie-name` is using oauth2-proxy's default `_oauth2_proxy`. If two of them +share that default **and** a zone-wide `--cookie-domain=.home.keithjackson.dev`, they are +evicting each other, and the symptom is intermittent and depends on which app you visited last. + +## Likely causes, most likely first + +1. **Cookie collision between two gates on one domain.** Default cookie names plus a zone-wide + cookie domain. Signature: high `No valid authentication`-to-`AuthSuccess` ratio, sessions + dying seconds after being created, symptom depends on browsing order, and *nothing* wrong at + Keycloak. Fix is a unique `--cookie-name` and a host-scoped `--cookie-domain` per gate — see + `decisions/2026-09-29-crafty-oauth2-proxy-signin.md`, which found and fixed exactly this. +2. **The cookie is too large to come back.** With `--pass-access-token` the session carries the + access token; a fat Keycloak token pushes the cookie past 4 KB and it gets split into + `_oauth2_proxy_0`, `_oauth2_proxy_1`… If a chunk is dropped the session is unreadable and the + gate logs the same line. Look for numbered cookie names in the log, and for a session that + never survives a single navigation. +3. **The XHR path is not the navigation path.** The app's API is under a prefix the gate handles + differently, or the app issues cross-origin XHRs the cookie does not accompany. Signature: + `/UI/...` returns 200 and `/api/...` returns 302 in the *same second*, which is precisely the + jackett trace above. +4. **Session lifetime shorter than the app's polling interval,** with no refresh. The jackett + session above was a 5-minute session. If `refresh_token:true` is present and sessions still + die early, the refresh is failing, not missing. +5. **Group check, not authentication.** `--allowed-group` is a floor: a valid Keycloak session in + the wrong group is refused *after* a successful login, which reads as a loop. Look for 403 and + for the `groups:[...]` list on the `[AuthSuccess]` line. +6. **The app, not the gate.** `[AuthSuccess]` followed by the app's own login page or 401. + +## When to escalate or stop + +- **Stop and hand over if the fix is a manifest change.** Everything above is read-only + diagnosis. Changing a `--cookie-name`, adding `--metrics-address`, or editing an alert goes + through a branch and a PR; `kubectl edit` on a gate is drift that the next Flux reconcile + reverts, and in the meantime you have an unreproducible cluster. +- **Stop if you find yourself about to print a Secret.** The cookie-secret and client-secret live + in `oauth2-proxy-credentials` and the per-app equivalents. Compare *key names* and whether two + Secrets differ; never dump `.data` or `.stringData`. A password has been leaked into a + transcript here exactly this way. +- **Escalate to the platform owner if Keycloak itself is the suspect** — check `KeycloakDown` and + `KeycloakDatabaseDown` first (`runbooks/keycloak-down.md`, `runbooks/keycloak-db.md`). In the + 2026-09-30 incident Keycloak was healthy throughout and blaming it would have cost days. +- **The honest limit of this page.** With no metrics from the sidecars there is no history beyond + Loki's **72h** retention (`platform/monitoring/loki.yaml`, `retention_period: 72h`). If the + report is older than three days, the evidence is gone and the only remaining move is to + reproduce it. That is the cost of the gap `OAuth2ProxyGateUnobserved` is tracking, and it is + the reason to close it rather than silence it. From bce101d03f13c147b90fa958cc9a0ff40c9d589d Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Thu, 1 Oct 2026 21:56:07 -0500 Subject: [PATCH 2/3] feat(obs): wire the gate-sidecar scrape, and fix the alert that would have gone silent doing it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR #28 gave the five fleet oauth2-proxy sidecars --metrics-address=0.0.0.0:44180 and a named `metrics` port. This wires a scraper to them and, in the process, found a real bug in the alert this branch added. PodMonitor, not ServiceMonitor. Measured: no Service exposes 44180 on any of the five (jackett/radarr/sonarr/qbittorrent/pihole all publish only their app port + authproxy 4180), so a ServiceMonitor would select nothing. A PodMonitor targets pod ports directly and needs no change to catalog/render.py or the apps/wave-* renders and goldens. Selector is the pod label homelab.keithjackson.dev/auth-mode=forward-auth — present on exactly those five gate pods, absent from crafty's, which still has no metrics listener. The property, not an app list. THE BUG: `up{container="oauth2-proxy"}` -> `up{container="oauth2-proxy"} == 1`. A target that is discovered but FAILING still publishes `up`, with value 0. The series exists, so count by (namespace) returned 1, `unless` removed the namespace, and OAuth2ProxyGateUnobserved went QUIET while nothing was being scraped at all — this branch's own disease reproduced inside the alert written to cure it. Not hypothetical: the PodMonitor lands five targets whose scrape is dropped, so the bare form would have silently cleared on this very commit. Proven by mutation: reverting the `== 1` fails the two new unit-test scenarios (got: nil) and leaves every other assertion green. The scrape does NOT work yet, and that is documented rather than papered over. From a pod in ns monitoring to the jackett pod, :44180 AND :4180 both time out; the shared proxy's :44180 answers. ns jackett's default-deny is podSelector:{} with policyTypes:[Ingress,Egress] and NO ingress key (deny-all), and the one allowance is fromEntities:[ingress] to 4180 — the Gateway's Envoy only. The endpoint itself is healthy: port-forwarded, /metrics returns http=200, 10208 bytes, all three oauth2_proxy_* families including requests_total{code="302"}. Same bug and fingerprint as platform/longhorn/networkpolicy-metrics.yaml records for Longhorn; the fix is one narrow rule per app, and it cannot ship from this tier because oauth2-proxy and apps-wave-0 are Flux SIBLINGS (both dependsOn platform-config), so naming ns jackett here would apply today and fail on a cold rebuild. Needs an owner in the renderer. Touches conformance/** : one KNOWN_EXTERNAL_MONITORS entry, which is the mechanism the failing check explicitly instructs ("DECLARE it ... with who creates it") and which flux-podmonitor.yaml already uses. It records that the port name was measured on prod instead of resolved from a Service, and that the targets are currently down. That makes this PR require a non-author review on the sensitive-path rule. Also: ratio alert still NOT shipped. oauth2_proxy_requests_total now exists on the endpoint but not in the TSDB, so writing it today would make it inert for exactly the reason it was refused on 2026-09-30. Runbook gains policy-vs-no-target triage and the radarr /signalr/messages/negotiate worked case — one orphaned browser tab, 206 identical 302s, nobody ever re-prompted. Co-Authored-By: Claude Opus 5 (1M context) --- .../servicemonitor-ports-resolve/check.sh | 16 +++ ...09-30-oauth2-proxy-auth-failure-visible.md | 84 ++++++++++++++ platform/oauth2-proxy/README.md | 33 ++++++ .../auth-visibility-alerts.promtool-test.yaml | 62 +++++++++- .../oauth2-proxy/auth-visibility-alerts.yaml | 36 +++++- platform/oauth2-proxy/kustomization.yaml | 5 + platform/oauth2-proxy/podmonitor.yaml | 109 ++++++++++++++++++ runbooks/README.md | 38 ++++-- runbooks/oauth2-proxy-auth-failures.md | 87 ++++++++++++-- 9 files changed, 446 insertions(+), 24 deletions(-) create mode 100644 platform/oauth2-proxy/podmonitor.yaml diff --git a/conformance/tests/l0-static/servicemonitor-ports-resolve/check.sh b/conformance/tests/l0-static/servicemonitor-ports-resolve/check.sh index f720678..570c4ad 100755 --- a/conformance/tests/l0-static/servicemonitor-ports-resolve/check.sh +++ b/conformance/tests/l0-static/servicemonitor-ports-resolve/check.sh @@ -129,6 +129,22 @@ KNOWN_EXTERNAL_MONITORS: dict[str, str] = { "a PodMonitor, so it resolves against POD ports, not Service ports — the Flux controller " "Deployments (and their `http-prom` container port) are installed by the Flux bootstrap " "manifests, not authored here. Live on prod: job flux-system/flux-system, 6 targets up.", + "platform/oauth2-proxy/podmonitor.yaml:oauth2-proxy-sidecars": + "a PodMonitor, so it resolves against POD ports, not Service ports. Unlike the other " + "entries here the target is NOT chart-owned — the `metrics` (44180) container port is " + "authored in this repo, on the oauth2-proxy sidecar in apps/wave-0/{jackett,qbittorrent," + "pihole}.yaml and apps/wave-1/{radarr,sonarr}.yaml. It is unauditable here because NO " + "Service exposes 44180 BY DESIGN: adding one would mean changing catalog/render.py and " + "every apps/wave-* render and golden, to publish an endpoint only Prometheus should dial. " + "A PodMonitor needs no Service, which is the whole reason it was chosen. What assertion " + "(2) would have checked was instead MEASURED on prod 2026-10-01, read-only: the pods " + "declare ports [('http', 4180), ('metrics', 44180)], the selector label " + "homelab.keithjackson.dev/auth-mode=forward-auth is on exactly those five gate pods, and " + "port-forwarding 44180 returns http=200 with 10208 bytes of oauth2_proxy_* metrics. " + "CAVEAT, deliberately recorded: those targets are DOWN on prod, because the app " + "namespaces' default-deny NetworkPolicy drops the scrape from ns monitoring (same bug and " + "fingerprint as platform/longhorn/networkpolicy-metrics.yaml documents). The port name " + "resolves; the network path does not yet.", } MONITOR_KINDS = ("ServiceMonitor", "PodMonitor") diff --git a/decisions/2026-09-30-oauth2-proxy-auth-failure-visible.md b/decisions/2026-09-30-oauth2-proxy-auth-failure-visible.md index edf73ca..9a185c9 100644 --- a/decisions/2026-09-30-oauth2-proxy-auth-failure-visible.md +++ b/decisions/2026-09-30-oauth2-proxy-auth-failure-visible.md @@ -173,3 +173,87 @@ Everything below was measured against live prod through the pinned container, re 3. **Consider whether `--set-xauthrequest` apps should get `--api-routes`** so API paths answer 401 instead of 302. That is the fix that would make this failure *loud* rather than merely observable, and it belongs with whoever owns the sidecar args. + +--- + +# Round 2 — 2026-10-01: the scrape got wired, and found a bug in this branch's own alert + +Added to the same PR (#29) rather than a fourth concurrent branch. PR #28 merged in the meantime +and `main` moved three PRs ahead; `origin/main` is merged in here, and the `runbooks/kustomization.yaml` +conflict was resolved by **keeping both** runbook entries (mine and `oauth2-proxy-session`). + +## What changed + +**`platform/oauth2-proxy/podmonitor.yaml` (new).** PR #28 gave the five fleet sidecars +`--metrics-address=0.0.0.0:44180` and a named `metrics` port. No Service exposes 44180 on any of +them, so a ServiceMonitor would have nothing to select — a **PodMonitor** targets pod ports directly +and needs no renderer change. Selects on the pod label `homelab.keithjackson.dev/auth-mode: +forward-auth`, which is on exactly the five fleet gate pods and absent from crafty's: the property, +not an app list, so a sixth forward-auth app needs no edit. + +**`auth-visibility-alerts.yaml`: `up{container="oauth2-proxy"}` → `up{container="oauth2-proxy"} == 1`. +This is a real bug fix, and it is the most important thing in this round.** A target that is +discovered but failing still publishes `up`, with value **0**. The series exists, so +`count by (namespace)` returned 1, `unless` removed the namespace, and `OAuth2ProxyGateUnobserved` +would have **gone quiet while nothing was being scraped at all** — this branch's own disease, +reproduced inside the alert written to cure it. Not hypothetical: the PodMonitor lands five targets +whose scrape is dropped, so the bare form would have silently cleared on this very commit. + +**Two new unit-test scenarios** pinning that: a discovered-but-failing target must still fire, and a +target that works and then breaks must re-fire rather than latch cleared. + +**Docs corrected rather than appended to.** The 2026-09-30 claim "the sidecars export no metrics at +all" is now only half true and was rewritten in `runbooks/README.md`, `platform/oauth2-proxy/README.md` +and the alert header, each showing the two-step history, because each step looked like the end of it. +The runbook gains the policy-vs-no-target triage (step 1b) and the **radarr `/signalr` worked case**. + +**Deliberately NOT shipped: the ratio alert.** `oauth2_proxy_requests_total` now exists on the +endpoint, but not in the TSDB, because the scrape is blocked. Writing the ratio rule today would make +it inert for precisely the reason it was refused on 2026-09-30. It becomes writable the day a scrape +succeeds, not the day a listener exists. + +## What I verified + +- **The endpoint serves — by payload, not by flag.** Port-forwarded to the jackett pod's 44180: + `http=200 bytes=10208`, exporting `oauth2_proxy_requests_in_flight`, `oauth2_proxy_requests_total` + (`code="200" 3`, `code="302" 2`) and `oauth2_proxy_response_duration_seconds`. `--metrics-address` + being in `args` and `/metrics` returning a body are different claims; both now hold. +- **The scrape path is DROPPED, measured from the right place.** From grafana in ns `monitoring` + (beside Prometheus, same policy position) to the jackett pod IP: `:44180` → `wget: download timed + out`; `:4180` → **also** timed out; the shared proxy's `:44180` → 200 with `oauth2_proxy_*` present. + Cause, read from the objects: ns jackett's `default-deny` is `podSelector: {}`, + `policyTypes: [Ingress, Egress]`, with an `egress` rule and **no `ingress` key** (deny-all ingress); + the only allowance is CNP `jackett-allow-gateway-ingress`, `fromEntities: [ingress]` to port 4180. + The shared proxy is scrapable only because ns `oauth2-proxy` has **no NetworkPolicy at all**. +- **No Service exposes 44180** on any of the five (enumerated). +- **The pod label selector is exact:** `auth-mode=forward-auth` on all five fleet gate pods, absent + from crafty's. Crafty's sidecar still has no `--metrics-address` (args read in full). +- **Flux ordering forbids shipping the fix from here:** `oauth2-proxy dependsOn=['platform-config']` + and `apps-wave-0 dependsOn=['platform-config']` — siblings, no ordering. A NetworkPolicy naming ns + `jackett` from this tier would apply today and fail on a cold rebuild. +- **The `== 1` guard is non-vacuous — proven by mutation.** Reverting it to the bare `up` makes the + two new scenarios FAIL (`got: nil` at 1h5m and 2h5m) and leaves every other assertion passing; + restoring it returns the suite to SUCCESS under pinned promtool v3.14.0. +- **Gate:** see the next section for the exit code and skips. + +## What I did NOT verify + +- **The PodMonitor has never been applied.** That it discovers the five pods is reasoned from the + measured pod labels and port name, not observed — no target list has been read with it in place, + because applying it is an operator action. **Expect `up == 0` on all five when it lands**, which is + the documented state, not a regression. +- **I did not fix the NetworkPolicy, and this PR therefore does not make the gates observable.** It + makes them *discovered*, and makes the remaining gap loud instead of silent. The one narrow rule + per app (ns monitoring → `auth-mode=forward-auth` → TCP 44180, per + `platform/longhorn/networkpolicy-metrics.yaml`) has to be rendered with each app's own policy, in + the catalog/renderer that is in flight on another branch. **It needs an owner.** +- **Whether Prometheus will actually select a PodMonitor in ns `oauth2-proxy` targeting other + namespaces.** `namespaceSelector.any: true` plus `podMonitorSelectorNilUsesHelmValues: false` and + the fact that cross-namespace ServiceMonitors/PrometheusRules already work here say yes; it is not + measured with this object. +- **No session-lifecycle claim can be made from the current window.** `AuthSuccess` is **0** on both + jackett and radarr since the PR #28 restart, so nobody has signed in and `--cookie-refresh=2m30s` + has never executed a refresh. The cookie fix is unproven in the one dimension that matters most. +- **The radarr `/signalr` finding is the coordinator's measurement, not mine** — I reproduced neither + the 206/209 sample nor the jackett 0-in-13,612. They are recorded as reported. +- **L2/L3 conformance still did not run** (`CLUSTER_AVAILABLE` unset). Nothing is certified. diff --git a/platform/oauth2-proxy/README.md b/platform/oauth2-proxy/README.md index c2e2cf2..66a0ec6 100644 --- a/platform/oauth2-proxy/README.md +++ b/platform/oauth2-proxy/README.md @@ -124,3 +124,36 @@ and it reads the sidecar logs out of Loki (Alloy collects them; 72h retention). Service port, and a ServiceMonitor selecting it. Only then does the auth-churn rule — sustained `code="302"` as a share of `oauth2_proxy_requests_total`, held across a window longer than the session lifetime — become writable rather than decorative. + +### Update 2026-10-01: the listener exists; the scrape path does not + +PR #28 added `--metrics-address=0.0.0.0:44180` and a named `metrics` container port to the five +fleet sidecars (jackett, radarr, sonarr, qbittorrent, pihole). `podmonitor.yaml` in this directory +discovers them by the pod label `homelab.keithjackson.dev/auth-mode: forward-auth` — a **PodMonitor** +rather than a ServiceMonitor because no Service exposes 44180 and adding one would mean changing the +renderer. + +The endpoint is real: port-forwarded straight to the jackett pod, `/metrics` returns **http=200, +10208 bytes**, exporting `oauth2_proxy_requests_in_flight`, `oauth2_proxy_requests_total` (with +`code` labels) and `oauth2_proxy_response_duration_seconds`. That is verified by payload, not by the +flag appearing in `args` — a distinction worth keeping, because the two claims came apart here. + +**The scrape is nevertheless dropped.** From a pod in ns `monitoring`, both `:44180` and `:4180` on +the jackett pod time out; the shared proxy's `:44180` answers. The only difference is that ns +`oauth2-proxy` has **no NetworkPolicy at all**, while each app namespace has a `default-deny` whose +`policyTypes` include `Ingress` with no `ingress` rules (deny-all) plus a single CNP admitting +`fromEntities: [ingress]` to port 4180 — the Gateway's Envoy and nothing else. Prometheus matches no +clause. + +So the targets read `up == 0` and the TSDB is still empty for those namespaces. This is the same bug +and the same fingerprint as `platform/longhorn/networkpolicy-metrics.yaml` documents for Longhorn, +and the same narrow fix applies: ns `monitoring` → the gate pod label → TCP 44180 only. It cannot +ship from this directory, because `oauth2-proxy` and `apps-wave-0` are **siblings** in the Flux graph +(both `dependsOn: [platform-config]`), so a NetworkPolicy naming ns `jackett` would apply on today's +cluster and fail on a cold rebuild. It belongs with each app's own rendered policy. + +Consequence for `auth-visibility-alerts.yaml`: its `unless` clause requires `up == 1`, not merely a +present `up` series. A discovered-but-failing target publishes `up` with value 0, so the bare form +would have cleared `OAuth2ProxyGateUnobserved` the moment the PodMonitor landed — reporting the gates +as observed while nothing was being scraped. The unit test pins that scenario; deleting the `== 1` +fails it and nothing else. diff --git a/platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml b/platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml index 3fb9270..52cc681 100644 --- a/platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml +++ b/platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml @@ -104,7 +104,67 @@ tests: exp_samples: [] # --------------------------------------------------------------------------------------------- - # 3. NO GATE, NO ALERT. A namespace with no oauth2-proxy container must never appear, or the + # 3. THE REGRESSION GUARD FOR `== 1` — the scenario that exposed a real bug on 2026-10-01. + # + # A target is DISCOVERED but every scrape FAILS: `up` exists with value 0. This is not + # hypothetical — it is the exact state podmonitor.yaml creates today, because the app + # namespaces' default-deny drops the scrape (measured; see podmonitor.yaml). + # + # With the original `unless count by (namespace) (up{container="oauth2-proxy"})`, the `up` + # SERIES EXISTS, so `count by (namespace)` returns 1, `unless` removes the namespace, and the + # alert GOES QUIET while nothing is being scraped at all — a green signal standing in for an + # absent one, which is the precise disease this whole rule was written to cure. Deleting the + # `== 1` makes THIS test fail and every other test in this file still pass, which is the + # property that makes it worth having. + - interval: 1m + name: a discovered-but-FAILING target does not count as observed (up==0 must still fire) + input_series: + - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-6877fcd9f5-ht9d9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}' + values: '1x130' + # The PodMonitor target exists and never succeeds: NetworkPolicy drops the packets, so the + # scrape times out and `up` is published as 0 rather than going absent. + - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.0.189:44180",job="oauth2-proxy-sidecars",namespace="jackett",pod="jackett-6877fcd9f5-ht9d9"}' + values: '0x130' + promql_expr_test: + - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett"}' + eval_time: 65m + exp_samples: + - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}' + value: 1 + # And the inner clause itself yields NOTHING while the scrape is failing — which is the + # mechanism the alert depends on, asserted directly so a future reader can see why the + # `== 1` matters without re-deriving it. (Scenario 4 below covers the other direction: + # a succeeding scrape really does clear the alert, so `== 1` has not made the rule + # impossible to satisfy.) + - expr: 'count by (namespace) (up{container="oauth2-proxy"} == 1)' + eval_time: 65m + exp_samples: [] + + # --------------------------------------------------------------------------------------------- + # 4. A FLAPPING target: up goes 1 -> 0 and stays there. The alert must come BACK, not latch + # cleared. Covers the ordering bug where a once-observed namespace is permanently excluded. + - interval: 1m + name: a target that starts working and then fails re-fires + input_series: + - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="radarr",pod="radarr-7d55cbffb9-42hmh",service="kube-prometheus-stack-kube-state-metrics",uid="3333cccc-0000-0000-0000-000000000001"}' + values: '1x250' + # Healthy for the first 60m, then every scrape fails for the rest. + - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.77:44180",job="oauth2-proxy-sidecars",namespace="radarr",pod="radarr-7d55cbffb9-42hmh"}' + values: '1x60 0x180' + promql_expr_test: + # Quiet while it was working. + - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="radarr"}' + eval_time: 55m + exp_samples: [] + # Firing again an hour after it broke. + - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="radarr"}' + eval_time: 125m + exp_samples: + - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="radarr",severity="warning"}' + value: 1 + + # --------------------------------------------------------------------------------------------- + # 5. NO GATE, NO ALERT. A namespace with no oauth2-proxy container must never appear, or the # rule would page for every app on the platform the day someone widened the matcher. - interval: 1m name: namespaces without a gate sidecar are never reported diff --git a/platform/oauth2-proxy/auth-visibility-alerts.yaml b/platform/oauth2-proxy/auth-visibility-alerts.yaml index bd1d73b..7a1edef 100644 --- a/platform/oauth2-proxy/auth-visibility-alerts.yaml +++ b/platform/oauth2-proxy/auth-visibility-alerts.yaml @@ -33,10 +33,18 @@ # $ promtool query instant :9090 'count by (__name__) ({__name__=~"oauth2_proxy.*"})' # oauth2_proxy_requests_in_flight => 2 <- the SHARED proxy's two replicas. That is ALL. # -# Every sidecar (crafty, jackett, pihole, qbittorrent, radarr, sonarr) runs with +# PARTLY SUPERSEDED 2026-10-01, and the correction matters more than the original. PR #28 gave +# the five FLEET sidecars `--metrics-address=0.0.0.0:44180` and a named `metrics` port, and the +# endpoint genuinely serves (verified by payload: http=200, 10208 bytes, all three +# `oauth2_proxy_*` families present — not merely by the flag being in `args`). podmonitor.yaml +# in this directory now discovers them. But the scrape is still DROPPED by the app namespaces' +# default-deny ingress, so the TSDB remains empty for those namespaces; and crafty still has no +# metrics listener at all. The rule below is unchanged in purpose and still has work to do. +# +# Every sidecar (crafty, jackett, pihole, qbittorrent, radarr, sonarr) ran with # `--http-address=0.0.0.0:4180` and NO `--metrics-address`, so oauth2-proxy's metrics listener -# is never started; the container declares only port `http:4180`, there is no `metrics` port on -# any of their Services, and there is no ServiceMonitor or PodMonitor for any of them anywhere +# was never started; the container declared only port `http:4180`, there was no `metrics` port on +# any of their Services, and there was no ServiceMonitor or PodMonitor for any of them anywhere # in the cluster (measured: `kubectl get servicemonitors,podmonitors -A`). Six gates, zero # telemetry. An alert written against `oauth2_proxy_requests_total{namespace="jackett"}` would # have been PERMANENTLY INERT — present, evaluating, silent, and indistinguishable from health. @@ -81,15 +89,31 @@ spec: # Nothing in this expression touches `oauth2_proxy_*`, so the rule cannot itself be # inert for the reason it is reporting. # - # MEASURED ON PROD 2026-09-30 — this expression returns six namespaces right now: + # MEASURED ON PROD 2026-09-30 — this expression returned six namespaces: # {namespace="crafty"} {namespace="jackett"} {namespace="pihole"} # {namespace="qbittorrent"} {namespace="radarr"} {namespace="sonarr"} - # and excludes namespace="oauth2-proxy", whose two replicas ARE scraped. It fires on + # and excluded namespace="oauth2-proxy", whose two replicas ARE scraped. It fires on # the state the cluster is in; it is not a rule waiting for a hypothetical. + # + # `== 1` IS LOAD-BEARING, AND IT IS A BUG FIX (2026-10-01). This clause was originally + # a bare `up{container="oauth2-proxy"}`, which was WRONG in the one way that matters + # here: a target that is DISCOVERED BUT FAILING still publishes `up`, with value 0. So + # the series exists, `count by (namespace)` returns 1, `unless` removes the namespace, + # and the alert GOES QUIET while nothing whatsoever is being scraped. That is this + # branch's own disease — a green signal standing in for an absent one — reproduced + # inside the alert written to cure it. It was not hypothetical: podmonitor.yaml lands + # five targets whose scrape is DROPPED by NetworkPolicy (measured; see that file), so + # the bare form would have silently cleared on the very next commit. `== 1` means + # "observed" requires a scrape that actually succeeds. + # + # A failing target is therefore reported by BOTH this rule and kube-prometheus-stack's + # generic TargetDown. That duplication is deliberate: TargetDown says "a target is + # down", this says "an authentication gate is unobserved", and during an incident those + # are different sentences. The overlap is the cheap half of the trade. expr: | count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless - count by (namespace) (up{container="oauth2-proxy"}) + count by (namespace) (up{container="oauth2-proxy"} == 1) # 1h, not 5m: a rollout briefly has a running sidecar whose target has not been # discovered yet, and this condition is a standing architectural gap, never an # emergency. Long enough that no deploy can produce it. diff --git a/platform/oauth2-proxy/kustomization.yaml b/platform/oauth2-proxy/kustomization.yaml index d35ecdf..82b58a9 100644 --- a/platform/oauth2-proxy/kustomization.yaml +++ b/platform/oauth2-proxy/kustomization.yaml @@ -18,3 +18,8 @@ resources: # for the same reason it does: monitoring.coreos.com CRDs are registered by the `platform` # tier, which is Ready before this tier's platform-config dependency completes. - auth-visibility-alerts.yaml + # Discovers the PER-APP gate sidecars' :44180 metrics endpoints by pod label (PR #28 added the + # listener; no Service exposes it, so this is a PodMonitor, not a ServiceMonitor). The scrape is + # still denied by the app namespaces' default-deny ingress — read podmonitor.yaml's header + # before concluding the targets should be up. + - podmonitor.yaml diff --git a/platform/oauth2-proxy/podmonitor.yaml b/platform/oauth2-proxy/podmonitor.yaml new file mode 100644 index 0000000..f3e2247 --- /dev/null +++ b/platform/oauth2-proxy/podmonitor.yaml @@ -0,0 +1,109 @@ +--- +# PodMonitor for the PER-APP oauth2-proxy gate sidecars (R-OBS-04). +# +# ============================================================================================ +# WHY A PodMonitor AND NOT A ServiceMonitor +# ============================================================================================ +# PR #28 gave all five fleet sidecars `--metrics-address=0.0.0.0:44180` and a named `metrics` +# container port. It did NOT add a Service port, and measured on prod 2026-10-01 none of the five +# Services exposes 44180: +# +# jackett jackett: [('port-9117', 9117), ('authproxy', 4180)] +# radarr radarr: [('port-7878', 7878), ('authproxy', 4180)] +# sonarr sonarr: [('port-8989', 8989), ('authproxy', 4180)] +# qbittorrent qbittorrent: [('port-8080', 8080), ('authproxy', 4180)] +# pihole pihole: [('port-80', 80), ('authproxy', 4180)] +# +# A ServiceMonitor selects Service PORTS, so it would have nothing to select. A PodMonitor targets +# pod ports directly and bypasses Services entirely — so this needs no change to the Services, and +# therefore no change to `catalog/render.py` or the `apps/wave-*` renders and goldens. +# +# SELECTOR: the `homelab.keithjackson.dev/auth-mode: forward-auth` POD LABEL, not a namespace list. +# Measured 2026-10-01: that label is present on exactly the five fleet gate pods (jackett, radarr, +# sonarr, qbittorrent, pihole) and absent from crafty's, which is a separately-managed deployment +# (platform/crafty) that still has no `--metrics-address` at all. So the selector describes the +# PROPERTY being scraped rather than enumerating today's apps, and a sixth forward-auth app is +# picked up with no edit here. `namespaceSelector.any: true` because those five namespaces are +# created by the apps-wave-0/1 tiers, which this tier does not and must not depend on. +# +# ============================================================================================ +# READ THIS BEFORE BELIEVING THE TARGETS ARE UP: THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY +# ============================================================================================ +# This PodMonitor is NECESSARY AND NOT SUFFICIENT. The endpoint serves, and the path from ns +# monitoring is blocked. Both halves measured on prod 2026-10-01, read-only: +# +# THE ENDPOINT IS HEALTHY — port-forward straight to the jackett pod's 44180: +# GET /metrics -> http=200 bytes=10208 +# # TYPE oauth2_proxy_requests_in_flight gauge +# # TYPE oauth2_proxy_requests_total counter +# # TYPE oauth2_proxy_response_duration_seconds histogram +# oauth2_proxy_requests_total{code="200"} 3 +# oauth2_proxy_requests_total{code="302"} 2 +# +# THE SCRAPE PATH IS DROPPED — from a pod in ns monitoring (grafana, which sits beside +# Prometheus and so occupies the same policy position) to the jackett pod IP: +# http://10.0.0.189:44180/metrics -> wget: download timed out +# http://10.0.0.189:4180/ping -> wget: download timed out (even the "allowed" port) +# http://:44180/metrics -> 200, oauth2_proxy_* present (control: ns oauth2-proxy +# has NO NetworkPolicy, which is the only reason the +# shared instance has ever been scrapable) +# +# WHY: ns jackett's `default-deny` NetworkPolicy is `podSelector: {}` with +# `policyTypes: [Ingress, Egress]` and an `egress` rule ONLY — no `ingress` key, which in +# NetworkPolicy semantics is deny-all-ingress. The only ingress allowance is the CNP +# `jackett-allow-gateway-ingress`: `fromEntities: [ingress]` to port 4180 — the Gateway's Envoy +# and nothing else. Prometheus matches no clause and its packets are dropped, not refused. +# +# SO THE TARGETS BELOW WILL READ `up == 0` UNTIL AN INGRESS ALLOWANCE LANDS. That is a known, +# measured, documented state — NOT a regression introduced here, and NOT something to silence. +# It is deliberately left visible: `OAuth2ProxyGateUnobserved` in auth-visibility-alerts.yaml +# requires `up == 1`, precisely so a discovered-but-failing target does not read as "observed" +# (see that file's header — this exact trap is why the `== 1` is there). +# +# THIS IS THE SAME BUG LONGHORN HAD, and the platform has a documented fix idiom for it: +# platform/longhorn/networkpolicy-metrics.yaml, whose header records the identical fingerprint +# measured on 2026-09-13 — ServiceMonitor in place, 3 targets DISCOVERED, `up` 0/0/0, TSDB empty, +# `lastError: context deadline exceeded` (a drop, not a refusal). The fix there is a narrow +# additive `networking.k8s.io/v1` NetworkPolicy: ns monitoring -> the pod label, that one port. +# +# WHY THAT POLICY IS NOT IN THIS DIRECTORY. It would have to be five objects, one per app +# namespace, and those namespaces are created by apps-wave-0/1. Measured Flux graph, 2026-10-01: +# oauth2-proxy dependsOn=['platform-config'] <- this tier +# apps-wave-0 dependsOn=['platform-config'] <- SIBLING, no ordering between them +# apps-wave-1 dependsOn=['apps-wave-0'] +# They are siblings, so on a cold bootstrap there is no guarantee ns jackett exists when this tier +# applies, and a NetworkPolicy naming a missing namespace fails to apply. It would work on today's +# running cluster and break on a rebuild — the "works now, broken on restore" shape this repo +# keeps paying for. Longhorn's precedent is safe only because longhorn-system is created by the +# same tier that ships its policy. The allowance therefore belongs WITH THE APPS, rendered next to +# the rest of each app's network policy — i.e. in the catalog/renderer, which is in flight on +# another branch. Tracked in the handoff; it is one narrow rule per app, not a redesign. +apiVersion: monitoring.coreos.com/v1 +kind: PodMonitor +metadata: + name: oauth2-proxy-sidecars + namespace: oauth2-proxy + labels: + release: kube-prometheus-stack # kube-prometheus-stack's podMonitorSelector + app.kubernetes.io/name: oauth2-proxy + homelab.keithjackson.dev/purpose: metrics-scrape +spec: + namespaceSelector: + # The gate sidecars live in the app namespaces, not this one. `any` rather than a fixed list so + # a new forward-auth app needs no edit here — the pod label below is what scopes this. + any: true + selector: + matchLabels: + # The PROPERTY, not the app list. Present on exactly the five fleet gate pods (measured + # 2026-10-01); absent from crafty, which has no metrics listener yet. + homelab.keithjackson.dev/auth-mode: forward-auth + podMetricsEndpoints: + # By PORT NAME (`metrics`, 44180), never by number: the name is what PR #28 declared on the + # container, and a numeric port here would silently stop matching if the port were renumbered. + - port: metrics + path: /metrics + interval: 30s + # 10s, matching the drop signature above: a DENIED path does not refuse, it hangs until the + # scrape times out. Leaving the default would make each failing target hold a scrape slot for + # the full interval. + scrapeTimeout: 10s diff --git a/runbooks/README.md b/runbooks/README.md index 0eaaa5d..28f7be3 100644 --- a/runbooks/README.md +++ b/runbooks/README.md @@ -51,18 +51,36 @@ rule groups still evaluate against nothing and **cannot fire**: | Flux (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` | | Hubble (`hubble_drop_total`) | `HubbleDeniedFlowSpike` | | etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all | -| per-app oauth2-proxy sidecars (`oauth2_proxy_*`) — measured 2026-09-30 | no authentication alerting exists at all for crafty, jackett, pihole, qbittorrent, radarr, sonarr | +| per-app oauth2-proxy sidecars (`oauth2_proxy_*`) — updated 2026-10-01 | no authentication alerting exists at all for crafty, jackett, pihole, qbittorrent, radarr, sonarr | -The oauth2-proxy row is the newest and the least obvious, because the scrape is not merely missing — -there is **nothing to scrape**. `count by (__name__) ({__name__=~"oauth2_proxy.*"})` on prod returns +The oauth2-proxy row is the newest, and it has **moved twice in two days** — worth reading in order, +because each step looked like the end of it. + +**2026-09-30 — nothing to scrape.** `count by (__name__) ({__name__=~"oauth2_proxy.*"})` returned exactly one family, `oauth2_proxy_requests_in_flight`, from the two replicas of the **shared** -forward-auth instance in the `oauth2-proxy` namespace. The six per-app gate sidecars run without -`--metrics-address`, so their metrics listener never starts; they declare no `metrics` container port; -and no ServiceMonitor or PodMonitor selects them. Any rule written against `oauth2_proxy_*` for those -namespaces would be inert on arrival. `OAuth2ProxyGateUnobserved` -(`platform/oauth2-proxy/auth-visibility-alerts.yaml`) alerts on that gap itself, using -`kube_pod_container_info` and `up` — two families that *are* scraped — and clears when a gate gains a -real scrape. See `oauth2-proxy-auth-failures`. +forward-auth instance. All six per-app gate sidecars ran without `--metrics-address`, so their metrics +listener never started, and no ServiceMonitor or PodMonitor selected them. + +**2026-10-01 — the listener exists, and the scrape is still blocked.** PR #28 added +`--metrics-address=0.0.0.0:44180` and a named `metrics` port to the five fleet sidecars, and the +endpoint genuinely serves (port-forwarded: `http=200`, 10208 bytes, all three `oauth2_proxy_*` +families, `oauth2_proxy_requests_total{code="302"}` present). `platform/oauth2-proxy/podmonitor.yaml` +now discovers them by pod label. **But the app namespaces' `default-deny` NetworkPolicy drops the +scrape**: from a pod in ns `monitoring`, both `:44180` and `:4180` on the jackett pod time out, while +the shared proxy's `:44180` answers — the only difference being that ns `oauth2-proxy` has no +NetworkPolicy at all. So the targets are discovered and `up == 0`, and the TSDB is still empty for +those namespaces. **This is the same bug Longhorn had**, with the same fingerprint; the fix idiom is +`platform/longhorn/networkpolicy-metrics.yaml` (ns monitoring → one pod label → one port), and for +these apps it has to be rendered alongside each app's own policy rather than shipped from the +oauth2-proxy tier, which is a *sibling* of `apps-wave-0` in the Flux graph and so cannot rely on those +namespaces existing yet. + +`OAuth2ProxyGateUnobserved` (`platform/oauth2-proxy/auth-visibility-alerts.yaml`) alerts on the gap +itself from `kube_pod_container_info` and `up` — two families that *are* scraped. Note its `up == 1`: +a discovered-but-failing target publishes `up` with value **0**, so the earlier bare form would have +gone quiet the moment the PodMonitor landed, while nothing was being scraped. That near-miss is the +whole reason to distrust a cleared alert here. Crafty remains outside all of this — no metrics +listener at all. See `oauth2-proxy-auth-failures`. Each affected runbook says so on its own page and gives the by-hand equivalent. See `targets-down`. diff --git a/runbooks/oauth2-proxy-auth-failures.md b/runbooks/oauth2-proxy-auth-failures.md index e667a02..5547dd4 100644 --- a/runbooks/oauth2-proxy-auth-failures.md +++ b/runbooks/oauth2-proxy-auth-failures.md @@ -42,12 +42,14 @@ signature of the session being **re-established over and over instead of held** another proxy is overwriting, or a cookie that never comes back at all — not of Keycloak being down and not of the app being broken. -**Read this before you trust any silence:** the per-app gate sidecars export **no metrics**. They -run without `--metrics-address`, so oauth2-proxy's metrics listener never starts; they declare no -`metrics` port; and no ServiceMonitor or PodMonitor selects them. Only the shared -`oauth2-proxy` namespace instance is scraped, and it is not in the data path for any of these -apps. So there is nothing to graph and nothing that could have alerted. The logs below are not a -convenience — until that changes they are the **only** record that this class of failure happened. +**Read this before you trust any silence (updated 2026-10-01).** The five fleet gate sidecars now +*do* run a metrics listener on `:44180`, and `platform/oauth2-proxy/podmonitor.yaml` discovers +them — but **the scrape is dropped by the app namespaces' `default-deny` NetworkPolicy**, so those +targets read `up == 0` and the TSDB holds nothing for them. Crafty has no listener at all. Only +the shared `oauth2-proxy` namespace instance is genuinely scraped, and it is not in the data path +for any of these apps. So there is still nothing to graph and nothing that could have alerted. +**The logs below remain the only record that this class of failure happened.** Check which state +you are in before you trust a quiet dashboard — "First commands" step 1 tells you in one query. ## First commands @@ -60,9 +62,49 @@ with an explicit context. infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \ -c prometheus -- /bin/promtool query instant http://localhost:9090 \ - "count by (namespace) (kube_pod_container_info{container=\"oauth2-proxy\"}) unless count by (namespace) (up{container=\"oauth2-proxy\"})"' + "count by (namespace) (kube_pod_container_info{container=\"oauth2-proxy\"}) unless count by (namespace) (up{container=\"oauth2-proxy\"} == 1)"' ``` +**1b. If a namespace is listed, is it "no target" or "target failing"?** The two need different +fixes and the alert deliberately does not distinguish them: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \ + -c prometheus -- /bin/promtool query instant http://localhost:9090 \ + "up{container=\"oauth2-proxy\"}"' +``` + +No series for the namespace → nothing discovers it (no PodMonitor match; check the pod carries +`homelab.keithjackson.dev/auth-mode=forward-auth` and a named `metrics` port). A series with +value **0** → discovered but every scrape fails; get the reason from the target's `lastError`: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ + curl -s localhost:19090/api/v1/targets | python3 -c " +import json,sys +for t in json.load(sys.stdin)[\"data\"][\"activeTargets\"]: + if t[\"labels\"].get(\"container\")==\"oauth2-proxy\": + print(t[\"labels\"].get(\"namespace\"), t[\"health\"], t.get(\"lastError\",\"\")[:90])"' +``` + +`context deadline exceeded` is a **drop**, not a refusal — that is NetworkPolicy, not a dead +endpoint. Confirm the endpoint itself is fine by port-forwarding straight to the pod, which +bypasses policy entirely: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n jackett port-forward \ + $(kubectl --context k3s-prod -n jackett get pod -l app.kubernetes.io/name=jackett -o name | head -1) \ + 14418:44180 & sleep 5; curl -s -o /dev/null -w "http=%{http_code} bytes=%{size_download}\n" \ + http://127.0.0.1:14418/metrics' +``` + +`http=200` with a few KB means the listener is healthy and the path is what is broken. As of +2026-10-01 that is the state of all five fleet apps, and the fix is an ingress allowance from ns +`monitoring` to port 44180 — see `platform/longhorn/networkpolicy-metrics.yaml` for the idiom. + **2. The sidecar log — but you must grep it.** The gate's stdout is ~11,500 lines a day per app and almost all of it is the kubelet health probe (`GET "/ping" ... 200`). A plain `kubectl logs --tail=100` shows you nothing but `/ping` and looks perfectly healthy. Filter: @@ -105,6 +147,37 @@ ratio in the tens means sessions are not surviving. Measured on prod 2026-09-30 All three of those are the failure, not the baseline. +**The cleanest worked case of all — radarr, measured 2026-10-01.** After the cookie fix landed, +jackett dropped to **0** `No valid authentication` lines in 13,612 (from 39), and `removing +session` / `cookie signature not valid` went to **0 on both apps** (radarr had 8 and 8 before). +The eviction fingerprint is gone. But radarr still logged **206** rejections — and sampling 209 of +them, **every single one was the same request**: + + GET "/signalr/messages/negotiate" ... 302 <- browser UA, over and over + +That is one left-open browser tab. Its cookie is gone, its SignalR socket keeps trying to +reconnect, and every attempt is silently 302'd to Keycloak. Nobody is at the keyboard, so nobody +is ever re-prompted; the tab just sits there looking like a working page that has stopped +updating. **This is the entire failure class in one line**: a non-navigation request answered with +a redirect, forever, with no error surfaced anywhere. + +So when you read a rejection count, **always group by request path before concluding anything**: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n radarr logs deploy/radarr -c oauth2-proxy --since=24h \ + | grep -v "/ping" | grep "No valid authentication" -A 1 \ + | grep -oE "\"[^\"]+\" HTTP" | sort | uniq -c | sort -rn | head' +``` + +One path dominating means an orphaned client, not a broken gate — annoying but harmless, and it +clears when the tab is closed. Many different paths from one session means the session itself is +not surviving, which is the real defect. A per-app `--api-route` is the structural fix for the +first shape: it makes those paths answer **401** instead of 302, so the client fails loudly +instead of looping. Note that a count of **0 `AuthSuccess`** alongside a high rejection count +means nobody has signed in since the last restart — so `--cookie-refresh` has never run, and no +session-lifecycle conclusion can be drawn from that window at all. + **4. Did the gate admit the user, or did the app refuse them?** Look for the `[AuthSuccess]` line and what follows it. If `[AuthSuccess]` appears and the very next thing is the *app's own* login page or the app's 401, the gate did its job and layer 3 is your problem — that is the Crafty From 1e71b578d507315addc237bd565a5391f1eaf292 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Thu, 1 Oct 2026 22:12:19 -0500 Subject: [PATCH 3/3] =?UTF-8?q?docs(obs):=20record=20why=20CI=20is=20red?= =?UTF-8?q?=20=E2=80=94=20approval-policy=20on=20the=20one=20conformance?= =?UTF-8?q?=20entry?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The gate job reported failure while the local run was rc 3 with nothing failed. Measured cause is a single sensitive path (the KNOWN_EXTERNAL_MONITORS entry) with no non-author approving review, which ci/approval-policy.sh blocks fail-closed by design. Also records the general trap: approval-policy short-circuits to N/A off a pull_request event, so a clean LOCAL gate is a false reassurance for any branch touching a sensitive path. Only the PR run tells the truth. Co-Authored-By: Claude Opus 5 (1M context) --- ...09-30-oauth2-proxy-auth-failure-visible.md | 30 +++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/decisions/2026-09-30-oauth2-proxy-auth-failure-visible.md b/decisions/2026-09-30-oauth2-proxy-auth-failure-visible.md index 9a185c9..dab0441 100644 --- a/decisions/2026-09-30-oauth2-proxy-auth-failure-visible.md +++ b/decisions/2026-09-30-oauth2-proxy-auth-failure-visible.md @@ -257,3 +257,33 @@ succeeds, not the day a listener exists. - **The radarr `/signalr` finding is the coordinator's measurement, not mine** — I reproduced neither the 206/209 sample nor the jackett 0-in-13,612. They are recorded as reported. - **L2/L3 conformance still did not run** (`CLUSTER_AVAILABLE` unset). Nothing is certified. + +### Round 2 addendum — CI is RED, by design, and the local gate cannot see why + +After pushing round 2, `gate / gate (pull_request)` reported **failure after 1m42s** while the local +toolbox run was rc 3 with nothing failed. Measured cause, singular: + + sensitive paths touched: + conformance/tests/l0-static/servicemonitor-ports-resolve/check.sh (matched conformance/**) + approving reviews: (no reviews at all) author=gitea-admin + +`ci/approval-policy.sh` is fail-closed and exits 1 when a PR touches a `sensitive_paths` glob with no +`APPROVED` review from a non-author — "a change to the gate cannot be gated by the gate". The single +`KNOWN_EXTERNAL_MONITORS` entry is what trips it. **Nothing in L0/L1 failed.** + +**The local gate structurally cannot catch this, and said `approval-policy: green (N/A)`.** The stage +short-circuits to N/A when `GITHUB_EVENT_NAME` is unset, which it always is off-CI. So for any branch +touching a sensitive path, a clean local gate is a FALSE REASSURANCE — the PR run is the only honest +signal. That is a general trap, not specific to this change, and it is worth knowing before someone +concludes a red PR means broken code. + +To clear: a non-author approving review, then **re-run the gate** — approval does not re-trigger the +status check, so the required context stays red until the job runs again. `required_approvals=1` +already applied to every PR here, so the human was needed regardless; the conformance entry only +turns the tick red while that is pending. + +The alternative is to drop the register entry, which means dropping `podmonitor.yaml` with it (the +check fails without the declaration). I did not take it: the entry is the "declare the limit" +mechanism the check explicitly asks for, it is shrink-only so it cannot rot quietly, and +`flux-podmonitor.yaml` is the same case already in the register. But it is a real trade and the +reviewer should make it knowingly.