--- date: 2026-10-05 branch: docs/oauth2-sidecar-scrape-verified title: The gate-sidecar scrape is verified working on prod — and four places in the tree still said it wasn't --- ## What changed Documentation only. No manifest, no rendered output, no behaviour. PR #49 merged as `21c88b0` and the fix works, but four places in the tree still asserted the *pre-fix* state as current — in the present tense, in capitals, in the files a reader would consult first. This is the stale-documentation failure mode this repo has been burned by twice already (`git push origin main` in the runbooks, "append to DECISIONS-LOG.md" in the old handoffs), so: - **`platform/oauth2-proxy/podmonitor.yaml`** — the banner read *"READ THIS BEFORE BELIEVING THE TARGETS ARE UP: THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY"* and *"SO THE TARGETS BELOW WILL READ `up == 0` UNTIL AN INGRESS ALLOWANCE LANDS"*. Both now false, and the first one actively instructs the reader to disbelieve a correct `up == 1`. Replaced with a `STATUS: THE SCRAPE WORKS` block carrying the post-fix numbers; the diagnosis is kept below it under an explicit `HISTORY` heading, because it is why the fix is shaped the way it is. Added the drop fingerprint to recognise if it regresses — `lastScrapeDuration` pinned at *exactly* the `scrapeTimeout` with `context deadline exceeded`, versus 1–6 ms when healthy. - **`catalog/render.py:159`** — *"THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED … Do not read this flag as 'metrics are working'."* Rewritten to enumerate the three pieces a working scrape needs and which PR landed each. The warning is **kept, deliberately**: the flag alone really is insufficient, and "1 without 3" is exactly what caused the 63.7h outage. Only the "not yet scraped" claim was false. - **`catalog/render.py:1365`** — the sidecar arg comment, same correction, one line. - **`platform/oauth2-proxy/README.md`** — its two observability subsections are *dated* ("measured 2026-09-30", "Update 2026-10-01"), so they are honest records rather than lies. But a reader hits the stale body long before any correction, so a `> SUPERSEDED` pointer now sits directly under the heading, and an `### Update 2026-10-05` section at the end carries the before/after table, the TSDB series counts and the alert outcome. Two substantive errors fixed while there, not just tense: 1. The README's *"To close the gap for an app, three things are needed"* listed **a Service port and a ServiceMonitor**. Wrong — a PodMonitor targets pod ports directly, which is the whole reason no Service exposes 44180 and the renderer needed no Service change. Following that paragraph would have sent someone to edit `catalog/render.py`'s Service rendering for no reason. 2. `podmonitor.yaml` ended by deferring the fix to *"the catalog/renderer, which is in flight on another branch"* — that branch does not exist and never did; the work happened in PR #49. It now names `r_monitoring_ingress_policy` directly, and a new `CRAFTY` section explains why crafty is the one namespace this file cannot help. ## What I verified - **The fix itself, on prod 2026-10-05, read-only** (this is the evidence the docs now quote): Flux applied `21c88b0` through the real path — `apps-wave-0` `lastAppliedRevision` at 21:37:01Z, `apps-wave-1` at 21:37:46Z, in dependency order — and all five `-allow-monitoring-scrape` CNPs exist, each `ports=['44180']`, `from=[['monitoring']]`. - **All 6 targets `up == 1`**, `health=up`, `lastError=''`, `lastScrapeDuration` 0.001–0.006s — against 10.0s and `context deadline exceeded` before. - **`oauth2_proxy_*` is in the TSDB for these namespaces for the first time:** 5 metric names, 148 series; `sum by (namespace) (oauth2_proxy_requests_total)` = qbittorrent 35072, radarr 23560, sonarr 1284. (Queried with `count by (__name__)` plus a matcher, not label-values.) - **Alerts:** all 5 `TargetDown` and 5 of 6 `OAuth2ProxyGateUnobserved` resolved; one series remains, `OAuth2ProxyGateUnobserved{namespace="crafty"}`, firing. - **The rendered output is byte-identical.** `catalog/render.py` changed only in comments, so this branch touches no `apps/` file and no golden. Confirmed by `render-golden` and `l1` both passing with no regeneration, and by `git status` showing nothing under `apps/`. - **This branch touches no sensitive path**, unlike PR #49: `platform/oauth2-proxy/**`, `catalog/**` and `decisions/**` are none of them in `infra/gitea/approval-policy.json`. So it should merge on a green gate with no human approval — the opposite of the PR it documents, and the reason the two were kept apart. - `render.py --selftest`: PASS. - Toolbox gate: see the handoff's own rule — reported below rather than claimed here. ## What I did NOT verify - **I did not re-measure the 63.7h figure or the ~22 emails/day rate**; both are carried forward from the 2026-10-04 measurements, now quoted inside the docs as historical. The 15-day `query_range` that established "one episode, not flapping" was run once, on 2026-10-04. - **`oauth2_proxy_requests_total` is still absent for jackett and pihole.** I assert in the README that this is because the counter is not created until the gate serves its first request. That is the documented behaviour of a Prometheus counter with no initialisation, and it is consistent with the other three namespaces having traffic — but I did not force a request through jackett's gate and watch the series appear. If it is still absent after real use, that is a finding. - **The auth-churn rule is still unwritten.** The README now says it is "writable rather than decorative"; that is a claim about the metrics being present, not a rule anyone has authored or unit-tested. The thing the 2026-09-30 incident actually needed does not exist yet. - **crafty remains unobserved.** I read its container spec on 2026-10-05 and confirmed no `--metrics-address`; I did not attempt the fix, and nothing here changes its alert. - **I did not audit the whole tree for other stale "metrics are not scraped" claims** beyond the four sites a targeted grep found (`not yet reachable`, `UNTIL AN INGRESS`, `up == 0`, `not reachable by Prometheus` and neighbours). `runbooks/oauth2-proxy-auth-failures.md` is described in the README as "the by-hand substitute meanwhile" and I left it alone — the by-hand log procedure still works and is not wrong, but whether it should now be rewritten against the new metrics is unexamined.