home-network-k3s-impl/decisions/2026-10-05-oauth2-sidecar-scrape-verified.md

date, branch, title

date	branch	title
2026-10-05	docs/oauth2-sidecar-scrape-verified	The gate-sidecar scrape is verified working on prod — and four places in the tree still said it wasn't

What changed

Documentation only. No manifest, no rendered output, no behaviour.

PR #49 merged as 21c88b0 and the fix works, but four places in the tree still asserted the pre-fix state as current — in the present tense, in capitals, in the files a reader would consult first. This is the stale-documentation failure mode this repo has been burned by twice already (git push origin main in the runbooks, "append to DECISIONS-LOG.md" in the old handoffs), so:

• platform/oauth2-proxy/podmonitor.yaml — the banner read "READ THIS BEFORE BELIEVING THE TARGETS ARE UP: THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY" and "SO THE TARGETS BELOW WILL READ up == 0 UNTIL AN INGRESS ALLOWANCE LANDS". Both now false, and the first one actively instructs the reader to disbelieve a correct up == 1. Replaced with a STATUS: THE SCRAPE WORKS block carrying the post-fix numbers; the diagnosis is kept below it under an explicit HISTORY heading, because it is why the fix is shaped the way it is. Added the drop fingerprint to recognise if it regresses — lastScrapeDuration pinned at exactly the scrapeTimeout with context deadline exceeded, versus 1–6 ms when healthy.
• catalog/render.py:159 — "THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED … Do not read this flag as 'metrics are working'." Rewritten to enumerate the three pieces a working scrape needs and which PR landed each. The warning is kept, deliberately: the flag alone really is insufficient, and "1 without 3" is exactly what caused the 63.7h outage. Only the "not yet scraped" claim was false.
• catalog/render.py:1365 — the sidecar arg comment, same correction, one line.
• platform/oauth2-proxy/README.md — its two observability subsections are dated ("measured 2026-09-30", "Update 2026-10-01"), so they are honest records rather than lies. But a reader hits the stale body long before any correction, so a > SUPERSEDED pointer now sits directly under the heading, and an ### Update 2026-10-05 section at the end carries the before/after table, the TSDB series counts and the alert outcome.

Two substantive errors fixed while there, not just tense:

1. The README's "To close the gap for an app, three things are needed" listed a Service port and a ServiceMonitor. Wrong — a PodMonitor targets pod ports directly, which is the whole reason no Service exposes 44180 and the renderer needed no Service change. Following that paragraph would have sent someone to edit catalog/render.py's Service rendering for no reason.
2. podmonitor.yaml ended by deferring the fix to "the catalog/renderer, which is in flight on another branch" — that branch does not exist and never did; the work happened in PR #49. It now names r_monitoring_ingress_policy directly, and a new CRAFTY section explains why crafty is the one namespace this file cannot help.

What I verified

• The fix itself, on prod 2026-10-05, read-only (this is the evidence the docs now quote): Flux applied 21c88b0 through the real path — apps-wave-0 lastAppliedRevision at 21:37:01Z, apps-wave-1 at 21:37:46Z, in dependency order — and all five <app>-allow-monitoring-scrape CNPs exist, each ports=['44180'], from=[['monitoring']].
• All 6 targets up == 1, health=up, lastError='', lastScrapeDuration 0.001–0.006s — against 10.0s and context deadline exceeded before.
• oauth2_proxy_* is in the TSDB for these namespaces for the first time: 5 metric names, 148 series; sum by (namespace) (oauth2_proxy_requests_total) = qbittorrent 35072, radarr 23560, sonarr 1284. (Queried with count by (__name__) plus a matcher, not label-values.)
• Alerts: all 5 TargetDown and 5 of 6 OAuth2ProxyGateUnobserved resolved; one series remains, OAuth2ProxyGateUnobserved{namespace="crafty"}, firing.
• The rendered output is byte-identical. catalog/render.py changed only in comments, so this branch touches no apps/ file and no golden. Confirmed by render-golden and l1 both passing with no regeneration, and by git status showing nothing under apps/.
• This branch touches no sensitive path, unlike PR #49: platform/oauth2-proxy/**, catalog/** and decisions/** are none of them in infra/gitea/approval-policy.json. So it should merge on a green gate with no human approval — the opposite of the PR it documents, and the reason the two were kept apart.
• render.py --selftest: PASS.
• Toolbox gate: see the handoff's own rule — reported below rather than claimed here.

What I did NOT verify

• I did not re-measure the 63.7h figure or the ~22 emails/day rate; both are carried forward from the 2026-10-04 measurements, now quoted inside the docs as historical. The 15-day query_range that established "one episode, not flapping" was run once, on 2026-10-04.
• oauth2_proxy_requests_total is still absent for jackett and pihole. I assert in the README that this is because the counter is not created until the gate serves its first request. That is the documented behaviour of a Prometheus counter with no initialisation, and it is consistent with the other three namespaces having traffic — but I did not force a request through jackett's gate and watch the series appear. If it is still absent after real use, that is a finding.
• The auth-churn rule is still unwritten. The README now says it is "writable rather than decorative"; that is a claim about the metrics being present, not a rule anyone has authored or unit-tested. The thing the 2026-09-30 incident actually needed does not exist yet.
• crafty remains unobserved. I read its container spec on 2026-10-05 and confirmed no --metrics-address; I did not attempt the fix, and nothing here changes its alert.
• I did not audit the whole tree for other stale "metrics are not scraped" claims beyond the four sites a targeted grep found (not yet reachable, UNTIL AN INGRESS, up == 0, not reachable by Prometheus and neighbours). runbooks/oauth2-proxy-auth-failures.md is described in the README as "the by-hand substitute meanwhile" and I left it alone — the by-hand log procedure still works and is not wrong, but whether it should now be rewritten against the new metrics is unexamined.