jakman85andClaude Opus 5
81f627167d
PR #49 (merged 21c88b0) fixed the blocked scrape and it is verified working on prod
2026-10-05: all 6 targets up == 1, health=up, lastError='', lastScrapeDuration 0.001-0.006s
against 10.0s and "context deadline exceeded" before. oauth2_proxy_* is in the TSDB for
these namespaces for the first time (5 metric names, 148 series).
Four places in the tree still asserted the pre-fix state as current, in the present tense,
in the files a reader consults first:
platform/oauth2-proxy/podmonitor.yaml "READ THIS BEFORE BELIEVING THE TARGETS ARE UP:
THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY" and "SO THE TARGETS BELOW WILL READ
up == 0 UNTIL AN INGRESS ALLOWANCE LANDS". The first actively instructs the reader to
disbelieve a correct up == 1. Now a STATUS block with the post-fix numbers, with the
diagnosis kept below it under an explicit HISTORY heading because it is why the fix is
shaped the way it is.
catalog/render.py:159 "THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED".
catalog/render.py:1365 the same claim on the sidecar arg.
platform/oauth2-proxy/README.md its subsections are dated and so are honest records, but
a reader hits the stale body long before any correction. A SUPERSEDED pointer now sits
under the heading and an Update 2026-10-05 section carries the before/after.
The "not scraped" claims are corrected; the "a flag alone is not enough" warning is KEPT
deliberately, because that part is still true and "flag without ingress allowance" is
exactly what caused the 63.7h outage.
Two substantive errors fixed while there, not just tense. The README's "three things are
needed" listed a Service port and a ServiceMonitor — wrong, a PodMonitor targets pod ports
directly, which is why no Service exposes 44180 and the renderer needed no Service change;
following it would have sent someone to edit Service rendering for nothing. And
podmonitor.yaml deferred the fix to "the catalog/renderer, in flight on another branch" —
no such branch existed; it now names r_monitoring_ingress_policy, with a new CRAFTY section
explaining the one namespace this file cannot help.
Comments only in render.py, so the rendered output is byte-identical: no apps/ file and no
golden changes, and therefore no sensitive path and no human approval needed — the opposite
of the PR this documents.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
91 lines
6.4 KiB
Markdown
91 lines
6.4 KiB
Markdown
---
|
||
date: 2026-10-05
|
||
branch: docs/oauth2-sidecar-scrape-verified
|
||
title: The gate-sidecar scrape is verified working on prod — and four places in the tree still said it wasn't
|
||
---
|
||
|
||
## What changed
|
||
|
||
Documentation only. No manifest, no rendered output, no behaviour.
|
||
|
||
PR #49 merged as `21c88b0` and the fix works, but four places in the tree still asserted the
|
||
*pre-fix* state as current — in the present tense, in capitals, in the files a reader would consult
|
||
first. This is the stale-documentation failure mode this repo has been burned by twice already
|
||
(`git push origin main` in the runbooks, "append to DECISIONS-LOG.md" in the old handoffs), so:
|
||
|
||
- **`platform/oauth2-proxy/podmonitor.yaml`** — the banner read *"READ THIS BEFORE BELIEVING THE
|
||
TARGETS ARE UP: THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY"* and *"SO THE TARGETS BELOW WILL
|
||
READ `up == 0` UNTIL AN INGRESS ALLOWANCE LANDS"*. Both now false, and the first one actively
|
||
instructs the reader to disbelieve a correct `up == 1`. Replaced with a `STATUS: THE SCRAPE WORKS`
|
||
block carrying the post-fix numbers; the diagnosis is kept below it under an explicit `HISTORY`
|
||
heading, because it is why the fix is shaped the way it is. Added the drop fingerprint to
|
||
recognise if it regresses — `lastScrapeDuration` pinned at *exactly* the `scrapeTimeout` with
|
||
`context deadline exceeded`, versus 1–6 ms when healthy.
|
||
- **`catalog/render.py:159`** — *"THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED … Do not
|
||
read this flag as 'metrics are working'."* Rewritten to enumerate the three pieces a working
|
||
scrape needs and which PR landed each. The warning is **kept, deliberately**: the flag alone
|
||
really is insufficient, and "1 without 3" is exactly what caused the 63.7h outage. Only the
|
||
"not yet scraped" claim was false.
|
||
- **`catalog/render.py:1365`** — the sidecar arg comment, same correction, one line.
|
||
- **`platform/oauth2-proxy/README.md`** — its two observability subsections are *dated*
|
||
("measured 2026-09-30", "Update 2026-10-01"), so they are honest records rather than lies. But a
|
||
reader hits the stale body long before any correction, so a `> SUPERSEDED` pointer now sits
|
||
directly under the heading, and an `### Update 2026-10-05` section at the end carries the
|
||
before/after table, the TSDB series counts and the alert outcome.
|
||
|
||
Two substantive errors fixed while there, not just tense:
|
||
|
||
1. The README's *"To close the gap for an app, three things are needed"* listed **a Service port
|
||
and a ServiceMonitor**. Wrong — a PodMonitor targets pod ports directly, which is the whole
|
||
reason no Service exposes 44180 and the renderer needed no Service change. Following that
|
||
paragraph would have sent someone to edit `catalog/render.py`'s Service rendering for no reason.
|
||
2. `podmonitor.yaml` ended by deferring the fix to *"the catalog/renderer, which is in flight on
|
||
another branch"* — that branch does not exist and never did; the work happened in PR #49. It now
|
||
names `r_monitoring_ingress_policy` directly, and a new `CRAFTY` section explains why crafty is
|
||
the one namespace this file cannot help.
|
||
|
||
## What I verified
|
||
|
||
- **The fix itself, on prod 2026-10-05, read-only** (this is the evidence the docs now quote):
|
||
Flux applied `21c88b0` through the real path — `apps-wave-0` `lastAppliedRevision` at 21:37:01Z,
|
||
`apps-wave-1` at 21:37:46Z, in dependency order — and all five `<app>-allow-monitoring-scrape`
|
||
CNPs exist, each `ports=['44180']`, `from=[['monitoring']]`.
|
||
- **All 6 targets `up == 1`**, `health=up`, `lastError=''`, `lastScrapeDuration` 0.001–0.006s —
|
||
against 10.0s and `context deadline exceeded` before.
|
||
- **`oauth2_proxy_*` is in the TSDB for these namespaces for the first time:** 5 metric names,
|
||
148 series; `sum by (namespace) (oauth2_proxy_requests_total)` = qbittorrent 35072, radarr 23560,
|
||
sonarr 1284. (Queried with `count by (__name__)` plus a matcher, not label-values.)
|
||
- **Alerts:** all 5 `TargetDown` and 5 of 6 `OAuth2ProxyGateUnobserved` resolved; one series
|
||
remains, `OAuth2ProxyGateUnobserved{namespace="crafty"}`, firing.
|
||
- **The rendered output is byte-identical.** `catalog/render.py` changed only in comments, so this
|
||
branch touches no `apps/` file and no golden. Confirmed by `render-golden` and `l1` both passing
|
||
with no regeneration, and by `git status` showing nothing under `apps/`.
|
||
- **This branch touches no sensitive path**, unlike PR #49: `platform/oauth2-proxy/**`,
|
||
`catalog/**` and `decisions/**` are none of them in `infra/gitea/approval-policy.json`. So it
|
||
should merge on a green gate with no human approval — the opposite of the PR it documents, and
|
||
the reason the two were kept apart.
|
||
- `render.py --selftest`: PASS.
|
||
- Toolbox gate: see the handoff's own rule — reported below rather than claimed here.
|
||
|
||
## What I did NOT verify
|
||
|
||
- **I did not re-measure the 63.7h figure or the ~22 emails/day rate**; both are carried forward
|
||
from the 2026-10-04 measurements, now quoted inside the docs as historical. The 15-day
|
||
`query_range` that established "one episode, not flapping" was run once, on 2026-10-04.
|
||
- **`oauth2_proxy_requests_total` is still absent for jackett and pihole.** I assert in the README
|
||
that this is because the counter is not created until the gate serves its first request. That is
|
||
the documented behaviour of a Prometheus counter with no initialisation, and it is consistent
|
||
with the other three namespaces having traffic — but I did not force a request through jackett's
|
||
gate and watch the series appear. If it is still absent after real use, that is a finding.
|
||
- **The auth-churn rule is still unwritten.** The README now says it is "writable rather than
|
||
decorative"; that is a claim about the metrics being present, not a rule anyone has authored or
|
||
unit-tested. The thing the 2026-09-30 incident actually needed does not exist yet.
|
||
- **crafty remains unobserved.** I read its container spec on 2026-10-05 and confirmed no
|
||
`--metrics-address`; I did not attempt the fix, and nothing here changes its alert.
|
||
- **I did not audit the whole tree for other stale "metrics are not scraped" claims** beyond the
|
||
four sites a targeted grep found (`not yet reachable`, `UNTIL AN INGRESS`, `up == 0`,
|
||
`not reachable by Prometheus` and neighbours). `runbooks/oauth2-proxy-auth-failures.md` is
|
||
described in the README as "the by-hand substitute meanwhile" and I left it alone — the by-hand
|
||
log procedure still works and is not wrong, but whether it should now be rewritten against the
|
||
new metrics is unexamined.
|