docs(obs): record that the gate-sidecar scrape works — four places still said it did not #54

2026-10-06T02:21:01Z

gitea-admin commented

2026-10-06 02:21:01 +00:00

Documentation only. No manifest, no rendered output, no behaviour change.

PR #49 fixed the blocked gate-sidecar scrape and it is verified working on prod 2026-10-05 — but four places in the tree still asserted the pre-fix state as current, in the present tense, in the files a reader consults first.

The verification the docs now quote

Flux applied 21c88b0 through the real path (apps-wave-0 lastAppliedRevision 21:37:01Z, apps-wave-1 21:37:46Z, in dependency order), and all five <app>-allow-monitoring-scrape CNPs exist with ports=['44180'], from=[['monitoring']].

	before	after
target health (6)	`down`	`up`
`lastError`	`context deadline exceeded`	`''`
`lastScrapeDuration`	10.0s — exactly the `scrapeTimeout`	0.001–0.006s
`up`	`0` × 6	`1` × 6

oauth2_proxy_* is in the TSDB for these namespaces for the first time: 5 metric names, 148 series; sum by (namespace) (oauth2_proxy_requests_total) = qbittorrent 35072, radarr 23560, sonarr 1284. All 5 TargetDown and 5 of 6 OAuth2ProxyGateUnobserved resolved — ~22 emails/day down to 2/day.

What was stale

platform/oauth2-proxy/podmonitor.yaml — "READ THIS BEFORE BELIEVING THE TARGETS ARE UP: THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY" and "SO THE TARGETS BELOW WILL READ up == 0 UNTIL AN INGRESS ALLOWANCE LANDS". The first actively instructs the reader to disbelieve a correct up == 1. Now a STATUS block with the post-fix numbers; the diagnosis is kept below it under an explicit HISTORY heading, because it is why the fix is shaped the way it is. Added the drop fingerprint for recognising a regression: lastScrapeDuration pinned at exactly the scrapeTimeout, versus 1–6 ms when healthy.
catalog/render.py:159 and :1365 — "THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED." Rewritten to enumerate the three pieces a working scrape needs and which PR landed each.
platform/oauth2-proxy/README.md — its subsections are dated, so they are honest records rather than lies, but a reader hits the stale body long before any correction. A > SUPERSEDED pointer now sits under the heading, with an ### Update 2026-10-05 section at the end.

The "a flag alone is not enough" warning is kept deliberately. That part is still true, and "flag without ingress allowance" is exactly what produced the 63.7h outage. Only the "not yet scraped" claim was false.

Two substantive errors fixed, not just tense

The README's "To close the gap for an app, three things are needed" listed a Service port and a ServiceMonitor. Wrong — a PodMonitor targets pod ports directly, which is why no Service exposes 44180 and why the renderer needed no Service change. Following that paragraph would have sent someone to edit Service rendering for no reason.
podmonitor.yaml deferred the fix to "the catalog/renderer, which is in flight on another branch" — no such branch existed. It now names r_monitoring_ingress_policy, plus a new CRAFTY section explaining the one namespace this file cannot help.

Gate

rc 3 — UNDETERMINED, not 0. Every stage that ran is green; the sole not-run stage is gitea-settings (no GITEA_SETTINGS_TOKEN locally). render-golden and l1 pass with no golden regeneration, which is the confirmation that the render.py edits are comments-only and left rendered output byte-identical.

Consequently this branch touches no sensitive path — platform/oauth2-proxy/**, catalog/** and decisions/** are none of them in approval-policy.json — so unlike PR #49 it should merge on a green gate alone. That is why the two were kept apart.

Note on the 2026-10-04 handoff entry

Its What I did NOT verify section led with "that the scrape now succeeds". Rather than rewrite a merged record, the original text is left exactly as written and a dated forward pointer was added above it. decisions/2026-10-05-oauth2-sidecar-scrape-verified.md carries the measurements.

Not verified

The 63.7h figure and the ~22 emails/day rate are carried forward from 2026-10-04, now quoted as historical; not re-measured.
oauth2_proxy_requests_total is still absent for jackett and pihole. I assert this is because the counter is not created until the gate serves its first request — consistent with the other three having traffic, but I did not force a request through and watch the series appear.
The auth-churn rule is still unwritten. The README now says it is "writable rather than decorative"; that is a claim about metrics being present, not a rule anyone has authored or unit-tested.
crafty remains unobserved; I confirmed it has no --metrics-address and did not attempt the fix.
I did not audit the whole tree beyond the four sites a targeted grep found. runbooks/oauth2-proxy-auth-failures.md is left alone — its by-hand log procedure still works, but whether it should now be rewritten against the new metrics is unexamined.

🤖 Generated with Claude Code

Documentation only. No manifest, no rendered output, no behaviour change. PR #49 fixed the blocked gate-sidecar scrape and it is **verified working on prod 2026-10-05** — but four places in the tree still asserted the *pre-fix* state as current, in the present tense, in the files a reader consults first. ## The verification the docs now quote Flux applied `21c88b0` through the real path (`apps-wave-0` `lastAppliedRevision` 21:37:01Z, `apps-wave-1` 21:37:46Z, in dependency order), and all five `<app>-allow-monitoring-scrape` CNPs exist with `ports=['44180']`, `from=[['monitoring']]`. | | before | after | |---|---|---| | target health (6) | `down` | `up` | | `lastError` | `context deadline exceeded` | `''` | | `lastScrapeDuration` | **10.0s** — exactly the `scrapeTimeout` | **0.001–0.006s** | | `up` | `0` × 6 | `1` × 6 | `oauth2_proxy_*` is in the TSDB for these namespaces for the first time: 5 metric names, 148 series; `sum by (namespace) (oauth2_proxy_requests_total)` = qbittorrent 35072, radarr 23560, sonarr 1284. All 5 `TargetDown` and 5 of 6 `OAuth2ProxyGateUnobserved` resolved — ~22 emails/day down to 2/day. ## What was stale - **`platform/oauth2-proxy/podmonitor.yaml`** — *"READ THIS BEFORE BELIEVING THE TARGETS ARE UP: THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY"* and *"SO THE TARGETS BELOW WILL READ `up == 0` UNTIL AN INGRESS ALLOWANCE LANDS"*. The first actively instructs the reader to **disbelieve a correct `up == 1`**. Now a `STATUS` block with the post-fix numbers; the diagnosis is kept below it under an explicit `HISTORY` heading, because it is why the fix is shaped the way it is. Added the drop fingerprint for recognising a regression: `lastScrapeDuration` pinned at *exactly* the `scrapeTimeout`, versus 1–6 ms when healthy. - **`catalog/render.py:159` and `:1365`** — *"THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED."* Rewritten to enumerate the three pieces a working scrape needs and which PR landed each. - **`platform/oauth2-proxy/README.md`** — its subsections are *dated*, so they are honest records rather than lies, but a reader hits the stale body long before any correction. A `> SUPERSEDED` pointer now sits under the heading, with an `### Update 2026-10-05` section at the end. **The "a flag alone is not enough" warning is kept deliberately.** That part is still true, and "flag without ingress allowance" is exactly what produced the 63.7h outage. Only the *"not yet scraped"* claim was false. ## Two substantive errors fixed, not just tense 1. The README's *"To close the gap for an app, three things are needed"* listed **a Service port and a ServiceMonitor**. Wrong — a PodMonitor targets pod ports directly, which is why no Service exposes 44180 and why the renderer needed no Service change. Following that paragraph would have sent someone to edit Service rendering for no reason. 2. `podmonitor.yaml` deferred the fix to *"the catalog/renderer, which is in flight on another branch"* — no such branch existed. It now names `r_monitoring_ingress_policy`, plus a new `CRAFTY` section explaining the one namespace this file cannot help. ## Gate **rc 3 — UNDETERMINED, not 0.** Every stage that ran is green; the sole not-run stage is `gitea-settings` (no `GITEA_SETTINGS_TOKEN` locally). `render-golden` and `l1` pass **with no golden regeneration**, which is the confirmation that the `render.py` edits are comments-only and left rendered output byte-identical. Consequently this branch touches **no sensitive path** — `platform/oauth2-proxy/**`, `catalog/**` and `decisions/**` are none of them in `approval-policy.json` — so unlike PR #49 it should merge on a green gate alone. That is why the two were kept apart. ## Note on the 2026-10-04 handoff entry Its *What I did NOT verify* section led with "that the scrape now succeeds". Rather than rewrite a merged record, the original text is left exactly as written and a dated forward pointer was added above it. `decisions/2026-10-05-oauth2-sidecar-scrape-verified.md` carries the measurements. ## Not verified - The 63.7h figure and the ~22 emails/day rate are **carried forward** from 2026-10-04, now quoted as historical; not re-measured. - `oauth2_proxy_requests_total` is still absent for jackett and pihole. I assert this is because the counter is not created until the gate serves its first request — consistent with the other three having traffic, but I did not force a request through and watch the series appear. - The auth-churn rule is **still unwritten**. The README now says it is "writable rather than decorative"; that is a claim about metrics being present, not a rule anyone has authored or unit-tested. - `crafty` remains unobserved; I confirmed it has no `--metrics-address` and did not attempt the fix. - I did not audit the whole tree beyond the four sites a targeted grep found. `runbooks/oauth2-proxy-auth-failures.md` is left alone — its by-hand log procedure still works, but whether it should now be rewritten against the new metrics is unexamined. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea-admin added 1 commit 2026-10-06 02:22:51 +00:00

docs(obs): record that the gate-sidecar scrape works — four places still said it did not

gate / gate (pull_request) Successful in 1m48s

Details

gate / conformance (pull_request) Has been skipped

Details

81f627167d

PR #49 (merged 21c88b0) fixed the blocked scrape and it is verified working on prod
2026-10-05: all 6 targets up == 1, health=up, lastError='', lastScrapeDuration 0.001-0.006s
against 10.0s and "context deadline exceeded" before. oauth2_proxy_* is in the TSDB for
these namespaces for the first time (5 metric names, 148 series).

Four places in the tree still asserted the pre-fix state as current, in the present tense,
in the files a reader consults first:

  platform/oauth2-proxy/podmonitor.yaml  "READ THIS BEFORE BELIEVING THE TARGETS ARE UP:
    THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY" and "SO THE TARGETS BELOW WILL READ
    up == 0 UNTIL AN INGRESS ALLOWANCE LANDS". The first actively instructs the reader to
    disbelieve a correct up == 1. Now a STATUS block with the post-fix numbers, with the
    diagnosis kept below it under an explicit HISTORY heading because it is why the fix is
    shaped the way it is.
  catalog/render.py:159  "THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED".
  catalog/render.py:1365  the same claim on the sidecar arg.
  platform/oauth2-proxy/README.md  its subsections are dated and so are honest records, but
    a reader hits the stale body long before any correction. A SUPERSEDED pointer now sits
    under the heading and an Update 2026-10-05 section carries the before/after.

The "not scraped" claims are corrected; the "a flag alone is not enough" warning is KEPT
deliberately, because that part is still true and "flag without ingress allowance" is
exactly what caused the 63.7h outage.

Two substantive errors fixed while there, not just tense. The README's "three things are
needed" listed a Service port and a ServiceMonitor — wrong, a PodMonitor targets pod ports
directly, which is why no Service exposes 44180 and the renderer needed no Service change;
following it would have sent someone to edit Service rendering for nothing. And
podmonitor.yaml deferred the fix to "the catalog/renderer, in flight on another branch" —
no such branch existed; it now names r_monitoring_ingress_policy, with a new CRAFTY section
explaining the one namespace this file cannot help.

Comments only in render.py, so the rendered output is byte-identical: no apps/ file and no
golden changes, and therefore no sensitive path and no human approval needed — the opposite
of the PR this documents.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

jakman85 closed this pull request

2026-10-06 02:31:03 +00:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Home-Network/home-network-k3s-impl#54