docs(obs): record that the gate-sidecar scrape works — four places still said it did not
PR #49 (merged 21c88b0) fixed the blocked scrape and it is verified working on prod
2026-10-05: all 6 targets up == 1, health=up, lastError='', lastScrapeDuration 0.001-0.006s
against 10.0s and "context deadline exceeded" before. oauth2_proxy_* is in the TSDB for
these namespaces for the first time (5 metric names, 148 series).
Four places in the tree still asserted the pre-fix state as current, in the present tense,
in the files a reader consults first:
platform/oauth2-proxy/podmonitor.yaml "READ THIS BEFORE BELIEVING THE TARGETS ARE UP:
THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY" and "SO THE TARGETS BELOW WILL READ
up == 0 UNTIL AN INGRESS ALLOWANCE LANDS". The first actively instructs the reader to
disbelieve a correct up == 1. Now a STATUS block with the post-fix numbers, with the
diagnosis kept below it under an explicit HISTORY heading because it is why the fix is
shaped the way it is.
catalog/render.py:159 "THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED".
catalog/render.py:1365 the same claim on the sidecar arg.
platform/oauth2-proxy/README.md its subsections are dated and so are honest records, but
a reader hits the stale body long before any correction. A SUPERSEDED pointer now sits
under the heading and an Update 2026-10-05 section carries the before/after.
The "not scraped" claims are corrected; the "a flag alone is not enough" warning is KEPT
deliberately, because that part is still true and "flag without ingress allowance" is
exactly what caused the 63.7h outage.
Two substantive errors fixed while there, not just tense. The README's "three things are
needed" listed a Service port and a ServiceMonitor — wrong, a PodMonitor targets pod ports
directly, which is why no Service exposes 44180 and the renderer needed no Service change;
following it would have sent someone to edit Service rendering for nothing. And
podmonitor.yaml deferred the fix to "the catalog/renderer, in flight on another branch" —
no such branch existed; it now names r_monitoring_ingress_policy, with a new CRAFTY section
explaining the one namespace this file cannot help.
Comments only in render.py, so the rendered output is byte-identical: no apps/ file and no
golden changes, and therefore no sensitive path and no human approval needed — the opposite
of the PR this documents.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
4386d1453b
commit
81f627167d
5 files changed
+208
-16
No files matched your search
@@ -91,6 +91,10 @@ This proves the R-IDP-02 property (unauthenticated cannot reach the upstream) on
|
||||
|
||||
## Observability: what this component can and cannot see (measured 2026-09-30)
|
||||
|
||||
> **SUPERSEDED — read "Update 2026-10-05" at the end of this section first.** The gate sidecars
|
||||
> **are** scraped now and the TSDB **does** hold `oauth2_proxy_*` for them. The two subsections
|
||||
> below are kept as the dated record of how this was diagnosed, not as current state.
|
||||
|
||||
`servicemonitor.yaml` scrapes **this namespace's** deployment and nothing else. On prod that is
|
||||
two targets, `job="oauth2-proxy"`, both `up=1`. The complete set of oauth2-proxy-specific series
|
||||
in the TSDB is:
|
||||
@@ -157,3 +161,55 @@ present `up` series. A discovered-but-failing target publishes `up` with value 0
|
||||
would have cleared `OAuth2ProxyGateUnobserved` the moment the PodMonitor landed — reporting the gates
|
||||
as observed while nothing was being scraped. The unit test pins that scenario; deleting the `== 1`
|
||||
fails it and nothing else.
|
||||
|
||||
### Update 2026-10-05: the scrape works
|
||||
|
||||
PR #49 (merged `21c88b0`) added the missing third piece. `catalog/render.py`'s
|
||||
`r_monitoring_ingress_policy` now also fires for the gate sidecar's metrics port, keyed on
|
||||
`_needs_oauth2_sidecar`, rendering `<app>-allow-monitoring-scrape` per app: ns `monitoring` → that
|
||||
app's `endpointSelector` → TCP 44180 only. Five objects, one each for jackett, pihole, qbittorrent,
|
||||
radarr, sonarr.
|
||||
|
||||
Measured on prod 2026-10-05 after Flux applied it (`apps-wave-0` 21:37:01Z, `apps-wave-1`
|
||||
21:37:46Z):
|
||||
|
||||
| | before | after |
|
||||
|---|---|---|
|
||||
| target health (6) | `down` | `up` |
|
||||
| `lastError` | `context deadline exceeded` | `''` |
|
||||
| `lastScrapeDuration` | **10.0s** — exactly the `scrapeTimeout` | **0.001–0.006s** |
|
||||
| `up` | `0` × 6 | `1` × 6 |
|
||||
|
||||
count by (__name__) ({__name__=~"oauth2_proxy_.*",
|
||||
namespace=~"jackett|radarr|sonarr|qbittorrent|pihole"})
|
||||
oauth2_proxy_requests_in_flight => 6
|
||||
oauth2_proxy_requests_total => 16
|
||||
oauth2_proxy_response_duration_seconds_bucket => 108
|
||||
oauth2_proxy_response_duration_seconds_sum => 9
|
||||
oauth2_proxy_response_duration_seconds_count => 9
|
||||
|
||||
sum by (namespace) (oauth2_proxy_requests_total)
|
||||
qbittorrent => 35072 radarr => 23560 sonarr => 1284
|
||||
|
||||
`oauth2_proxy_requests_total` is absent for jackett and pihole simply because that counter is not
|
||||
created until the gate has served its first request.
|
||||
|
||||
**So the auth-churn rule is now writable rather than decorative** — sustained `code="302"` as a
|
||||
share of `oauth2_proxy_requests_total`, held across a window longer than the session lifetime. That
|
||||
is the rule the 2026-09-30 incident above could not be caught by. It is still unwritten.
|
||||
|
||||
**Alert outcome:** all 5 `TargetDown` resolved, and 5 of 6 `OAuth2ProxyGateUnobserved` with them.
|
||||
At `repeat_interval: 12h` with `group_by: [alertname, namespace]`, those 11 groups had been
|
||||
re-notifying at roughly **22 emails/day** since 2026-10-02; that drops to 2/day.
|
||||
|
||||
**`crafty` still fires, and that is correct.** It is a hand-maintained Deployment under
|
||||
`platform/crafty`, outside the renderer, whose `oauth2-proxy` container has **no
|
||||
`--metrics-address` at all** — no listener, so no port to open and nothing for the PodMonitor's
|
||||
pod-label selector to match. Closing it needs the flag and a `metrics` containerPort on crafty's
|
||||
own manifest first, then a hand-written allowance beside it. Open follow-up work.
|
||||
|
||||
**The paragraph above headed "To close the gap for an app" is superseded**: a *Service* port and a
|
||||
ServiceMonitor are **not** required. A PodMonitor targets pod ports directly, which is why no
|
||||
Service exposes 44180 and why the renderer needed no Service change. The three things actually
|
||||
needed are `--metrics-address` + named container port, a PodMonitor, and the monitoring-ingress
|
||||
allowance.
|
||||
@@ -27,10 +27,28 @@
|
||||
# created by the apps-wave-0/1 tiers, which this tier does not and must not depend on.
|
||||
#
|
||||
# ============================================================================================
|
||||
# READ THIS BEFORE BELIEVING THE TARGETS ARE UP: THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY
|
||||
# STATUS: THE SCRAPE WORKS. Verified on prod 2026-10-05 (PR #49 merged as 21c88b0).
|
||||
# ============================================================================================
|
||||
# This PodMonitor is NECESSARY AND NOT SUFFICIENT. The endpoint serves, and the path from ns
|
||||
# monitoring is blocked. Both halves measured on prod 2026-10-01, read-only:
|
||||
# All 6 targets in pool podMonitor/oauth2-proxy/oauth2-proxy-sidecars/0 read `up == 1`,
|
||||
# health=up, lastError='', lastScrapeDuration 0.001-0.006s. The TSDB holds oauth2_proxy_* for
|
||||
# these namespaces for the first time: 5 metric names / 148 series, with live counters
|
||||
# (oauth2_proxy_requests_total 35072 qbittorrent, 23560 radarr, 1284 sonarr). All 5 TargetDown
|
||||
# and 5 of 6 OAuth2ProxyGateUnobserved resolved; only namespace="crafty" still fires, and that
|
||||
# is correct — see CRAFTY below.
|
||||
#
|
||||
# The missing piece was an ingress allowance, and it now ships WITH EACH APP, rendered by
|
||||
# catalog/render.py's r_monitoring_ingress_policy as `<app>-allow-monitoring-scrape`:
|
||||
# ns monitoring -> the app's endpointSelector -> TCP 44180 only. See WHY NOT IN THIS DIRECTORY
|
||||
# below for why it is rendered per-app instead of shipped from this tier.
|
||||
#
|
||||
# Keep the history below. It is why the fix is shaped the way it is, and `up == 1` in
|
||||
# auth-visibility-alerts.yaml is load-bearing precisely because of it.
|
||||
#
|
||||
# --------------------------------------------------------------------------------------------
|
||||
# HISTORY: 2026-10-02 -> 2026-10-05, THE SCRAPE WAS DENIED BY NETWORKPOLICY FOR 63.7h
|
||||
# --------------------------------------------------------------------------------------------
|
||||
# This PodMonitor was NECESSARY AND NOT SUFFICIENT: the endpoint served, and the path from ns
|
||||
# monitoring was blocked. Both halves measured on prod 2026-10-01, read-only:
|
||||
#
|
||||
# THE ENDPOINT IS HEALTHY — port-forward straight to the jackett pod's 44180:
|
||||
# GET /metrics -> http=200 bytes=10208
|
||||
@@ -54,19 +72,24 @@
|
||||
# `jackett-allow-gateway-ingress`: `fromEntities: [ingress]` to port 4180 — the Gateway's Envoy
|
||||
# and nothing else. Prometheus matches no clause and its packets are dropped, not refused.
|
||||
#
|
||||
# SO THE TARGETS BELOW WILL READ `up == 0` UNTIL AN INGRESS ALLOWANCE LANDS. That is a known,
|
||||
# measured, documented state — NOT a regression introduced here, and NOT something to silence.
|
||||
# It is deliberately left visible: `OAuth2ProxyGateUnobserved` in auth-visibility-alerts.yaml
|
||||
# THE TARGETS READ `up == 0` FROM 2026-10-02 05:09 UTC UNTIL THE ALLOWANCE LANDED — one
|
||||
# continuous 63.7h episode (a 15d query_range shows 1 episode per namespace, not flapping),
|
||||
# starting 11 minutes after this file was created, i.e. TargetDown's `for: 10m`. It was never
|
||||
# silenced, and that was the right call: `OAuth2ProxyGateUnobserved` in auth-visibility-alerts.yaml
|
||||
# requires `up == 1`, precisely so a discovered-but-failing target does not read as "observed"
|
||||
# (see that file's header — this exact trap is why the `== 1` is there).
|
||||
#
|
||||
# The signature to recognise if it ever returns: lastScrapeDuration pinned at EXACTLY the
|
||||
# scrapeTimeout (10.0s) with `context deadline exceeded`. That is a DROP, not a refusal — a
|
||||
# refused connection fails in milliseconds. Post-fix the same targets scrape in 1-6ms.
|
||||
#
|
||||
# THIS IS THE SAME BUG LONGHORN HAD, and the platform has a documented fix idiom for it:
|
||||
# platform/longhorn/networkpolicy-metrics.yaml, whose header records the identical fingerprint
|
||||
# measured on 2026-09-13 — ServiceMonitor in place, 3 targets DISCOVERED, `up` 0/0/0, TSDB empty,
|
||||
# `lastError: context deadline exceeded` (a drop, not a refusal). The fix there is a narrow
|
||||
# additive `networking.k8s.io/v1` NetworkPolicy: ns monitoring -> the pod label, that one port.
|
||||
#
|
||||
# WHY THAT POLICY IS NOT IN THIS DIRECTORY. It would have to be five objects, one per app
|
||||
# WHY NOT IN THIS DIRECTORY (the reasoning the fix followed). It would have to be five objects, one per app
|
||||
# namespace, and those namespaces are created by apps-wave-0/1. Measured Flux graph, 2026-10-01:
|
||||
# oauth2-proxy dependsOn=['platform-config'] <- this tier
|
||||
# apps-wave-0 dependsOn=['platform-config'] <- SIBLING, no ordering between them
|
||||
@@ -76,8 +99,19 @@
|
||||
# running cluster and break on a rebuild — the "works now, broken on restore" shape this repo
|
||||
# keeps paying for. Longhorn's precedent is safe only because longhorn-system is created by the
|
||||
# same tier that ships its policy. The allowance therefore belongs WITH THE APPS, rendered next to
|
||||
# the rest of each app's network policy — i.e. in the catalog/renderer, which is in flight on
|
||||
# another branch. Tracked in the handoff; it is one narrow rule per app, not a redesign.
|
||||
# the rest of each app's network policy — i.e. in the catalog/renderer. THAT IS WHERE IT LANDED:
|
||||
# `r_monitoring_ingress_policy`, keyed on `_needs_oauth2_sidecar`, one narrow rule per app.
|
||||
#
|
||||
# --------------------------------------------------------------------------------------------
|
||||
# CRAFTY: STILL UNOBSERVED, AND THIS FILE CANNOT HELP IT
|
||||
# --------------------------------------------------------------------------------------------
|
||||
# `OAuth2ProxyGateUnobserved{namespace="crafty"}` still fires after the fix, correctly. crafty is
|
||||
# a hand-maintained Deployment under platform/crafty, OUTSIDE the renderer, and its oauth2-proxy
|
||||
# container has NO `--metrics-address` at all (measured 2026-10-05) — so its metrics listener
|
||||
# never starts and there is no port to open. The selector below therefore does not match it, and
|
||||
# the per-app allowance is never rendered for it. Closing crafty's gap needs the flag plus a
|
||||
# `metrics` containerPort added to its own manifest FIRST; the allowance is the second half, and
|
||||
# it would have to be written by hand there rather than rendered. Open follow-up work.
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PodMonitor
|
||||
metadata:
|
||||
|
||||
Reference in new issue
Block a user