feat(obs): make oauth2-proxy auth failures visible — the gates emit nothing #29

2026-09-30T17:45:24Z

gitea-admin commented

2026-09-30 17:45:24 +00:00

Third of three branches on the Jackett auth failure. The other two fixed the auth defects; this one is about the property that made it expensive: nothing anywhere recorded that it was happening. Updated 2026-10-01 to wire the scrape that PR #28 unblocked.

The failure

oauth2-proxy answers an unauthenticated XHR with a 302 to Keycloak rather than a 401. The browser follows it silently, the live SSO session satisfies it with no prompt, and the XHR resolves 200 carrying a 44 KB HTML dashboard where Jackett's JavaScript wanted JSON. The user is never re-prompted, reloading never helps, every uptime check stays green. jackett-down watches kube_deployment_status_replicas_available, which was 1 throughout — and would have been 1 if the gate had rejected every request ever made.

Round 1 — alert on the gap, because the metric did not exist

Measured 2026-09-30: count by (__name__) ({__name__=~"oauth2_proxy.*"}) returned two series, both from the shared forward-auth Deployment. All six per-app gate sidecars ran without --metrics-address; no Service port, no ServiceMonitor, no PodMonitor. The ratio alert everyone expects here would have been inert on arrival — present, evaluating, permanently silent, indistinguishable from health. This repo has shipped that twice (VeleroNoRecentBackup, the Longhorn rules). It was not shipped a third time.

Shipped instead — OAuth2ProxyGateUnobserved, from two families that are scraped:

count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
  unless
count by (namespace) (up{container="oauth2-proxy"} == 1)

Plus runbooks/oauth2-proxy-auth-failures.md (registered in the generator and README), an inert-alerts row in runbooks/README.md, and a promtool companion test.

Round 2 — the scrape, and a bug this branch had put in its own alert

PR #28 added --metrics-address=0.0.0.0:44180 and a named metrics port to the five fleet sidecars. Three measurements decided what to do with that.

1. A ServiceMonitor cannot do it. No Service exposes 44180 on any of the five:

jackett  [('port-9117',9117),('authproxy',4180)]   radarr   [('port-7878',7878),('authproxy',4180)]
sonarr   [('port-8989',8989),('authproxy',4180)]   qbittorrent [('port-8080',8080),('authproxy',4180)]
pihole   [('port-80',80),('authproxy',4180)]

So platform/oauth2-proxy/podmonitor.yaml is a PodMonitor, selecting the pod label homelab.keithjackson.dev/auth-mode: forward-auth — on exactly those five gate pods, absent from crafty's (which still has no listener). The property, not an app list. No renderer change, no collision with work in flight.

2. The endpoint serves — verified by payload, not by the flag being in args. Port-forwarded to the jackett pod's 44180: http=200, bytes=10208, exporting oauth2_proxy_requests_in_flight, oauth2_proxy_requests_total (code="200" 3, code="302" 2) and oauth2_proxy_response_duration_seconds.

3. But the scrape is dropped, so this PR does not yet make the gates observable. From grafana in ns monitoring (beside Prometheus, same policy position) to the jackett pod: :44180 → wget: download timed out, and :4180 → also timed out; the shared proxy's :44180 answers. ns jackett's default-deny is podSelector: {} / policyTypes: [Ingress, Egress] with an egress rule and no ingress key — deny-all ingress — and the only allowance is fromEntities: [ingress] to 4180, the Gateway's Envoy. Prometheus matches no clause. The shared proxy has only ever been scrapable because ns oauth2-proxy has no NetworkPolicy at all.

This is the same bug, with the same fingerprint, that platform/longhorn/networkpolicy-metrics.yaml documents from 2026-09-13 (ServiceMonitor present, targets discovered, up 0/0/0, TSDB empty, context deadline exceeded — a drop, not a refusal). The fix is one narrow additive rule per app. It cannot ship from this tier: measured Flux graph, oauth2-proxy dependsOn=[platform-config] and apps-wave-0 dependsOn=[platform-config] — siblings, no ordering. A NetworkPolicy naming ns jackett from here would apply on today's cluster and fail on a cold rebuild. It belongs with each app's rendered policy. This is the one open item and it needs an owner.

The bug: `up` → `up == 1`

A target that is discovered but failing still publishes up, with value 0. The series exists, so count by (namespace) returns 1, unless removes the namespace, and OAuth2ProxyGateUnobserved goes quiet while nothing is being scraped at all — this branch's own disease reproduced inside the alert written to cure it. Not hypothetical: the PodMonitor lands five targets whose scrape is dropped, so the bare form would have silently cleared on this very commit.

Proven by mutation, not asserted. Reverting the == 1:

expr: ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett"}, time: 1h5m
    exp: ... 1E+00
    got: nil
expr: ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="radarr"}, time: 2h5m
    exp: ... 1E+00
    got: nil

Two new test scenarios pin it: a discovered-but-failing target must still fire, and a target that works and then breaks must re-fire rather than latch cleared.

Still not shipped: the ratio alert

oauth2_proxy_requests_total now exists on the endpoint but not in the TSDB, because the scrape is blocked. Writing it today would make it inert for exactly the reason it was declined on 2026-09-30. It becomes writable the day a scrape succeeds — not the day a listener exists.

⚠️ This PR touches `conformance/**` — non-author review required on the sensitive-path rule

One entry in KNOWN_EXTERNAL_MONITORS (servicemonitor-ports-resolve). This is the mechanism the failing check explicitly instructs ("DECLARE it in KNOWN_EXTERNAL_MONITORS with who creates it") and which flux-podmonitor.yaml already uses for the same pod-port reason. The entry is honest about being different from its neighbours: the target is not chart-owned, it is unauditable because no Service exposes 44180 by design, what assertion (2) would have checked was measured on prod instead, and the targets are currently down. The register is shrink-only, so if a Service ever gains that label the entry goes stale and fails. Reviewers should read that entry first — it is the only thing here that changes what green means.

Also in round 2

origin/main merged in (three PRs landed mid-flight). The runbooks/kustomization.yaml conflict was resolved by keeping both runbook entries — mine and oauth2-proxy-session are different pages, and a comment now says which is which.
Docs corrected, not appended to: the "sidecars export no metrics at all" claim is now half true, and runbooks/README.md, platform/oauth2-proxy/README.md and the alert header each show the two-step history, because each step looked like the end of it.
Runbook gains step 1b — telling "no target" from "target failing" (up absent vs up == 0, then the target's lastError; context deadline exceeded means policy, not a dead endpoint).
Runbook gains the radarr /signalr/messages/negotiate worked case: 206 rejections, and sampling 209 of them every one was that single request from a browser UA. One left-open tab, orphaned cookie, silently 302'd forever, nobody ever re-prompted. The whole failure class in one line. With the grouping command, because a rejection count without a path breakdown is misleading.

Gate

Toolbox ./ci/gate.sh (pinned container, worktree ENGINE shim, spec repo mounted): rc 3 — UNDETERMINED, not green.

stage	outcome
spec-immutability	green
decisions-log	green
lint	green
l0	green
render-golden	green
l1	green
approval-policy	green (N/A locally — not a PR event)
gitea-settings	SKIPPED — no `SETTINGS_API_TOKEN`
conformance (L2/L3)	SKIPPED — `CLUSTER_AVAILABLE` unset

Touched requirements R-IDP-02 R-IDP-06 R-OBS-01 R-OBS-04 R-TEST-01..04 are not certified. alert-metrics-have-scrape-targets, alert-rules-unit-tested, alert-runbooks-resolvable and servicemonitor-ports-resolve all PASS; the unit suite is green under pinned promtool v3.14.0.

Unverified

The PodMonitor has never been applied. That it discovers the five pods is reasoned from measured pod labels and port names, not observed. Expect up == 0 on all five when it lands — documented state, not a regression.
No session-lifecycle claim is possible from the current window: AuthSuccess is 0 on both jackett and radarr since the PR #28 restart, so nobody has signed in and --cookie-refresh=2m30s has never executed a refresh.
The radarr /signalr counts and the jackett 0-in-13,612 are the coordinator's measurements, recorded as reported — I did not reproduce them.
All cluster work was read-only. Nothing applied, no flux reconcile.

Handoff: decisions/2026-09-30-oauth2-proxy-auth-failure-visible.md (round 2 appended under its own heading).

🤖 Generated with Claude Code

⚠️ CI shows `gate / gate` FAILING, and that is the gate working correctly — read this before "fixing" it

gate / gate (pull_request) = failure after 1m42s. The cause is measured, singular, and expected:

sensitive paths touched:
  conformance/tests/l0-static/servicemonitor-ports-resolve/check.sh   (matched conformance/**)
approving reviews: (no reviews at all)
author=gitea-admin

ci/approval-policy.sh is fail-closed and exits 1 when a PR touches a sensitive_paths glob with no APPROVED review from someone other than its author — "a change to the gate cannot be gated by the gate." One file triggers it: the single KNOWN_EXTERNAL_MONITORS entry. Nothing in L0/L1 failed; the local toolbox run was rc 3 with only the gitea-settings environment skip.

Note that the local gate cannot catch this, and reported approval-policy: green (N/A) — the stage short-circuits to N/A off a PR event (GITHUB_EVENT_NAME unset). So for any PR touching a sensitive path, a clean local gate is a false reassurance; only the PR run tells the truth. Worth knowing generally, not just here.

To clear it: a non-author approving review, then re-run the gate — approval does not re-trigger the status check by itself, so the required context stays red until the job runs again. required_approvals=1 already applied to this PR regardless, so the review was needed either way; the conformance entry only changes the tick from green to red while it is pending.

If you would rather this PR merge on green alone, the only thing to remove is the register entry — which means dropping podmonitor.yaml with it, since the check fails without the declaration. That trade is yours; I would keep both, because the entry is exactly the "declare the limit" mechanism the check asks for and it is shrink-only, so it cannot rot quietly.

Third of three branches on the Jackett auth failure. The other two fixed the auth defects; this one is about the property that made it expensive: **nothing anywhere recorded that it was happening.** Updated 2026-10-01 to wire the scrape that PR #28 unblocked. ## The failure oauth2-proxy answers an unauthenticated **XHR** with a `302` to Keycloak rather than a `401`. The browser follows it silently, the live SSO session satisfies it with no prompt, and the XHR resolves **200 carrying a 44 KB HTML dashboard** where Jackett's JavaScript wanted JSON. The user is never re-prompted, reloading never helps, every uptime check stays green. `jackett-down` watches `kube_deployment_status_replicas_available`, which was `1` throughout — and would have been `1` if the gate had rejected every request ever made. ## Round 1 — alert on the gap, because the metric did not exist Measured 2026-09-30: `count by (__name__) ({__name__=~"oauth2_proxy.*"})` returned **two series**, both from the shared forward-auth Deployment. All six per-app gate sidecars ran without `--metrics-address`; no Service port, no ServiceMonitor, no PodMonitor. The ratio alert everyone expects here would have been **inert on arrival** — present, evaluating, permanently silent, indistinguishable from health. This repo has shipped that twice (`VeleroNoRecentBackup`, the Longhorn rules). It was not shipped a third time. Shipped instead — `OAuth2ProxyGateUnobserved`, from two families that *are* scraped: ```promql count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by (namespace) (up{container="oauth2-proxy"} == 1) ``` Plus `runbooks/oauth2-proxy-auth-failures.md` (registered in the generator **and** README), an inert-alerts row in `runbooks/README.md`, and a promtool companion test. ## Round 2 — the scrape, and a bug this branch had put in its own alert PR #28 added `--metrics-address=0.0.0.0:44180` and a named `metrics` port to the five fleet sidecars. Three measurements decided what to do with that. **1. A ServiceMonitor cannot do it.** No Service exposes 44180 on any of the five: ``` jackett [('port-9117',9117),('authproxy',4180)] radarr [('port-7878',7878),('authproxy',4180)] sonarr [('port-8989',8989),('authproxy',4180)] qbittorrent [('port-8080',8080),('authproxy',4180)] pihole [('port-80',80),('authproxy',4180)] ``` So `platform/oauth2-proxy/podmonitor.yaml` is a **PodMonitor**, selecting the pod label `homelab.keithjackson.dev/auth-mode: forward-auth` — on exactly those five gate pods, absent from crafty's (which still has no listener). The property, not an app list. No renderer change, no collision with work in flight. **2. The endpoint serves — verified by payload, not by the flag being in `args`.** Port-forwarded to the jackett pod's 44180: `http=200`, `bytes=10208`, exporting `oauth2_proxy_requests_in_flight`, `oauth2_proxy_requests_total` (`code="200" 3`, `code="302" 2`) and `oauth2_proxy_response_duration_seconds`. **3. But the scrape is dropped, so this PR does not yet make the gates observable.** From grafana in ns `monitoring` (beside Prometheus, same policy position) to the jackett pod: `:44180` → `wget: download timed out`, and `:4180` → **also** timed out; the shared proxy's `:44180` answers. ns `jackett`'s `default-deny` is `podSelector: {}` / `policyTypes: [Ingress, Egress]` with an `egress` rule and **no `ingress` key** — deny-all ingress — and the only allowance is `fromEntities: [ingress]` to 4180, the Gateway's Envoy. Prometheus matches no clause. The shared proxy has only ever been scrapable because ns `oauth2-proxy` has **no NetworkPolicy at all**. This is the same bug, with the same fingerprint, that `platform/longhorn/networkpolicy-metrics.yaml` documents from 2026-09-13 (ServiceMonitor present, targets discovered, `up` 0/0/0, TSDB empty, `context deadline exceeded` — a drop, not a refusal). The fix is one narrow additive rule per app. **It cannot ship from this tier:** measured Flux graph, `oauth2-proxy dependsOn=[platform-config]` and `apps-wave-0 dependsOn=[platform-config]` — *siblings*, no ordering. A NetworkPolicy naming ns `jackett` from here would apply on today's cluster and fail on a cold rebuild. It belongs with each app's rendered policy. **This is the one open item and it needs an owner.** ### The bug: `up` → `up == 1` A target that is **discovered but failing still publishes `up`, with value 0.** The series exists, so `count by (namespace)` returns 1, `unless` removes the namespace, and `OAuth2ProxyGateUnobserved` **goes quiet while nothing is being scraped at all** — this branch's own disease reproduced inside the alert written to cure it. Not hypothetical: the PodMonitor lands five targets whose scrape is dropped, so the bare form would have silently cleared on this very commit. Proven by mutation, not asserted. Reverting the `== 1`: ``` expr: ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett"}, time: 1h5m exp: ... 1E+00 got: nil expr: ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="radarr"}, time: 2h5m exp: ... 1E+00 got: nil ``` Two new test scenarios pin it: a discovered-but-failing target must still fire, and a target that works and then breaks must **re-fire** rather than latch cleared. ### Still not shipped: the ratio alert `oauth2_proxy_requests_total` now exists on the endpoint but **not in the TSDB**, because the scrape is blocked. Writing it today would make it inert for exactly the reason it was declined on 2026-09-30. It becomes writable the day a scrape succeeds — not the day a listener exists. ## ⚠️ This PR touches `conformance/**` — non-author review required on the sensitive-path rule One entry in `KNOWN_EXTERNAL_MONITORS` (`servicemonitor-ports-resolve`). This is the mechanism the failing check explicitly instructs ("DECLARE it in KNOWN_EXTERNAL_MONITORS with who creates it") and which `flux-podmonitor.yaml` already uses for the same pod-port reason. The entry is honest about being different from its neighbours: the target is **not** chart-owned, it is unauditable because no Service exposes 44180 *by design*, what assertion (2) would have checked was measured on prod instead, and **the targets are currently down**. The register is shrink-only, so if a Service ever gains that label the entry goes stale and fails. Reviewers should read that entry first — it is the only thing here that changes what green means. ## Also in round 2 - `origin/main` merged in (three PRs landed mid-flight). The `runbooks/kustomization.yaml` conflict was resolved by **keeping both** runbook entries — mine and `oauth2-proxy-session` are different pages, and a comment now says which is which. - Docs **corrected, not appended to**: the "sidecars export no metrics at all" claim is now half true, and `runbooks/README.md`, `platform/oauth2-proxy/README.md` and the alert header each show the two-step history, because each step looked like the end of it. - Runbook gains step **1b** — telling "no target" from "target failing" (`up` absent vs `up == 0`, then the target's `lastError`; `context deadline exceeded` means policy, not a dead endpoint). - Runbook gains the **radarr `/signalr/messages/negotiate` worked case**: 206 rejections, and sampling 209 of them every one was that single request from a browser UA. One left-open tab, orphaned cookie, silently 302'd forever, nobody ever re-prompted. The whole failure class in one line. With the grouping command, because a rejection count without a path breakdown is misleading. ## Gate Toolbox `./ci/gate.sh` (pinned container, worktree `ENGINE` shim, spec repo mounted): **rc 3 — UNDETERMINED, not green.** | stage | outcome | |---|---| | spec-immutability | green | | decisions-log | green | | lint | green | | l0 | green | | render-golden | green | | l1 | green | | approval-policy | green (N/A locally — not a PR event) | | **gitea-settings** | **SKIPPED** — no `SETTINGS_API_TOKEN` | | **conformance (L2/L3)** | **SKIPPED** — `CLUSTER_AVAILABLE` unset | Touched requirements `R-IDP-02 R-IDP-06 R-OBS-01 R-OBS-04 R-TEST-01..04` are **not certified**. `alert-metrics-have-scrape-targets`, `alert-rules-unit-tested`, `alert-runbooks-resolvable` and `servicemonitor-ports-resolve` all PASS; the unit suite is green under pinned promtool v3.14.0. ## Unverified - **The PodMonitor has never been applied.** That it discovers the five pods is reasoned from measured pod labels and port names, not observed. **Expect `up == 0` on all five when it lands** — documented state, not a regression. - **No session-lifecycle claim is possible from the current window:** `AuthSuccess` is **0** on both jackett and radarr since the PR #28 restart, so nobody has signed in and `--cookie-refresh=2m30s` has never executed a refresh. - The radarr `/signalr` counts and the jackett 0-in-13,612 are the coordinator's measurements, recorded as reported — I did not reproduce them. - All cluster work was read-only. Nothing applied, no `flux reconcile`. Handoff: `decisions/2026-09-30-oauth2-proxy-auth-failure-visible.md` (round 2 appended under its own heading). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- ## ⚠️ CI shows `gate / gate` FAILING, and that is the gate working correctly — read this before "fixing" it `gate / gate (pull_request)` = **failure after 1m42s**. The cause is measured, singular, and expected: ``` sensitive paths touched: conformance/tests/l0-static/servicemonitor-ports-resolve/check.sh (matched conformance/**) approving reviews: (no reviews at all) author=gitea-admin ``` `ci/approval-policy.sh` is **fail-closed and exits 1** when a PR touches a `sensitive_paths` glob with no `APPROVED` review from someone other than its author — *"a change to the gate cannot be gated by the gate."* One file triggers it: the single `KNOWN_EXTERNAL_MONITORS` entry. Nothing in L0/L1 failed; the local toolbox run was rc 3 with only the `gitea-settings` environment skip. **Note that the local gate cannot catch this**, and reported `approval-policy: green (N/A)` — the stage short-circuits to N/A off a PR event (`GITHUB_EVENT_NAME` unset). So for any PR touching a sensitive path, a clean local gate is a false reassurance; only the PR run tells the truth. Worth knowing generally, not just here. **To clear it:** a non-author approving review, then **re-run the gate** — approval does not re-trigger the status check by itself, so the required context stays red until the job runs again. `required_approvals=1` already applied to this PR regardless, so the review was needed either way; the conformance entry only changes the tick from green to red while it is pending. **If you would rather this PR merge on green alone,** the only thing to remove is the register entry — which means dropping `podmonitor.yaml` with it, since the check fails without the declaration. That trade is yours; I would keep both, because the entry is exactly the "declare the limit" mechanism the check asks for and it is shrink-only, so it cannot rot quietly.

gitea-admin added 1 commit 2026-09-30 17:45:25 +00:00

feat(obs): make oauth2-proxy auth failures visible — alert on the gap, not on absent metrics

gate / gate (pull_request) Successful in 1m42s

Details

gate / conformance (pull_request) Has been skipped

Details

fc7ec01628

Jackett failed on 2026-09-30 in the most expensive way available: HTTP 200 and a
plausible page. oauth2-proxy answers an unauthenticated XHR with a 302 to Keycloak
rather than a 401; the browser follows it silently against a live SSO session and
the call resolves 200 carrying a 44 KB HTML dashboard where the app's JavaScript
wanted JSON. The user is never re-prompted, reloading never helps, and every uptime
check stays green. `jackett-down` watches replicas_available, which was 1 throughout.

The reason nothing alerted turns out not to be a missing rule. Measured on prod:

    count by (__name__) ({__name__=~"oauth2_proxy.*"})
      oauth2_proxy_requests_in_flight => 2

Two series, both from the SHARED forward-auth Deployment. The six per-app gate
sidecars (crafty, jackett, pihole, qbittorrent, radarr, sonarr) publish nothing at
all: no --metrics-address, so the metrics listener never starts; no `metrics`
container port; no ServiceMonitor or PodMonitor selects them. The alert the incident
calls for — sustained auth redirects to an authenticated user — is therefore NOT
WRITABLE, and writing it anyway would have shipped a rule that is present,
evaluating, permanently silent and indistinguishable from health.

So alert on the condition that made the incident invisible, from families that ARE
scraped:

  OAuth2ProxyGateUnobserved
    count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
      unless
    count by (namespace) (up{container="oauth2-proxy"})

kube_pod_container_info is kube-state-metrics (a declared scrape owner) and sees the
container whether or not it exports anything; `up` is UNIVERSAL and carries a
`container` label for any target, so the rule needs no advance knowledge of what job
name a future per-app scrape picks. Nothing in it touches oauth2_proxy_*, so it
cannot be inert for the reason it is reporting. Run verbatim against live prod it
returns six namespaces and correctly excludes oauth2-proxy. It clears per namespace
as each gate gains a real scrape.

Also:
- promtool companion test pinning all four states: fires unscraped, quiet when
  scraped, honours for:1h, and STOPS once a scrape appears under a different job name.
- runbooks/oauth2-proxy-auth-failures.md — tell the gate's refusal from the group
  check from the app's own login from an XHR that silently got HTML, with the
  2026-09-30 jackett trace as the worked example and the Loki queries to use while
  there are no metrics. Registered in the configMapGenerator and README.
- runbooks/README.md gains an inert-alerts row for the unscraped sidecars.

Rejected: an alert on the shared proxy's oauth2_proxy_requests_total ratio. It needs
a conformance/ SCRAPE_OWNERS entry (human approval) and, decisively, would not have
fired on this incident — the shared instance is not in any sidecar-gated app's data
path and its current pods have proxied nothing. It would have looked like coverage
and been none.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

gitea-admin added 2 commits 2026-10-02 02:57:56 +00:00

Merge remote-tracking branch 'origin/main' into feat/oauth2-proxy-auth-failure-visible ba79a1b59b

# Conflicts:
#	runbooks/kustomization.yaml

feat(obs): wire the gate-sidecar scrape, and fix the alert that would have gone silent doing it

gate / gate (pull_request) Failing after 1m42s

Details

gate / conformance (pull_request) Has been skipped

Details

bce101d03f

PR #28 gave the five fleet oauth2-proxy sidecars --metrics-address=0.0.0.0:44180 and a
named `metrics` port. This wires a scraper to them and, in the process, found a real bug
in the alert this branch added.

PodMonitor, not ServiceMonitor. Measured: no Service exposes 44180 on any of the five
(jackett/radarr/sonarr/qbittorrent/pihole all publish only their app port + authproxy
4180), so a ServiceMonitor would select nothing. A PodMonitor targets pod ports directly
and needs no change to catalog/render.py or the apps/wave-* renders and goldens.
Selector is the pod label homelab.keithjackson.dev/auth-mode=forward-auth — present on
exactly those five gate pods, absent from crafty's, which still has no metrics listener.
The property, not an app list.

THE BUG: `up{container="oauth2-proxy"}` -> `up{container="oauth2-proxy"} == 1`.

A target that is discovered but FAILING still publishes `up`, with value 0. The series
exists, so count by (namespace) returned 1, `unless` removed the namespace, and
OAuth2ProxyGateUnobserved went QUIET while nothing was being scraped at all — this
branch's own disease reproduced inside the alert written to cure it. Not hypothetical:
the PodMonitor lands five targets whose scrape is dropped, so the bare form would have
silently cleared on this very commit. Proven by mutation: reverting the `== 1` fails the
two new unit-test scenarios (got: nil) and leaves every other assertion green.

The scrape does NOT work yet, and that is documented rather than papered over. From a
pod in ns monitoring to the jackett pod, :44180 AND :4180 both time out; the shared
proxy's :44180 answers. ns jackett's default-deny is podSelector:{} with
policyTypes:[Ingress,Egress] and NO ingress key (deny-all), and the one allowance is
fromEntities:[ingress] to 4180 — the Gateway's Envoy only. The endpoint itself is
healthy: port-forwarded, /metrics returns http=200, 10208 bytes, all three oauth2_proxy_*
families including requests_total{code="302"}. Same bug and fingerprint as
platform/longhorn/networkpolicy-metrics.yaml records for Longhorn; the fix is one narrow
rule per app, and it cannot ship from this tier because oauth2-proxy and apps-wave-0 are
Flux SIBLINGS (both dependsOn platform-config), so naming ns jackett here would apply
today and fail on a cold rebuild. Needs an owner in the renderer.

Touches conformance/** : one KNOWN_EXTERNAL_MONITORS entry, which is the mechanism the
failing check explicitly instructs ("DECLARE it ... with who creates it") and which
flux-podmonitor.yaml already uses. It records that the port name was measured on prod
instead of resolved from a Service, and that the targets are currently down. That makes
this PR require a non-author review on the sensitive-path rule.

Also: ratio alert still NOT shipped. oauth2_proxy_requests_total now exists on the
endpoint but not in the TSDB, so writing it today would make it inert for exactly the
reason it was refused on 2026-09-30. Runbook gains policy-vs-no-target triage and the
radarr /signalr/messages/negotiate worked case — one orphaned browser tab, 206 identical
302s, nobody ever re-prompted.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

gitea-admin added 1 commit 2026-10-02 03:12:20 +00:00

docs(obs): record why CI is red — approval-policy on the one conformance entry

gate / gate (pull_request) Successful in 1m41s

Details

gate / conformance (pull_request) Has been skipped

Details

1e71b578d5

The gate job reported failure while the local run was rc 3 with nothing failed. Measured
cause is a single sensitive path (the KNOWN_EXTERNAL_MONITORS entry) with no non-author
approving review, which ci/approval-policy.sh blocks fail-closed by design.

Also records the general trap: approval-policy short-circuits to N/A off a pull_request
event, so a clean LOCAL gate is a false reassurance for any branch touching a sensitive
path. Only the PR run tells the truth.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

gitea-admin referenced this issue from a commit

2026-10-02 03:27:01 +00:00

chore(decisions): record the CI reading, and why the metrics policy is NOT here

gitea-admin referenced this pull request

2026-10-02 03:27:55 +00:00

feat(auth): declare API paths per service, so --api-route can ship at all #34

gitea-admin referenced this issue from a commit

2026-10-02 03:31:39 +00:00

chore(decisions): record the CI reading, and why the metrics policy is NOT here

gitea-admin referenced this issue from a commit

2026-10-02 03:38:44 +00:00

chore(decisions): record the CI reading, and why the metrics policy is NOT here