jakman85andClaude Opus 5
81f627167d
PR #49 (merged 21c88b0) fixed the blocked scrape and it is verified working on prod
2026-10-05: all 6 targets up == 1, health=up, lastError='', lastScrapeDuration 0.001-0.006s
against 10.0s and "context deadline exceeded" before. oauth2_proxy_* is in the TSDB for
these namespaces for the first time (5 metric names, 148 series).
Four places in the tree still asserted the pre-fix state as current, in the present tense,
in the files a reader consults first:
platform/oauth2-proxy/podmonitor.yaml "READ THIS BEFORE BELIEVING THE TARGETS ARE UP:
THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY" and "SO THE TARGETS BELOW WILL READ
up == 0 UNTIL AN INGRESS ALLOWANCE LANDS". The first actively instructs the reader to
disbelieve a correct up == 1. Now a STATUS block with the post-fix numbers, with the
diagnosis kept below it under an explicit HISTORY heading because it is why the fix is
shaped the way it is.
catalog/render.py:159 "THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED".
catalog/render.py:1365 the same claim on the sidecar arg.
platform/oauth2-proxy/README.md its subsections are dated and so are honest records, but
a reader hits the stale body long before any correction. A SUPERSEDED pointer now sits
under the heading and an Update 2026-10-05 section carries the before/after.
The "not scraped" claims are corrected; the "a flag alone is not enough" warning is KEPT
deliberately, because that part is still true and "flag without ingress allowance" is
exactly what caused the 63.7h outage.
Two substantive errors fixed while there, not just tense. The README's "three things are
needed" listed a Service port and a ServiceMonitor — wrong, a PodMonitor targets pod ports
directly, which is why no Service exposes 44180 and the renderer needed no Service change;
following it would have sent someone to edit Service rendering for nothing. And
podmonitor.yaml deferred the fix to "the catalog/renderer, in flight on another branch" —
no such branch existed; it now names r_monitoring_ingress_policy, with a new CRAFTY section
explaining the one namespace this file cannot help.
Comments only in render.py, so the rendered output is byte-identical: no apps/ file and no
golden changes, and therefore no sensitive path and no human approval needed — the opposite
of the PR this documents.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
156 lines
10 KiB
Markdown
156 lines
10 KiB
Markdown
---
|
|
date: 2026-10-04
|
|
branch: fix/oauth2-sidecar-metrics-scrape-netpol
|
|
title: The forward-auth gate sidecars' /metrics was NetworkPolicy-blocked — the renderer now opens 44180 to ns monitoring
|
|
---
|
|
|
|
> **Added 2026-10-05, after merge:** the top item under *What I did NOT verify* below — "that the
|
|
> scrape now succeeds" — **has since been verified on prod**. All 6 targets read `up == 1`. The
|
|
> original text is left exactly as written; the measurements are in
|
|
> [[2026-10-05-oauth2-sidecar-scrape-verified]].
|
|
|
|
## What changed
|
|
|
|
`catalog/render.py`'s `r_monitoring_ingress_policy` now emits its CiliumNetworkPolicy for a
|
|
**second** reason: the forward-auth gate sidecar's own `/metrics` port (44180), keyed on
|
|
`_needs_oauth2_sidecar(s)`. Previously it fired only from `observability.metricsEndpoint`.
|
|
|
|
That matters more than it sounds, because **no ServiceDefinition in this catalog declares a
|
|
`metricsEndpoint`** (measured: zero grep hits across `catalog/*.yaml`). So the function was wired
|
|
in at the call site and had **never emitted a single object on any branch** — correct, and
|
|
entirely latent. Its presence in the tree read as "scrape ingress is handled". It was not.
|
|
|
|
Re-rendered `apps/` from that change: five new CNPs, one per forward-auth app
|
|
(`jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`), each opening **only** 44180 to **only**
|
|
`fromEndpoints: [k8s:io.kubernetes.pod.namespace: monitoring]`.
|
|
|
|
The bug being fixed: `platform/oauth2-proxy/podmonitor.yaml` landed 2026-10-02T04:58:44Z and
|
|
discovered 6 targets that have **never once scraped successfully**. Each app namespace ships a
|
|
`default-deny` NetworkPolicy with `policyTypes: [Ingress, Egress]` and **no `ingress` key at all**
|
|
— deny-all-ingress in NetworkPolicy semantics — and the only ingress allowance is
|
|
`<app>-allow-gateway-ingress`: `fromEntities: [ingress]` to port 4180. Prometheus matched no
|
|
clause, so its packets were dropped.
|
|
|
|
Two deliberate choices, both recorded in the function's docstring:
|
|
|
|
- **Clause 2 keys on `_needs_oauth2_sidecar`, not on `observability.metrics`.** It has to match
|
|
the condition the *scrape* keys on: `podmonitor.yaml` selects pods by the
|
|
`homelab.keithjackson.dev/auth-mode: forward-auth` pod label and nothing else, so a target
|
|
exists iff the sidecar exists. Keying on `observability.metrics` would let `metrics: false`
|
|
produce a discovered-but-blocked target — the same "two halves authored apart" failure this
|
|
function exists to prevent, moved one field over.
|
|
- **The policy is rendered here, not added to `platform/oauth2-proxy/`.** podmonitor.yaml's own
|
|
header records why that tier cannot host it: it would name app namespaces created by the
|
|
*sibling* `apps-wave-0/1` tiers, so it would apply on a running cluster and fail on a cold
|
|
bootstrap. Rendered per-app it ships in the same tier that creates the namespace.
|
|
|
|
## What I verified
|
|
|
|
Measured on prod 2026-10-04, read-only, via the pinned toolbox:
|
|
|
|
- **The cause, not a guess.** All 6 sidecar targets in pool
|
|
`podMonitor/oauth2-proxy/oauth2-proxy-sidecars/0`: `health=down`,
|
|
`lastError: context deadline exceeded`, `lastScrapeDuration` exactly `10.0s` — the configured
|
|
`scrapeTimeout`. Hitting the timeout precisely means packets **dropped, not refused**.
|
|
- **The listener is fine.** jackett's `oauth2-proxy` container carries
|
|
`--metrics-address=0.0.0.0:44180` and a named port `metrics: 44180`.
|
|
- **The deny is universal.** All six namespaces (`jackett radarr sonarr qbittorrent pihole
|
|
crafty`): `default-deny` with `ingress` key absent, `nIngress=0`; the only ingress CNP allowance
|
|
is 4180 `fromEntities: [ingress]`. **Zero** namespaces had any `-allow-monitoring-scrape` object.
|
|
- **Frequency.** Single continuous episode, no flapping: `ALERTS_FOR_STATE` = 229343s (63.7h),
|
|
firing since 2026-10-02 05:09 UTC — 11 minutes after the PodMonitor was created, i.e. TargetDown's
|
|
`for: 10m`. A 15-day `query_range` shows **1 episode** per namespace, not many. Alertmanager:
|
|
`group_by: [alertname, namespace]`, `repeat_interval: 12h` on `email-warning` → 5 TargetDown
|
|
groups = 10 emails/day, plus 6 `OAuth2ProxyGateUnobserved` groups = 12/day, ~22/day total.
|
|
0 silences.
|
|
- **The re-render is deterministic and the diff is exactly the intended one.** `apps/` is
|
|
byte-identical to `render.py --all catalog -o apps --wave-layout --no-image-automation`
|
|
(12/12 SAME before my edit; the 11-file "drift" against a default render is solely the
|
|
ImageRepository/ImagePolicy pair, which `apps/` is generated without). After the edit:
|
|
5 files changed, +21 lines each, nothing else touched.
|
|
- `render.py --selftest`: **PASS**.
|
|
- **Three L1 goldens needed regenerating, and the diff is +63 lines / -0.** The first gate run
|
|
caught `container-workload-render` (R-PLAT-02) and `egress-class-render` (R-NET-05/06/07) as
|
|
FAIL with exactly the new CNP as the drift. Regenerated with the same invocations the checks
|
|
use (`render.py <fixture>`, no flags). Evidence the regen is faithful rather than a rubber
|
|
stamp: `default-deny-baseline-render`'s golden came back **byte-identical** (its fixture is not
|
|
forward-auth), and the other three gained only the one CNP, port 44180, ns monitoring.
|
|
- Toolbox gate `./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`: **rc 3 — UNDETERMINED,
|
|
NOT rc 0, and I am not reporting it as green.** Every stage that ran is green
|
|
(spec-immutability, approval-policy, decisions-log, lint, l0, render-golden, l1); `l1` went
|
|
FAIL -> pass once the goldens were regenerated. The gate's own words: "Stages that did not run:
|
|
gitea-settings" — skipped for want of `GITEA_SETTINGS_TOKEN`, which is an environment gap on
|
|
every local run and unrelated to this change. L2/L3 conformance is a `[warn]`, not a not-run
|
|
stage: UNVERIFIED for all 8 touched requirements (R-GITOPS-02 R-IDP-02 R-IDP-06 R-NET-01
|
|
R-NET-05 R-NET-06 R-NET-07 R-PLAT-02), no cluster.
|
|
- `--emit-waves` was **not** run (it re-onboards `vm-demo` + `keycloak`); only the per-service
|
|
render path was used, so no wave kustomization changed.
|
|
|
|
## This PR needs a human approval (sensitive path) — MEASURED, not predicted
|
|
|
|
CI run #230 on PR #49, head `f9446be3`: **`MERGE GATE: BLOCKED`, and `approval-policy` is the ONLY
|
|
failing stage.** Read from the job log, not from the status tick:
|
|
|
|
[skip] stage spec-immutability: SKIPPED (not run) <- sibling spec repo absent in CI; expected
|
|
[ ok ] stage decisions-log: pass
|
|
[ ok ] stage lint: pass
|
|
[ ok ] stage l0: pass
|
|
[ ok ] stage render-golden: pass
|
|
[ ok ] stage l1: pass
|
|
[fail] stage approval-policy: FAIL
|
|
[ ok ] stage gitea-settings: pass <- could NOT run locally; green in CI
|
|
[skip] stage conformance: SKIPPED (not run)
|
|
|
|
sensitive paths touched:
|
|
conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml
|
|
conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml
|
|
conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml
|
|
|
|
Exactly the three regenerated goldens, nothing else. `l1` and `render-golden` both pass in CI,
|
|
which is the independent confirmation that the regenerated goldens are right rather than merely
|
|
self-consistent with my local run.
|
|
|
|
One local-reproduction note worth recording, because it wasted a step: running
|
|
`PR_NUMBER=49 SETTINGS_API_TOKEN=... ci/approval-policy.sh` with the credential from
|
|
`git credential fill` reports `could not determine PR author — BLOCKING`. That is an **artifact**,
|
|
not a finding — `~/.git-credentials` holds a *password*, and `Authorization: token <password>`
|
|
returns 401 `user does not exist [uid: 0]`. The stage needs a real API token. I nearly reported
|
|
that 401 as the CI failure; the job log is what settled it.
|
|
|
|
Regenerating the L1 goldens means this branch touches `conformance/**`, which is in
|
|
`sensitive_paths` in `infra/gitea/approval-policy.json` ("the tests and the harness; changing them
|
|
changes what green MEANS"). So `ci/approval-policy.sh` will require an approving review from a
|
|
non-author, and the local gate **cannot** see that: it reports `approval-policy: green` because
|
|
offline it evaluates N/A.
|
|
|
|
The golden update is not optional — a renderer change that adds an object necessarily drifts the
|
|
goldens that pin the renderer's output — but it does mean the gate correctly refuses to
|
|
self-approve this one. **An agent must not clear this with `/approve-override`**, even though the
|
|
`gitea-admin` account holds `owner`: that is precisely the self-approval the stage exists to
|
|
prevent. The operator posts the override as a PR *review* of type "Comment", or approves normally.
|
|
|
|
## What I did NOT verify
|
|
|
|
- **That the scrape now succeeds.** This is a git-only change and nothing is applied until Flux
|
|
reconciles `main`. The 5 CNPs have never existed on the cluster, so `up == 1` for those targets
|
|
is **UNVERIFIED** — it is a prediction from the measured drop signature, not an observation.
|
|
After merge, confirm `up{container="oauth2-proxy"}` goes to 1 for all 6 targets and that the 11
|
|
alerts resolve. `send_resolved: true` is set, so expect ~11 resolved emails.
|
|
- **That the CNP `endpointSelector` matches pihole's pods.** It selects
|
|
`app.kubernetes.io/name: pihole`, identical to the shape of `pihole-allow-gateway-ingress`
|
|
which demonstrably works for 4180 — so this is inference from a working sibling, not a
|
|
measurement of the new object.
|
|
- **`crafty` is still unobserved and will keep firing `OAuth2ProxyGateUnobserved`.** It is
|
|
hand-managed under `platform/crafty`, outside the renderer, and has **no `--metrics-address`
|
|
at all** — so there is no port to open and this change cannot help it. Separate work.
|
|
- **Whether 44180 is reachable in the Cilium data path once allowed.** Only the policy was
|
|
reasoned about; no L2 probe was run (CI has `vars.CLUSTER_AVAILABLE` unset, so L2/L3 did not
|
|
run here either — the gate proves the L0/L1 offline subset only).
|
|
- **Why `OAuth2ProxyGateUnobserved` started at 2026-10-04 06:44** rather than 10-02 with
|
|
TargetDown. Alertmanager did not restart (TargetDown's `startsAt` is still 10-02) and the rule's
|
|
`for` is 1h, so neither explains it. Something changed on the monitoring side that morning which
|
|
I did not identify. It does not affect this fix.
|
|
- I did not re-run a port-forward to 44180 to prove the endpoint serves *today*; I confirmed the
|
|
flag and named port on the live pod spec. The 200/10208-byte measurement is podmonitor.yaml's,
|
|
from 2026-10-01.
|