home-network-k3s-impl/decisions/2026-10-04-oauth2-sidecar-metrics-scrape-netpol.md

date, branch, title

date	branch	title
2026-10-04	fix/oauth2-sidecar-metrics-scrape-netpol	The forward-auth gate sidecars' /metrics was NetworkPolicy-blocked — the renderer now opens 44180 to ns monitoring

Added 2026-10-05, after merge: the top item under What I did NOT verify below — "that the scrape now succeeds" — has since been verified on prod. All 6 targets read up == 1. The original text is left exactly as written; the measurements are in 2026-10-05-oauth2-sidecar-scrape-verified.

What changed

catalog/render.py's r_monitoring_ingress_policy now emits its CiliumNetworkPolicy for a second reason: the forward-auth gate sidecar's own /metrics port (44180), keyed on _needs_oauth2_sidecar(s). Previously it fired only from observability.metricsEndpoint.

That matters more than it sounds, because no ServiceDefinition in this catalog declares a metricsEndpoint (measured: zero grep hits across catalog/*.yaml). So the function was wired in at the call site and had never emitted a single object on any branch — correct, and entirely latent. Its presence in the tree read as "scrape ingress is handled". It was not.

Re-rendered apps/ from that change: five new CNPs, one per forward-auth app (jackett, pihole, qbittorrent, radarr, sonarr), each opening only 44180 to only fromEndpoints: [k8s:io.kubernetes.pod.namespace: monitoring].

The bug being fixed: platform/oauth2-proxy/podmonitor.yaml landed 2026-10-02T04:58:44Z and discovered 6 targets that have never once scraped successfully. Each app namespace ships a default-deny NetworkPolicy with policyTypes: [Ingress, Egress] and no ingress key at all — deny-all-ingress in NetworkPolicy semantics — and the only ingress allowance is <app>-allow-gateway-ingress: fromEntities: [ingress] to port 4180. Prometheus matched no clause, so its packets were dropped.

Two deliberate choices, both recorded in the function's docstring:

• Clause 2 keys on _needs_oauth2_sidecar, not on observability.metrics. It has to match the condition the scrape keys on: podmonitor.yaml selects pods by the homelab.keithjackson.dev/auth-mode: forward-auth pod label and nothing else, so a target exists iff the sidecar exists. Keying on observability.metrics would let metrics: false produce a discovered-but-blocked target — the same "two halves authored apart" failure this function exists to prevent, moved one field over.
• The policy is rendered here, not added to platform/oauth2-proxy/. podmonitor.yaml's own header records why that tier cannot host it: it would name app namespaces created by the sibling apps-wave-0/1 tiers, so it would apply on a running cluster and fail on a cold bootstrap. Rendered per-app it ships in the same tier that creates the namespace.

What I verified

Measured on prod 2026-10-04, read-only, via the pinned toolbox:

• The cause, not a guess. All 6 sidecar targets in pool podMonitor/oauth2-proxy/oauth2-proxy-sidecars/0: health=down, lastError: context deadline exceeded, lastScrapeDuration exactly 10.0s — the configured scrapeTimeout. Hitting the timeout precisely means packets dropped, not refused.
• The listener is fine. jackett's oauth2-proxy container carries --metrics-address=0.0.0.0:44180 and a named port metrics: 44180.
• The deny is universal. All six namespaces (jackett radarr sonarr qbittorrent pihole crafty): default-deny with ingress key absent, nIngress=0; the only ingress CNP allowance is 4180 fromEntities: [ingress]. Zero namespaces had any -allow-monitoring-scrape object.
• Frequency. Single continuous episode, no flapping: ALERTS_FOR_STATE = 229343s (63.7h), firing since 2026-10-02 05:09 UTC — 11 minutes after the PodMonitor was created, i.e. TargetDown's for: 10m. A 15-day query_range shows 1 episode per namespace, not many. Alertmanager: group_by: [alertname, namespace], repeat_interval: 12h on email-warning → 5 TargetDown groups = 10 emails/day, plus 6 OAuth2ProxyGateUnobserved groups = 12/day, ~22/day total. 0 silences.
• The re-render is deterministic and the diff is exactly the intended one. apps/ is byte-identical to render.py --all catalog -o apps --wave-layout --no-image-automation (12/12 SAME before my edit; the 11-file "drift" against a default render is solely the ImageRepository/ImagePolicy pair, which apps/ is generated without). After the edit: 5 files changed, +21 lines each, nothing else touched.
• render.py --selftest: PASS.
• Three L1 goldens needed regenerating, and the diff is +63 lines / -0. The first gate run caught container-workload-render (R-PLAT-02) and egress-class-render (R-NET-05/06/07) as FAIL with exactly the new CNP as the drift. Regenerated with the same invocations the checks use (render.py <fixture>, no flags). Evidence the regen is faithful rather than a rubber stamp: default-deny-baseline-render's golden came back byte-identical (its fixture is not forward-auth), and the other three gained only the one CNP, port 44180, ns monitoring.
• Toolbox gate ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh': rc 3 — UNDETERMINED, NOT rc 0, and I am not reporting it as green. Every stage that ran is green (spec-immutability, approval-policy, decisions-log, lint, l0, render-golden, l1); l1 went FAIL -> pass once the goldens were regenerated. The gate's own words: "Stages that did not run: gitea-settings" — skipped for want of GITEA_SETTINGS_TOKEN, which is an environment gap on every local run and unrelated to this change. L2/L3 conformance is a [warn], not a not-run stage: UNVERIFIED for all 8 touched requirements (R-GITOPS-02 R-IDP-02 R-IDP-06 R-NET-01 R-NET-05 R-NET-06 R-NET-07 R-PLAT-02), no cluster.
• --emit-waves was not run (it re-onboards vm-demo + keycloak); only the per-service render path was used, so no wave kustomization changed.

This PR needs a human approval (sensitive path) — MEASURED, not predicted

CI run #230 on PR #49, head f9446be3: MERGE GATE: BLOCKED, and approval-policy is the ONLY failing stage. Read from the job log, not from the status tick:

[skip] stage spec-immutability: SKIPPED (not run)   <- sibling spec repo absent in CI; expected
[ ok ] stage decisions-log: pass
[ ok ] stage lint: pass
[ ok ] stage l0: pass
[ ok ] stage render-golden: pass
[ ok ] stage l1: pass
[fail] stage approval-policy: FAIL
[ ok ] stage gitea-settings: pass                   <- could NOT run locally; green in CI
[skip] stage conformance: SKIPPED (not run)

sensitive paths touched:
  conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml
  conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml
  conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml

Exactly the three regenerated goldens, nothing else. l1 and render-golden both pass in CI, which is the independent confirmation that the regenerated goldens are right rather than merely self-consistent with my local run.

One local-reproduction note worth recording, because it wasted a step: running PR_NUMBER=49 SETTINGS_API_TOKEN=... ci/approval-policy.sh with the credential from git credential fill reports could not determine PR author — BLOCKING. That is an artifact, not a finding — ~/.git-credentials holds a password, and Authorization: token <password> returns 401 user does not exist [uid: 0]. The stage needs a real API token. I nearly reported that 401 as the CI failure; the job log is what settled it.

Regenerating the L1 goldens means this branch touches conformance/**, which is in sensitive_paths in infra/gitea/approval-policy.json ("the tests and the harness; changing them changes what green MEANS"). So ci/approval-policy.sh will require an approving review from a non-author, and the local gate cannot see that: it reports approval-policy: green because offline it evaluates N/A.

The golden update is not optional — a renderer change that adds an object necessarily drifts the goldens that pin the renderer's output — but it does mean the gate correctly refuses to self-approve this one. An agent must not clear this with /approve-override, even though the gitea-admin account holds owner: that is precisely the self-approval the stage exists to prevent. The operator posts the override as a PR review of type "Comment", or approves normally.

What I did NOT verify

• That the scrape now succeeds. This is a git-only change and nothing is applied until Flux reconciles main. The 5 CNPs have never existed on the cluster, so up == 1 for those targets is UNVERIFIED — it is a prediction from the measured drop signature, not an observation. After merge, confirm up{container="oauth2-proxy"} goes to 1 for all 6 targets and that the 11 alerts resolve. send_resolved: true is set, so expect ~11 resolved emails.
• That the CNP endpointSelector matches pihole's pods. It selects app.kubernetes.io/name: pihole, identical to the shape of pihole-allow-gateway-ingress which demonstrably works for 4180 — so this is inference from a working sibling, not a measurement of the new object.
• crafty is still unobserved and will keep firing OAuth2ProxyGateUnobserved. It is hand-managed under platform/crafty, outside the renderer, and has no --metrics-address at all — so there is no port to open and this change cannot help it. Separate work.
• Whether 44180 is reachable in the Cilium data path once allowed. Only the policy was reasoned about; no L2 probe was run (CI has vars.CLUSTER_AVAILABLE unset, so L2/L3 did not run here either — the gate proves the L0/L1 offline subset only).
• Why OAuth2ProxyGateUnobserved started at 2026-10-04 06:44 rather than 10-02 with TargetDown. Alertmanager did not restart (TargetDown's startsAt is still 10-02) and the rule's for is 1h, so neither explains it. Something changed on the monitoring side that morning which I did not identify. It does not affect this fix.
• I did not re-run a port-forward to 44180 to prove the endpoint serves today; I confirmed the flag and named port on the live pod spec. The 200/10208-byte measurement is podmonitor.yaml's, from 2026-10-01.