---
date: 2026-10-04
branch: fix/oauth2-sidecar-metrics-scrape-netpol
title: The forward-auth gate sidecars' /metrics was NetworkPolicy-blocked — the renderer now opens 44180 to ns monitoring
---

> **Added 2026-10-05, after merge:** the top item under *What I did NOT verify* below — "that the
> scrape now succeeds" — **has since been verified on prod**. All 6 targets read `up == 1`. The
> original text is left exactly as written; the measurements are in
> [[2026-10-05-oauth2-sidecar-scrape-verified]].

## What changed

`catalog/render.py`'s `r_monitoring_ingress_policy` now emits its CiliumNetworkPolicy for a
**second** reason: the forward-auth gate sidecar's own `/metrics` port (44180), keyed on
`_needs_oauth2_sidecar(s)`. Previously it fired only from `observability.metricsEndpoint`.

That matters more than it sounds, because **no ServiceDefinition in this catalog declares a
`metricsEndpoint`** (measured: zero grep hits across `catalog/*.yaml`). So the function was wired
in at the call site and had **never emitted a single object on any branch** — correct, and
entirely latent. Its presence in the tree read as "scrape ingress is handled". It was not.

Re-rendered `apps/` from that change: five new CNPs, one per forward-auth app
(`jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`), each opening **only** 44180 to **only**
`fromEndpoints: [k8s:io.kubernetes.pod.namespace: monitoring]`.

The bug being fixed: `platform/oauth2-proxy/podmonitor.yaml` landed 2026-10-02T04:58:44Z and
discovered 6 targets that have **never once scraped successfully**. Each app namespace ships a
`default-deny` NetworkPolicy with `policyTypes: [Ingress, Egress]` and **no `ingress` key at all**
— deny-all-ingress in NetworkPolicy semantics — and the only ingress allowance is
`<app>-allow-gateway-ingress`: `fromEntities: [ingress]` to port 4180. Prometheus matched no
clause, so its packets were dropped.

Two deliberate choices, both recorded in the function's docstring:

- **Clause 2 keys on `_needs_oauth2_sidecar`, not on `observability.metrics`.** It has to match
  the condition the *scrape* keys on: `podmonitor.yaml` selects pods by the
  `homelab.keithjackson.dev/auth-mode: forward-auth` pod label and nothing else, so a target
  exists iff the sidecar exists. Keying on `observability.metrics` would let `metrics: false`
  produce a discovered-but-blocked target — the same "two halves authored apart" failure this
  function exists to prevent, moved one field over.
- **The policy is rendered here, not added to `platform/oauth2-proxy/`.** podmonitor.yaml's own
  header records why that tier cannot host it: it would name app namespaces created by the
  *sibling* `apps-wave-0/1` tiers, so it would apply on a running cluster and fail on a cold
  bootstrap. Rendered per-app it ships in the same tier that creates the namespace.

## What I verified

Measured on prod 2026-10-04, read-only, via the pinned toolbox:

- **The cause, not a guess.** All 6 sidecar targets in pool
  `podMonitor/oauth2-proxy/oauth2-proxy-sidecars/0`: `health=down`,
  `lastError: context deadline exceeded`, `lastScrapeDuration` exactly `10.0s` — the configured
  `scrapeTimeout`. Hitting the timeout precisely means packets **dropped, not refused**.
- **The listener is fine.** jackett's `oauth2-proxy` container carries
  `--metrics-address=0.0.0.0:44180` and a named port `metrics: 44180`.
- **The deny is universal.** All six namespaces (`jackett radarr sonarr qbittorrent pihole
  crafty`): `default-deny` with `ingress` key absent, `nIngress=0`; the only ingress CNP allowance
  is 4180 `fromEntities: [ingress]`. **Zero** namespaces had any `-allow-monitoring-scrape` object.
- **Frequency.** Single continuous episode, no flapping: `ALERTS_FOR_STATE` = 229343s (63.7h),
  firing since 2026-10-02 05:09 UTC — 11 minutes after the PodMonitor was created, i.e. TargetDown's
  `for: 10m`. A 15-day `query_range` shows **1 episode** per namespace, not many. Alertmanager:
  `group_by: [alertname, namespace]`, `repeat_interval: 12h` on `email-warning` → 5 TargetDown
  groups = 10 emails/day, plus 6 `OAuth2ProxyGateUnobserved` groups = 12/day, ~22/day total.
  0 silences.
- **The re-render is deterministic and the diff is exactly the intended one.** `apps/` is
  byte-identical to `render.py --all catalog -o apps --wave-layout --no-image-automation`
  (12/12 SAME before my edit; the 11-file "drift" against a default render is solely the
  ImageRepository/ImagePolicy pair, which `apps/` is generated without). After the edit:
  5 files changed, +21 lines each, nothing else touched.
- `render.py --selftest`: **PASS**.
- **Three L1 goldens needed regenerating, and the diff is +63 lines / -0.** The first gate run
  caught `container-workload-render` (R-PLAT-02) and `egress-class-render` (R-NET-05/06/07) as
  FAIL with exactly the new CNP as the drift. Regenerated with the same invocations the checks
  use (`render.py <fixture>`, no flags). Evidence the regen is faithful rather than a rubber
  stamp: `default-deny-baseline-render`'s golden came back **byte-identical** (its fixture is not
  forward-auth), and the other three gained only the one CNP, port 44180, ns monitoring.
- Toolbox gate `./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`: **rc 3 — UNDETERMINED,
  NOT rc 0, and I am not reporting it as green.** Every stage that ran is green
  (spec-immutability, approval-policy, decisions-log, lint, l0, render-golden, l1); `l1` went
  FAIL -> pass once the goldens were regenerated. The gate's own words: "Stages that did not run:
  gitea-settings" — skipped for want of `GITEA_SETTINGS_TOKEN`, which is an environment gap on
  every local run and unrelated to this change. L2/L3 conformance is a `[warn]`, not a not-run
  stage: UNVERIFIED for all 8 touched requirements (R-GITOPS-02 R-IDP-02 R-IDP-06 R-NET-01
  R-NET-05 R-NET-06 R-NET-07 R-PLAT-02), no cluster.
- `--emit-waves` was **not** run (it re-onboards `vm-demo` + `keycloak`); only the per-service
  render path was used, so no wave kustomization changed.

## This PR needs a human approval (sensitive path) — MEASURED, not predicted

CI run #230 on PR #49, head `f9446be3`: **`MERGE GATE: BLOCKED`, and `approval-policy` is the ONLY
failing stage.** Read from the job log, not from the status tick:

    [skip] stage spec-immutability: SKIPPED (not run)   <- sibling spec repo absent in CI; expected
    [ ok ] stage decisions-log: pass
    [ ok ] stage lint: pass
    [ ok ] stage l0: pass
    [ ok ] stage render-golden: pass
    [ ok ] stage l1: pass
    [fail] stage approval-policy: FAIL
    [ ok ] stage gitea-settings: pass                   <- could NOT run locally; green in CI
    [skip] stage conformance: SKIPPED (not run)

    sensitive paths touched:
      conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml
      conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml
      conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml

Exactly the three regenerated goldens, nothing else. `l1` and `render-golden` both pass in CI,
which is the independent confirmation that the regenerated goldens are right rather than merely
self-consistent with my local run.

One local-reproduction note worth recording, because it wasted a step: running
`PR_NUMBER=49 SETTINGS_API_TOKEN=... ci/approval-policy.sh` with the credential from
`git credential fill` reports `could not determine PR author — BLOCKING`. That is an **artifact**,
not a finding — `~/.git-credentials` holds a *password*, and `Authorization: token <password>`
returns 401 `user does not exist [uid: 0]`. The stage needs a real API token. I nearly reported
that 401 as the CI failure; the job log is what settled it.

Regenerating the L1 goldens means this branch touches `conformance/**`, which is in
`sensitive_paths` in `infra/gitea/approval-policy.json` ("the tests and the harness; changing them
changes what green MEANS"). So `ci/approval-policy.sh` will require an approving review from a
non-author, and the local gate **cannot** see that: it reports `approval-policy: green` because
offline it evaluates N/A.

The golden update is not optional — a renderer change that adds an object necessarily drifts the
goldens that pin the renderer's output — but it does mean the gate correctly refuses to
self-approve this one. **An agent must not clear this with `/approve-override`**, even though the
`gitea-admin` account holds `owner`: that is precisely the self-approval the stage exists to
prevent. The operator posts the override as a PR *review* of type "Comment", or approves normally.

## What I did NOT verify

- **That the scrape now succeeds.** This is a git-only change and nothing is applied until Flux
  reconciles `main`. The 5 CNPs have never existed on the cluster, so `up == 1` for those targets
  is **UNVERIFIED** — it is a prediction from the measured drop signature, not an observation.
  After merge, confirm `up{container="oauth2-proxy"}` goes to 1 for all 6 targets and that the 11
  alerts resolve. `send_resolved: true` is set, so expect ~11 resolved emails.
- **That the CNP `endpointSelector` matches pihole's pods.** It selects
  `app.kubernetes.io/name: pihole`, identical to the shape of `pihole-allow-gateway-ingress`
  which demonstrably works for 4180 — so this is inference from a working sibling, not a
  measurement of the new object.
- **`crafty` is still unobserved and will keep firing `OAuth2ProxyGateUnobserved`.** It is
  hand-managed under `platform/crafty`, outside the renderer, and has **no `--metrics-address`
  at all** — so there is no port to open and this change cannot help it. Separate work.
- **Whether 44180 is reachable in the Cilium data path once allowed.** Only the policy was
  reasoned about; no L2 probe was run (CI has `vars.CLUSTER_AVAILABLE` unset, so L2/L3 did not
  run here either — the gate proves the L0/L1 offline subset only).
- **Why `OAuth2ProxyGateUnobserved` started at 2026-10-04 06:44** rather than 10-02 with
  TargetDown. Alertmanager did not restart (TargetDown's `startsAt` is still 10-02) and the rule's
  `for` is 1h, so neither explains it. Something changed on the monitoring side that morning which
  I did not identify. It does not affect this fix.
- I did not re-run a port-forward to 44180 to prove the endpoint serves *today*; I confirmed the
  flag and named port on the live pod spec. The 200/10208-byte measurement is podmonitor.yaml's,
  from 2026-10-01.