fix(obs): open 44180 to ns monitoring so the forward-auth gate sidecars are actually scrapable #49

2026-10-04T21:01:44Z

gitea-admin commented

2026-10-04 21:01:44 +00:00

The problem

platform/oauth2-proxy/podmonitor.yaml landed on 2026-10-02 and discovered six targets that have never once scraped successfully. Measured on prod 2026-10-04, read-only:

All 6 targets in pool podMonitor/oauth2-proxy/oauth2-proxy-sidecars/0: health=down, lastError: context deadline exceeded, lastScrapeDuration exactly 10.0s — the configured scrapeTimeout. Hitting the timeout precisely means packets are dropped, not refused.
5 × TargetDown firing continuously for 63.7h (ALERTS_FOR_STATE = 229343s), since 2026-10-02 05:09 UTC — 11 minutes after the PodMonitor was created, i.e. for: 10m. A 15-day query_range shows one episode per namespace: this never worked, it is not flapping.
Alertmanager group_by: [alertname, namespace] + repeat_interval: 12h → 5 TargetDown groups = 10 emails/day, plus 6 OAuth2ProxyGateUnobserved groups = 12/day. ~22 emails/day from this one root cause. 0 silences.

Cause. All six app namespaces (jackett radarr sonarr qbittorrent pihole crafty) ship a default-deny NetworkPolicy with policyTypes: [Ingress, Egress] and no ingress key at all — deny-all-ingress in NetworkPolicy semantics. The only ingress allowance is <app>-allow-gateway-ingress: fromEntities: [ingress] → port 4180. Prometheus matched no clause. Zero namespaces had any -allow-monitoring-scrape object.

The listener itself is fine: jackett's oauth2-proxy container carries --metrics-address=0.0.0.0:44180 and a named port metrics: 44180.

The fix

catalog/render.py's r_monitoring_ingress_policy already was the right idiom — but it returned None unless the service declared observability.metricsEndpoint, and no ServiceDefinition in this catalog declares one. It was wired in at the call site and had never emitted a single object on any branch: correct, and entirely latent. Its presence read as "scrape ingress is handled". It was not.

It now also fires for the forward-auth gate sidecar's own metrics port, producing five CNPs — one per forward-auth app — each opening only 44180 to only fromEndpoints: [k8s:io.kubernetes.pod.namespace: monitoring].

Two deliberate choices:

Keyed on _needs_oauth2_sidecar, not on observability.metrics. It must match the condition the scrape keys on: podmonitor.yaml selects pods by the homelab.keithjackson.dev/auth-mode: forward-auth pod label and nothing else, so a target exists iff the sidecar exists. Keying on observability.metrics would let metrics: false produce a discovered-but-blocked target — the same "two halves authored apart" failure this function exists to prevent, moved one field over.
Rendered per-app, not added to platform/oauth2-proxy/. podmonitor.yaml's header records why that tier cannot host it: it would name app namespaces created by the sibling apps-wave-0/1 tiers (oauth2-proxy and apps-wave-0 both dependsOn=[platform-config], no ordering between them), so it would apply on a running cluster and fail on a cold bootstrap. Rendered here it ships in the same tier that creates the namespace.

⚠️ This PR needs a non-author approving review

Regenerating the L1 goldens puts conformance/** in the diff, which is in sensitive_paths ("the tests and the harness; changing them changes what green MEANS"). The golden update is not optional — a renderer change that adds an object necessarily drifts the goldens that pin the renderer's output — but it does mean ci/approval-policy.sh will require a human. The local gate cannot see this: offline it reports approval-policy: green (N/A).

I have deliberately not posted /approve-override. The gitea-admin agent account holds owner and could, but that is exactly the self-approval the stage exists to prevent.

Verification

render.py --selftest: PASS.
The re-render is deterministic. apps/ is byte-identical to render.py --all catalog -o apps --wave-layout --no-image-automation (12/12 SAME before the edit — the 11-file "drift" against a default render is solely the ImageRepository/ImagePolicy pair, which apps/ is generated without). After the edit: 5 files, +21 lines each, nothing else.
--emit-waves deliberately not run (it re-onboards vm-demo + keycloak); no wave kustomization moved.
Goldens regenerated faithfully, +63 / -0. The first gate run caught container-workload-render (R-PLAT-02) and egress-class-render (R-NET-05/06/07) as FAIL with exactly this CNP as the drift. default-deny-baseline-render's golden came back byte-identical (its fixture is not forward-auth) — evidence the regen is faithful, not a rubber stamp.
Local toolbox gate: rc 3 — UNDETERMINED, not 0, and I am not calling it green. Every stage that ran is green (spec-immutability, approval-policy, decisions-log, lint, l0, render-golden, l1), and l1 went FAIL → pass once the goldens were regenerated. Sole not-run stage: gitea-settings (no GITEA_SETTINGS_TOKEN), an environment gap on every local run. L2/L3 conformance is UNVERIFIED for all 8 touched requirements (no cluster).

NOT verified

That the scrape now succeeds. Git-only change; nothing applies until Flux reconciles main. The 5 CNPs have never existed on the cluster, so up == 1 is a prediction from the measured drop signature. After merge, confirm up{container="oauth2-proxy"} → 1 for all 6 targets. send_resolved: true is set, so expect ~11 resolved emails.
That the CNP's endpointSelector matches pihole's pods. It selects app.kubernetes.io/name: pihole, the same shape as pihole-allow-gateway-ingress which demonstrably works for 4180 — inference from a working sibling, not a measurement of the new object.
crafty is unaffected and will keep firing OAuth2ProxyGateUnobserved. Hand-managed under platform/crafty, outside the renderer, with no --metrics-address at all — no port to open. Separate work.
Why OAuth2ProxyGateUnobserved started at 2026-10-04 06:44 rather than 10-02 with TargetDown. Alertmanager did not restart (TargetDown's startsAt is still 10-02) and the rule's for is 1h, so neither explains it. Unidentified; does not affect this fix.

🤖 Generated with Claude Code

## The problem `platform/oauth2-proxy/podmonitor.yaml` landed on 2026-10-02 and discovered six targets that have **never once scraped successfully**. Measured on prod 2026-10-04, read-only: - All 6 targets in pool `podMonitor/oauth2-proxy/oauth2-proxy-sidecars/0`: `health=down`, `lastError: context deadline exceeded`, `lastScrapeDuration` **exactly 10.0s** — the configured `scrapeTimeout`. Hitting the timeout precisely means packets are **dropped, not refused**. - 5 × `TargetDown` firing continuously for **63.7h** (`ALERTS_FOR_STATE` = 229343s), since 2026-10-02 05:09 UTC — 11 minutes after the PodMonitor was created, i.e. `for: 10m`. A 15-day `query_range` shows **one episode** per namespace: this never worked, it is not flapping. - Alertmanager `group_by: [alertname, namespace]` + `repeat_interval: 12h` → 5 TargetDown groups = **10 emails/day**, plus 6 `OAuth2ProxyGateUnobserved` groups = 12/day. **~22 emails/day** from this one root cause. 0 silences. **Cause.** All six app namespaces (`jackett radarr sonarr qbittorrent pihole crafty`) ship a `default-deny` NetworkPolicy with `policyTypes: [Ingress, Egress]` and **no `ingress` key at all** — deny-all-ingress in NetworkPolicy semantics. The only ingress allowance is `<app>-allow-gateway-ingress`: `fromEntities: [ingress]` → port **4180**. Prometheus matched no clause. **Zero** namespaces had any `-allow-monitoring-scrape` object. The listener itself is fine: jackett's `oauth2-proxy` container carries `--metrics-address=0.0.0.0:44180` and a named port `metrics: 44180`. ## The fix `catalog/render.py`'s `r_monitoring_ingress_policy` already was the right idiom — but it returned `None` unless the service declared `observability.metricsEndpoint`, and **no ServiceDefinition in this catalog declares one**. It was wired in at the call site and had **never emitted a single object on any branch**: correct, and entirely latent. Its presence read as "scrape ingress is handled". It was not. It now also fires for the forward-auth gate sidecar's own metrics port, producing five CNPs — one per forward-auth app — each opening **only** 44180 to **only** `fromEndpoints: [k8s:io.kubernetes.pod.namespace: monitoring]`. Two deliberate choices: - **Keyed on `_needs_oauth2_sidecar`, not on `observability.metrics`.** It must match the condition the *scrape* keys on: `podmonitor.yaml` selects pods by the `homelab.keithjackson.dev/auth-mode: forward-auth` pod label and nothing else, so a target exists iff the sidecar exists. Keying on `observability.metrics` would let `metrics: false` produce a discovered-but-blocked target — the same "two halves authored apart" failure this function exists to prevent, moved one field over. - **Rendered per-app, not added to `platform/oauth2-proxy/`.** podmonitor.yaml's header records why that tier cannot host it: it would name app namespaces created by the *sibling* `apps-wave-0/1` tiers (`oauth2-proxy` and `apps-wave-0` both `dependsOn=[platform-config]`, no ordering between them), so it would apply on a running cluster and fail on a cold bootstrap. Rendered here it ships in the same tier that creates the namespace. ## ⚠️ This PR needs a non-author approving review Regenerating the L1 goldens puts **`conformance/**` in the diff**, which is in `sensitive_paths` ("the tests and the harness; changing them changes what green MEANS"). The golden update is not optional — a renderer change that adds an object necessarily drifts the goldens that pin the renderer's output — but it does mean `ci/approval-policy.sh` will require a human. The local gate cannot see this: offline it reports `approval-policy: green` (N/A). I have deliberately **not** posted `/approve-override`. The `gitea-admin` agent account holds `owner` and *could*, but that is exactly the self-approval the stage exists to prevent. ## Verification - `render.py --selftest`: **PASS**. - **The re-render is deterministic.** `apps/` is byte-identical to `render.py --all catalog -o apps --wave-layout --no-image-automation` (12/12 SAME *before* the edit — the 11-file "drift" against a default render is solely the ImageRepository/ImagePolicy pair, which `apps/` is generated without). After the edit: 5 files, +21 lines each, nothing else. - `--emit-waves` deliberately **not** run (it re-onboards `vm-demo` + `keycloak`); no wave kustomization moved. - **Goldens regenerated faithfully, +63 / -0.** The first gate run caught `container-workload-render` (R-PLAT-02) and `egress-class-render` (R-NET-05/06/07) as FAIL with exactly this CNP as the drift. `default-deny-baseline-render`'s golden came back **byte-identical** (its fixture is not forward-auth) — evidence the regen is faithful, not a rubber stamp. - **Local toolbox gate: rc 3 — UNDETERMINED, not 0, and I am not calling it green.** Every stage that ran is green (spec-immutability, approval-policy, decisions-log, lint, l0, render-golden, l1), and `l1` went FAIL → pass once the goldens were regenerated. Sole not-run stage: `gitea-settings` (no `GITEA_SETTINGS_TOKEN`), an environment gap on every local run. L2/L3 conformance is UNVERIFIED for all 8 touched requirements (no cluster). ## NOT verified - **That the scrape now succeeds.** Git-only change; nothing applies until Flux reconciles `main`. The 5 CNPs have never existed on the cluster, so `up == 1` is a **prediction** from the measured drop signature. After merge, confirm `up{container="oauth2-proxy"}` → 1 for all 6 targets. `send_resolved: true` is set, so expect ~11 resolved emails. - **That the CNP's `endpointSelector` matches pihole's pods.** It selects `app.kubernetes.io/name: pihole`, the same shape as `pihole-allow-gateway-ingress` which demonstrably works for 4180 — inference from a working sibling, not a measurement of the new object. - **`crafty` is unaffected and will keep firing `OAuth2ProxyGateUnobserved`.** Hand-managed under `platform/crafty`, outside the renderer, with **no `--metrics-address` at all** — no port to open. Separate work. - **Why `OAuth2ProxyGateUnobserved` started at 2026-10-04 06:44** rather than 10-02 with TargetDown. Alertmanager did not restart (TargetDown's `startsAt` is still 10-02) and the rule's `for` is 1h, so neither explains it. Unidentified; does not affect this fix. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea-admin added 1 commit 2026-10-04 21:01:44 +00:00

fix(obs): open 44180 to ns monitoring so the forward-auth gate sidecars are actually scrapable

gate / gate (pull_request) Failing after 1m51s

Details

gate / conformance (pull_request) Has been skipped

Details

f9446be381

platform/oauth2-proxy/podmonitor.yaml landed 2026-10-02 and discovered six targets that
have never once scraped. Measured on prod 2026-10-04: all six read health=down with
lastError "context deadline exceeded" and lastScrapeDuration exactly 10.0s — the
configured scrapeTimeout, i.e. packets DROPPED rather than refused.

Cause: every app namespace ships a default-deny NetworkPolicy with
policyTypes: [Ingress, Egress] and NO ingress key at all (deny-all-ingress), and the only
ingress allowance is <app>-allow-gateway-ingress: fromEntities: [ingress] -> 4180.
Prometheus matched no clause. Verified identical in all six of jackett, radarr, sonarr,
qbittorrent, pihole, crafty; zero namespaces had any -allow-monitoring-scrape object.

render.py already had the right idiom — r_monitoring_ingress_policy — but it returned None
unless the service declared observability.metricsEndpoint, and NO ServiceDefinition in this
catalog declares one. It was wired in at the call site and had never emitted a single
object on any branch: correct, and entirely latent.

It now also fires for the forward-auth gate sidecar's own metrics port, keyed on
_needs_oauth2_sidecar — deliberately the same condition the scrape keys on, since
podmonitor.yaml selects pods purely by the auth-mode label, so a target exists iff the
sidecar exists. Keying on observability.metrics instead would let metrics: false produce a
discovered-but-blocked target: the same "two halves authored apart" failure this function
exists to prevent.

Rendered here rather than in platform/oauth2-proxy because that tier would have to name app
namespaces created by the sibling apps-wave-0/1 tiers — it would apply on a running cluster
and fail on a cold bootstrap. podmonitor.yaml's header records that objection; rendering
per-app dissolves it.

Five CNPs result, one per forward-auth app, each opening only 44180 to only ns monitoring.
apps/ re-rendered with the invocation it was generated by (--no-image-automation, verified
byte-identical beforehand); --emit-waves deliberately not run, so no wave kustomization moved.

Three L1 goldens are regenerated with it (+63 lines, -0): a renderer change that adds an
object necessarily drifts the goldens that pin the renderer's output. The first gate run
caught container-workload-render and egress-class-render as FAIL with exactly this CNP as
the drift. default-deny-baseline-render's golden regenerated byte-identical, which is the
evidence that the regen is faithful and not a rubber stamp.

That puts conformance/** in the diff, which is a sensitive path: this PR needs a non-author
approving review. The local gate cannot see that — offline it reports approval-policy green
(N/A).

Local toolbox gate: rc 3 UNDETERMINED, not 0. Every stage that ran is green and l1 went
FAIL -> pass with the goldens; the only not-run stage is gitea-settings (no
GITEA_SETTINGS_TOKEN). L2/L3 conformance is UNVERIFIED for all 8 touched requirements.

Not verified: that the scrape now succeeds. Nothing is applied until Flux reconciles main.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

gitea-admin force-pushed fix/oauth2-sidecar-metrics-scrape-netpol from f9446be381 to b156ee6cfa

2026-10-04 21:07:35 +00:00

Compare

jakman85 commented

2026-10-04 21:31:17 +00:00

/approve-override renderer adds one CNP per forward-auth app; the three conformance goldens are a mechanical regen of that same output

jakman85 merged commit 21c88b0dc7 into main

2026-10-04 21:33:29 +00:00

jakman85 referenced this issue from a commit

2026-10-04 21:33:31 +00:00

Merge pull request 'fix(obs): open 44180 to ns monitoring so the forward-auth gate sidecars are actually scrapable' (#49) from fix/oauth2-sidecar-metrics-scrape-netpol into main

gitea-admin referenced this issue from a commit