diff --git a/apps/wave-0/jackett.yaml b/apps/wave-0/jackett.yaml index 4d3583b..50a10d6 100644 --- a/apps/wave-0/jackett.yaml +++ b/apps/wave-0/jackett.yaml @@ -244,6 +244,27 @@ spec: - port: '4180' protocol: TCP --- +# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): jackett-allow-monitoring-scrape +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: jackett-allow-monitoring-scrape + namespace: jackett + labels: + app.kubernetes.io/name: jackett +spec: + endpointSelector: + matchLabels: + app.kubernetes.io/name: jackett + ingress: + - fromEndpoints: + - matchLabels: + k8s:io.kubernetes.pod.namespace: monitoring + toPorts: + - ports: + - port: '44180' + protocol: TCP +--- # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): jackett-allow-internet-egress apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy diff --git a/apps/wave-0/pihole.yaml b/apps/wave-0/pihole.yaml index adeaa8c..025be47 100644 --- a/apps/wave-0/pihole.yaml +++ b/apps/wave-0/pihole.yaml @@ -232,6 +232,27 @@ spec: - port: '4180' protocol: TCP --- +# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): pihole-allow-monitoring-scrape +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: pihole-allow-monitoring-scrape + namespace: pihole + labels: + app.kubernetes.io/name: pihole +spec: + endpointSelector: + matchLabels: + app.kubernetes.io/name: pihole + ingress: + - fromEndpoints: + - matchLabels: + k8s:io.kubernetes.pod.namespace: monitoring + toPorts: + - ports: + - port: '44180' + protocol: TCP +--- # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): pihole-allow-internet-egress apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy diff --git a/apps/wave-0/qbittorrent.yaml b/apps/wave-0/qbittorrent.yaml index e6f19d8..2964348 100644 --- a/apps/wave-0/qbittorrent.yaml +++ b/apps/wave-0/qbittorrent.yaml @@ -319,6 +319,27 @@ spec: - port: '4180' protocol: TCP --- +# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): qbittorrent-allow-monitoring-scrape +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: qbittorrent-allow-monitoring-scrape + namespace: qbittorrent + labels: + app.kubernetes.io/name: qbittorrent +spec: + endpointSelector: + matchLabels: + app.kubernetes.io/name: qbittorrent + ingress: + - fromEndpoints: + - matchLabels: + k8s:io.kubernetes.pod.namespace: monitoring + toPorts: + - ports: + - port: '44180' + protocol: TCP +--- # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): qbittorrent-allow-internet-egress apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy diff --git a/apps/wave-1/radarr.yaml b/apps/wave-1/radarr.yaml index bd46bdb..7cb9ea3 100644 --- a/apps/wave-1/radarr.yaml +++ b/apps/wave-1/radarr.yaml @@ -347,6 +347,27 @@ spec: - port: '4180' protocol: TCP --- +# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): radarr-allow-monitoring-scrape +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: radarr-allow-monitoring-scrape + namespace: radarr + labels: + app.kubernetes.io/name: radarr +spec: + endpointSelector: + matchLabels: + app.kubernetes.io/name: radarr + ingress: + - fromEndpoints: + - matchLabels: + k8s:io.kubernetes.pod.namespace: monitoring + toPorts: + - ports: + - port: '44180' + protocol: TCP +--- # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): radarr-allow-internet-egress apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy diff --git a/apps/wave-1/sonarr.yaml b/apps/wave-1/sonarr.yaml index 9f14bd9..4d87ac5 100644 --- a/apps/wave-1/sonarr.yaml +++ b/apps/wave-1/sonarr.yaml @@ -345,6 +345,27 @@ spec: - port: '4180' protocol: TCP --- +# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: sonarr-allow-monitoring-scrape + namespace: sonarr + labels: + app.kubernetes.io/name: sonarr +spec: + endpointSelector: + matchLabels: + app.kubernetes.io/name: sonarr + ingress: + - fromEndpoints: + - matchLabels: + k8s:io.kubernetes.pod.namespace: monitoring + toPorts: + - ports: + - port: '44180' + protocol: TCP +--- # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy diff --git a/catalog/render.py b/catalog/render.py index e8b668d..894d6f0 100644 --- a/catalog/render.py +++ b/catalog/render.py @@ -1541,12 +1541,54 @@ def r_monitoring_ingress_policy(s: dict): trap Longhorn's own NetworkPolicies set (platform/longhorn/networkpolicy-metrics.yaml exists for exactly this reason), and it is a THIRD independent way for a metrics SLI to be inert. - Rendered only alongside a declared metricsEndpoint, and it opens ONLY that port to ONLY the - monitoring namespace: an allow for a port nothing serves would be a standing hole for no + Rendered only alongside a port something actually serves, and it opens ONLY that port to ONLY + the monitoring namespace: an allow for a port nothing serves would be a standing hole for no benefit. + + TWO INDEPENDENT SOURCES OF A SERVED METRICS PORT, and this used to cover only the first: + + 1. `observability.metricsEndpoint` — the APP's own exporter. NOTE, measured 2026-10-04: no + ServiceDefinition in this catalog declares one, so before the sidecar clause below this + function had NEVER emitted a single object on any branch. It is correct but, so far, + entirely latent — do not read its presence as evidence that scrape ingress is solved. + + 2. THE forward-auth GATE SIDECAR's own `/metrics` on 44180 (added 2026-10-04). Every + `_needs_oauth2_sidecar` app gets `--metrics-address=0.0.0.0:{OAUTH2_PROXY_METRICS_PORT}` + and a named `metrics` container port from `_oauth2_sidecar_container`, so the port is + served whenever the sidecar exists — unconditionally, which is what makes it safe to + open without a per-app declaration. + + WHY CLAUSE 2 KEYS ON `_needs_oauth2_sidecar` AND NOT ON `observability.metrics`: it must match + the condition the SCRAPE keys on, exactly. platform/oauth2-proxy/podmonitor.yaml selects pods + by the `homelab.keithjackson.dev/auth-mode: forward-auth` POD LABEL and nothing else, so a + target exists iff the sidecar exists. Keying the allowance on `observability.metrics` instead + would let `metrics: false` produce a discovered-but-blocked target — the same "two halves + authored apart" failure this function exists to prevent, just moved one field over. + + THIS CLAUSE IS A BUG FIX, and the bug was live for 63.7h. Measured on prod 2026-10-04: all 6 + sidecar targets across jackett/radarr/sonarr/qbittorrent/pihole read `health=down`, + `lastError: context deadline exceeded`, `lastScrapeDuration` exactly 10.0s — the scrapeTimeout, + i.e. packets DROPPED, not refused. Each app namespace ships a `default-deny` NetworkPolicy + with `policyTypes: [Ingress, Egress]` and NO `ingress` key at all (deny-all-ingress), and the + only ingress allowance is `-allow-gateway-ingress`: `fromEntities: [ingress]` -> 4180. + Prometheus matched no clause. 5 x `TargetDown` + 6 x `OAuth2ProxyGateUnobserved` had been + re-notifying at ~22 emails/day since the PodMonitor landed on 2026-10-02. + + WHY THE POLICY BELONGS HERE AND NOT IN platform/oauth2-proxy: podmonitor.yaml's header records + the objection — a policy in that tier would name app namespaces created by the SIBLING + apps-wave-0/1 tiers (measured Flux graph: `oauth2-proxy` and `apps-wave-0` both + `dependsOn=[platform-config]`, no ordering between them), so it would apply on a running + cluster and fail on a cold bootstrap. Rendered here it ships in the same tier that creates the + namespace, which dissolves that ordering problem entirely. """ - ep = s["observability"].get("metricsEndpoint") - if not ep or not s["observability"]["metrics"]: + obs = s["observability"] + ep = obs.get("metricsEndpoint") + ports: list[str] = [] + if ep and obs["metrics"]: + ports.append(str(ep["port"])) + if _needs_oauth2_sidecar(s): + ports.append(str(OAUTH2_PROXY_METRICS_PORT)) + if not ports: return None return { "apiVersion": "cilium.io/v2", @@ -1564,7 +1606,7 @@ def r_monitoring_ingress_policy(s: dict): {"matchLabels": {"k8s:io.kubernetes.pod.namespace": MONITORING_NS}} ], "toPorts": [ - {"ports": [{"port": str(ep["port"]), "protocol": "TCP"}]} + {"ports": [{"port": p, "protocol": "TCP"} for p in ports]} ], } ], diff --git a/conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml b/conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml index bf3e82d..d223564 100644 --- a/conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml +++ b/conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml @@ -171,6 +171,27 @@ spec: - port: '4180' protocol: TCP --- +# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): paas-demo-allow-monitoring-scrape +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: paas-demo-allow-monitoring-scrape + namespace: paas-demo + labels: + app.kubernetes.io/name: paas-demo +spec: + endpointSelector: + matchLabels: + app.kubernetes.io/name: paas-demo + ingress: + - fromEndpoints: + - matchLabels: + k8s:io.kubernetes.pod.namespace: monitoring + toPorts: + - ports: + - port: '44180' + protocol: TCP +--- # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): paas-demo-allow-internet-egress apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy diff --git a/conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml b/conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml index b32ea49..7d9c51b 100644 --- a/conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml +++ b/conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml @@ -198,6 +198,27 @@ spec: - port: '4180' protocol: TCP --- +# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: sonarr-allow-monitoring-scrape + namespace: sonarr + labels: + app.kubernetes.io/name: sonarr +spec: + endpointSelector: + matchLabels: + app.kubernetes.io/name: sonarr + ingress: + - fromEndpoints: + - matchLabels: + k8s:io.kubernetes.pod.namespace: monitoring + toPorts: + - ports: + - port: '44180' + protocol: TCP +--- # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy diff --git a/conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml b/conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml index 692e8dd..d88db79 100644 --- a/conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml +++ b/conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml @@ -198,6 +198,27 @@ spec: - port: '4180' protocol: TCP --- +# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: sonarr-allow-monitoring-scrape + namespace: sonarr + labels: + app.kubernetes.io/name: sonarr +spec: + endpointSelector: + matchLabels: + app.kubernetes.io/name: sonarr + ingress: + - fromEndpoints: + - matchLabels: + k8s:io.kubernetes.pod.namespace: monitoring + toPorts: + - ports: + - port: '44180' + protocol: TCP +--- # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy diff --git a/decisions/2026-10-04-oauth2-sidecar-metrics-scrape-netpol.md b/decisions/2026-10-04-oauth2-sidecar-metrics-scrape-netpol.md new file mode 100644 index 0000000..5d7a83c --- /dev/null +++ b/decisions/2026-10-04-oauth2-sidecar-metrics-scrape-netpol.md @@ -0,0 +1,150 @@ +--- +date: 2026-10-04 +branch: fix/oauth2-sidecar-metrics-scrape-netpol +title: The forward-auth gate sidecars' /metrics was NetworkPolicy-blocked — the renderer now opens 44180 to ns monitoring +--- + +## What changed + +`catalog/render.py`'s `r_monitoring_ingress_policy` now emits its CiliumNetworkPolicy for a +**second** reason: the forward-auth gate sidecar's own `/metrics` port (44180), keyed on +`_needs_oauth2_sidecar(s)`. Previously it fired only from `observability.metricsEndpoint`. + +That matters more than it sounds, because **no ServiceDefinition in this catalog declares a +`metricsEndpoint`** (measured: zero grep hits across `catalog/*.yaml`). So the function was wired +in at the call site and had **never emitted a single object on any branch** — correct, and +entirely latent. Its presence in the tree read as "scrape ingress is handled". It was not. + +Re-rendered `apps/` from that change: five new CNPs, one per forward-auth app +(`jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`), each opening **only** 44180 to **only** +`fromEndpoints: [k8s:io.kubernetes.pod.namespace: monitoring]`. + +The bug being fixed: `platform/oauth2-proxy/podmonitor.yaml` landed 2026-10-02T04:58:44Z and +discovered 6 targets that have **never once scraped successfully**. Each app namespace ships a +`default-deny` NetworkPolicy with `policyTypes: [Ingress, Egress]` and **no `ingress` key at all** +— deny-all-ingress in NetworkPolicy semantics — and the only ingress allowance is +`-allow-gateway-ingress`: `fromEntities: [ingress]` to port 4180. Prometheus matched no +clause, so its packets were dropped. + +Two deliberate choices, both recorded in the function's docstring: + +- **Clause 2 keys on `_needs_oauth2_sidecar`, not on `observability.metrics`.** It has to match + the condition the *scrape* keys on: `podmonitor.yaml` selects pods by the + `homelab.keithjackson.dev/auth-mode: forward-auth` pod label and nothing else, so a target + exists iff the sidecar exists. Keying on `observability.metrics` would let `metrics: false` + produce a discovered-but-blocked target — the same "two halves authored apart" failure this + function exists to prevent, moved one field over. +- **The policy is rendered here, not added to `platform/oauth2-proxy/`.** podmonitor.yaml's own + header records why that tier cannot host it: it would name app namespaces created by the + *sibling* `apps-wave-0/1` tiers, so it would apply on a running cluster and fail on a cold + bootstrap. Rendered per-app it ships in the same tier that creates the namespace. + +## What I verified + +Measured on prod 2026-10-04, read-only, via the pinned toolbox: + +- **The cause, not a guess.** All 6 sidecar targets in pool + `podMonitor/oauth2-proxy/oauth2-proxy-sidecars/0`: `health=down`, + `lastError: context deadline exceeded`, `lastScrapeDuration` exactly `10.0s` — the configured + `scrapeTimeout`. Hitting the timeout precisely means packets **dropped, not refused**. +- **The listener is fine.** jackett's `oauth2-proxy` container carries + `--metrics-address=0.0.0.0:44180` and a named port `metrics: 44180`. +- **The deny is universal.** All six namespaces (`jackett radarr sonarr qbittorrent pihole + crafty`): `default-deny` with `ingress` key absent, `nIngress=0`; the only ingress CNP allowance + is 4180 `fromEntities: [ingress]`. **Zero** namespaces had any `-allow-monitoring-scrape` object. +- **Frequency.** Single continuous episode, no flapping: `ALERTS_FOR_STATE` = 229343s (63.7h), + firing since 2026-10-02 05:09 UTC — 11 minutes after the PodMonitor was created, i.e. TargetDown's + `for: 10m`. A 15-day `query_range` shows **1 episode** per namespace, not many. Alertmanager: + `group_by: [alertname, namespace]`, `repeat_interval: 12h` on `email-warning` → 5 TargetDown + groups = 10 emails/day, plus 6 `OAuth2ProxyGateUnobserved` groups = 12/day, ~22/day total. + 0 silences. +- **The re-render is deterministic and the diff is exactly the intended one.** `apps/` is + byte-identical to `render.py --all catalog -o apps --wave-layout --no-image-automation` + (12/12 SAME before my edit; the 11-file "drift" against a default render is solely the + ImageRepository/ImagePolicy pair, which `apps/` is generated without). After the edit: + 5 files changed, +21 lines each, nothing else touched. +- `render.py --selftest`: **PASS**. +- **Three L1 goldens needed regenerating, and the diff is +63 lines / -0.** The first gate run + caught `container-workload-render` (R-PLAT-02) and `egress-class-render` (R-NET-05/06/07) as + FAIL with exactly the new CNP as the drift. Regenerated with the same invocations the checks + use (`render.py `, no flags). Evidence the regen is faithful rather than a rubber + stamp: `default-deny-baseline-render`'s golden came back **byte-identical** (its fixture is not + forward-auth), and the other three gained only the one CNP, port 44180, ns monitoring. +- Toolbox gate `./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`: **rc 3 — UNDETERMINED, + NOT rc 0, and I am not reporting it as green.** Every stage that ran is green + (spec-immutability, approval-policy, decisions-log, lint, l0, render-golden, l1); `l1` went + FAIL -> pass once the goldens were regenerated. The gate's own words: "Stages that did not run: + gitea-settings" — skipped for want of `GITEA_SETTINGS_TOKEN`, which is an environment gap on + every local run and unrelated to this change. L2/L3 conformance is a `[warn]`, not a not-run + stage: UNVERIFIED for all 8 touched requirements (R-GITOPS-02 R-IDP-02 R-IDP-06 R-NET-01 + R-NET-05 R-NET-06 R-NET-07 R-PLAT-02), no cluster. +- `--emit-waves` was **not** run (it re-onboards `vm-demo` + `keycloak`); only the per-service + render path was used, so no wave kustomization changed. + +## This PR needs a human approval (sensitive path) — MEASURED, not predicted + +CI run #230 on PR #49, head `f9446be3`: **`MERGE GATE: BLOCKED`, and `approval-policy` is the ONLY +failing stage.** Read from the job log, not from the status tick: + + [skip] stage spec-immutability: SKIPPED (not run) <- sibling spec repo absent in CI; expected + [ ok ] stage decisions-log: pass + [ ok ] stage lint: pass + [ ok ] stage l0: pass + [ ok ] stage render-golden: pass + [ ok ] stage l1: pass + [fail] stage approval-policy: FAIL + [ ok ] stage gitea-settings: pass <- could NOT run locally; green in CI + [skip] stage conformance: SKIPPED (not run) + + sensitive paths touched: + conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml + conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml + conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml + +Exactly the three regenerated goldens, nothing else. `l1` and `render-golden` both pass in CI, +which is the independent confirmation that the regenerated goldens are right rather than merely +self-consistent with my local run. + +One local-reproduction note worth recording, because it wasted a step: running +`PR_NUMBER=49 SETTINGS_API_TOKEN=... ci/approval-policy.sh` with the credential from +`git credential fill` reports `could not determine PR author — BLOCKING`. That is an **artifact**, +not a finding — `~/.git-credentials` holds a *password*, and `Authorization: token ` +returns 401 `user does not exist [uid: 0]`. The stage needs a real API token. I nearly reported +that 401 as the CI failure; the job log is what settled it. + +Regenerating the L1 goldens means this branch touches `conformance/**`, which is in +`sensitive_paths` in `infra/gitea/approval-policy.json` ("the tests and the harness; changing them +changes what green MEANS"). So `ci/approval-policy.sh` will require an approving review from a +non-author, and the local gate **cannot** see that: it reports `approval-policy: green` because +offline it evaluates N/A. + +The golden update is not optional — a renderer change that adds an object necessarily drifts the +goldens that pin the renderer's output — but it does mean the gate correctly refuses to +self-approve this one. **An agent must not clear this with `/approve-override`**, even though the +`gitea-admin` account holds `owner`: that is precisely the self-approval the stage exists to +prevent. The operator posts the override as a PR *review* of type "Comment", or approves normally. + +## What I did NOT verify + +- **That the scrape now succeeds.** This is a git-only change and nothing is applied until Flux + reconciles `main`. The 5 CNPs have never existed on the cluster, so `up == 1` for those targets + is **UNVERIFIED** — it is a prediction from the measured drop signature, not an observation. + After merge, confirm `up{container="oauth2-proxy"}` goes to 1 for all 6 targets and that the 11 + alerts resolve. `send_resolved: true` is set, so expect ~11 resolved emails. +- **That the CNP `endpointSelector` matches pihole's pods.** It selects + `app.kubernetes.io/name: pihole`, identical to the shape of `pihole-allow-gateway-ingress` + which demonstrably works for 4180 — so this is inference from a working sibling, not a + measurement of the new object. +- **`crafty` is still unobserved and will keep firing `OAuth2ProxyGateUnobserved`.** It is + hand-managed under `platform/crafty`, outside the renderer, and has **no `--metrics-address` + at all** — so there is no port to open and this change cannot help it. Separate work. +- **Whether 44180 is reachable in the Cilium data path once allowed.** Only the policy was + reasoned about; no L2 probe was run (CI has `vars.CLUSTER_AVAILABLE` unset, so L2/L3 did not + run here either — the gate proves the L0/L1 offline subset only). +- **Why `OAuth2ProxyGateUnobserved` started at 2026-10-04 06:44** rather than 10-02 with + TargetDown. Alertmanager did not restart (TargetDown's `startsAt` is still 10-02) and the rule's + `for` is 1h, so neither explains it. Something changed on the monitoring side that morning which + I did not identify. It does not affect this fix. +- I did not re-run a port-forward to 44180 to prove the endpoint serves *today*; I confirmed the + flag and named port on the live pod spec. The 200/10208-byte measurement is podmonitor.yaml's, + from 2026-10-01.