Compare commits

..
Author SHA1 Message Date
jakman85andClaude Opus 5 b156ee6cfa fix(obs): open 44180 to ns monitoring so the forward-auth gate sidecars are actually scrapable
gate / gate (pull_request) Successful in 1m51s
gate / conformance (pull_request) Has been skipped
platform/oauth2-proxy/podmonitor.yaml landed 2026-10-02 and discovered six targets that
have never once scraped. Measured on prod 2026-10-04: all six read health=down with
lastError "context deadline exceeded" and lastScrapeDuration exactly 10.0s — the
configured scrapeTimeout, i.e. packets DROPPED rather than refused.

Cause: every app namespace ships a default-deny NetworkPolicy with
policyTypes: [Ingress, Egress] and NO ingress key at all (deny-all-ingress), and the only
ingress allowance is <app>-allow-gateway-ingress: fromEntities: [ingress] -> 4180.
Prometheus matched no clause. Verified identical in all six of jackett, radarr, sonarr,
qbittorrent, pihole, crafty; zero namespaces had any -allow-monitoring-scrape object.

render.py already had the right idiom — r_monitoring_ingress_policy — but it returned None
unless the service declared observability.metricsEndpoint, and NO ServiceDefinition in this
catalog declares one. It was wired in at the call site and had never emitted a single
object on any branch: correct, and entirely latent.

It now also fires for the forward-auth gate sidecar's own metrics port, keyed on
_needs_oauth2_sidecar — deliberately the same condition the scrape keys on, since
podmonitor.yaml selects pods purely by the auth-mode label, so a target exists iff the
sidecar exists. Keying on observability.metrics instead would let metrics: false produce a
discovered-but-blocked target: the same "two halves authored apart" failure this function
exists to prevent.

Rendered here rather than in platform/oauth2-proxy because that tier would have to name app
namespaces created by the sibling apps-wave-0/1 tiers — it would apply on a running cluster
and fail on a cold bootstrap. podmonitor.yaml's header records that objection; rendering
per-app dissolves it.

Five CNPs result, one per forward-auth app, each opening only 44180 to only ns monitoring.
apps/ re-rendered with the invocation it was generated by (--no-image-automation, verified
byte-identical beforehand); --emit-waves deliberately not run, so no wave kustomization moved.

Three L1 goldens are regenerated with it (+63 lines, -0): a renderer change that adds an
object necessarily drifts the goldens that pin the renderer's output. The first gate run
caught container-workload-render and egress-class-render as FAIL with exactly this CNP as
the drift. default-deny-baseline-render's golden regenerated byte-identical, which is the
evidence that the regen is faithful and not a rubber stamp.

That puts conformance/** in the diff, which is a sensitive path: this PR needs a non-author
approving review. The local gate cannot see that — offline it reports approval-policy green
(N/A).

Local toolbox gate: rc 3 UNDETERMINED, not 0. Every stage that ran is green and l1 went
FAIL -> pass with the goldens; the only not-run stage is gitea-settings (no
GITEA_SETTINGS_TOKEN). L2/L3 conformance is UNVERIFIED for all 8 touched requirements.

Not verified: that the scrape now succeeds. Nothing is applied until Flux reconciles main.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-04 16:05:44 -05:00
jakman85 422834d4e2 Merge pull request 'fix(plex): advertise the plex.direct name, not the bare VIP (server vanished from the plex.tv account)' (#47) from fix/plex-advertise-plexdirect into main
gate / gate (push) Successful in 1m46s
gate / conformance (push) Has been skipped
Reviewed-on: #47
Reviewed-by: Keith Jackson <[email protected]>
2026-10-04 07:59:14 +00:00
10 changed files with 365 additions and 5 deletions

No files matched your search

+21
View File
@@ -244,6 +244,27 @@ spec:
- port: '4180'
protocol: TCP
---
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): jackett-allow-monitoring-scrape
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: jackett-allow-monitoring-scrape
namespace: jackett
labels:
app.kubernetes.io/name: jackett
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: jackett
ingress:
- fromEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: monitoring
toPorts:
- ports:
- port: '44180'
protocol: TCP
---
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): jackett-allow-internet-egress
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
+21
View File
@@ -232,6 +232,27 @@ spec:
- port: '4180'
protocol: TCP
---
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): pihole-allow-monitoring-scrape
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: pihole-allow-monitoring-scrape
namespace: pihole
labels:
app.kubernetes.io/name: pihole
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: pihole
ingress:
- fromEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: monitoring
toPorts:
- ports:
- port: '44180'
protocol: TCP
---
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): pihole-allow-internet-egress
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
+21
View File
@@ -319,6 +319,27 @@ spec:
- port: '4180'
protocol: TCP
---
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): qbittorrent-allow-monitoring-scrape
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: qbittorrent-allow-monitoring-scrape
namespace: qbittorrent
labels:
app.kubernetes.io/name: qbittorrent
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: qbittorrent
ingress:
- fromEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: monitoring
toPorts:
- ports:
- port: '44180'
protocol: TCP
---
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): qbittorrent-allow-internet-egress
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
+21
View File
@@ -347,6 +347,27 @@ spec:
- port: '4180'
protocol: TCP
---
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): radarr-allow-monitoring-scrape
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: radarr-allow-monitoring-scrape
namespace: radarr
labels:
app.kubernetes.io/name: radarr
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: radarr
ingress:
- fromEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: monitoring
toPorts:
- ports:
- port: '44180'
protocol: TCP
---
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): radarr-allow-internet-egress
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
+21
View File
@@ -345,6 +345,27 @@ spec:
- port: '4180'
protocol: TCP
---
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: sonarr-allow-monitoring-scrape
namespace: sonarr
labels:
app.kubernetes.io/name: sonarr
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: sonarr
ingress:
- fromEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: monitoring
toPorts:
- ports:
- port: '44180'
protocol: TCP
---
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
+47 -5
View File
@@ -1541,12 +1541,54 @@ def r_monitoring_ingress_policy(s: dict):
trap Longhorn's own NetworkPolicies set (platform/longhorn/networkpolicy-metrics.yaml exists
for exactly this reason), and it is a THIRD independent way for a metrics SLI to be inert.
Rendered only alongside a declared metricsEndpoint, and it opens ONLY that port to ONLY the
monitoring namespace: an allow for a port nothing serves would be a standing hole for no
Rendered only alongside a port something actually serves, and it opens ONLY that port to ONLY
the monitoring namespace: an allow for a port nothing serves would be a standing hole for no
benefit.
TWO INDEPENDENT SOURCES OF A SERVED METRICS PORT, and this used to cover only the first:
1. `observability.metricsEndpoint` — the APP's own exporter. NOTE, measured 2026-10-04: no
ServiceDefinition in this catalog declares one, so before the sidecar clause below this
function had NEVER emitted a single object on any branch. It is correct but, so far,
entirely latent — do not read its presence as evidence that scrape ingress is solved.
2. THE forward-auth GATE SIDECAR's own `/metrics` on 44180 (added 2026-10-04). Every
`_needs_oauth2_sidecar` app gets `--metrics-address=0.0.0.0:{OAUTH2_PROXY_METRICS_PORT}`
and a named `metrics` container port from `_oauth2_sidecar_container`, so the port is
served whenever the sidecar exists — unconditionally, which is what makes it safe to
open without a per-app declaration.
WHY CLAUSE 2 KEYS ON `_needs_oauth2_sidecar` AND NOT ON `observability.metrics`: it must match
the condition the SCRAPE keys on, exactly. platform/oauth2-proxy/podmonitor.yaml selects pods
by the `homelab.keithjackson.dev/auth-mode: forward-auth` POD LABEL and nothing else, so a
target exists iff the sidecar exists. Keying the allowance on `observability.metrics` instead
would let `metrics: false` produce a discovered-but-blocked target — the same "two halves
authored apart" failure this function exists to prevent, just moved one field over.
THIS CLAUSE IS A BUG FIX, and the bug was live for 63.7h. Measured on prod 2026-10-04: all 6
sidecar targets across jackett/radarr/sonarr/qbittorrent/pihole read `health=down`,
`lastError: context deadline exceeded`, `lastScrapeDuration` exactly 10.0s — the scrapeTimeout,
i.e. packets DROPPED, not refused. Each app namespace ships a `default-deny` NetworkPolicy
with `policyTypes: [Ingress, Egress]` and NO `ingress` key at all (deny-all-ingress), and the
only ingress allowance is `<app>-allow-gateway-ingress`: `fromEntities: [ingress]` -> 4180.
Prometheus matched no clause. 5 x `TargetDown` + 6 x `OAuth2ProxyGateUnobserved` had been
re-notifying at ~22 emails/day since the PodMonitor landed on 2026-10-02.
WHY THE POLICY BELONGS HERE AND NOT IN platform/oauth2-proxy: podmonitor.yaml's header records
the objection — a policy in that tier would name app namespaces created by the SIBLING
apps-wave-0/1 tiers (measured Flux graph: `oauth2-proxy` and `apps-wave-0` both
`dependsOn=[platform-config]`, no ordering between them), so it would apply on a running
cluster and fail on a cold bootstrap. Rendered here it ships in the same tier that creates the
namespace, which dissolves that ordering problem entirely.
"""
ep = s["observability"].get("metricsEndpoint")
if not ep or not s["observability"]["metrics"]:
obs = s["observability"]
ep = obs.get("metricsEndpoint")
ports: list[str] = []
if ep and obs["metrics"]:
ports.append(str(ep["port"]))
if _needs_oauth2_sidecar(s):
ports.append(str(OAUTH2_PROXY_METRICS_PORT))
if not ports:
return None
return {
"apiVersion": "cilium.io/v2",
@@ -1564,7 +1606,7 @@ def r_monitoring_ingress_policy(s: dict):
{"matchLabels": {"k8s:io.kubernetes.pod.namespace": MONITORING_NS}}
],
"toPorts": [
{"ports": [{"port": str(ep["port"]), "protocol": "TCP"}]}
{"ports": [{"port": p, "protocol": "TCP"} for p in ports]}
],
}
],
@@ -171,6 +171,27 @@ spec:
- port: '4180'
protocol: TCP
---
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): paas-demo-allow-monitoring-scrape
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: paas-demo-allow-monitoring-scrape
namespace: paas-demo
labels:
app.kubernetes.io/name: paas-demo
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: paas-demo
ingress:
- fromEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: monitoring
toPorts:
- ports:
- port: '44180'
protocol: TCP
---
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): paas-demo-allow-internet-egress
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
@@ -198,6 +198,27 @@ spec:
- port: '4180'
protocol: TCP
---
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: sonarr-allow-monitoring-scrape
namespace: sonarr
labels:
app.kubernetes.io/name: sonarr
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: sonarr
ingress:
- fromEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: monitoring
toPorts:
- ports:
- port: '44180'
protocol: TCP
---
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
@@ -198,6 +198,27 @@ spec:
- port: '4180'
protocol: TCP
---
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: sonarr-allow-monitoring-scrape
namespace: sonarr
labels:
app.kubernetes.io/name: sonarr
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: sonarr
ingress:
- fromEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: monitoring
toPorts:
- ports:
- port: '44180'
protocol: TCP
---
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
@@ -0,0 +1,150 @@
---
date: 2026-10-04
branch: fix/oauth2-sidecar-metrics-scrape-netpol
title: The forward-auth gate sidecars' /metrics was NetworkPolicy-blocked — the renderer now opens 44180 to ns monitoring
---
## What changed
`catalog/render.py`'s `r_monitoring_ingress_policy` now emits its CiliumNetworkPolicy for a
**second** reason: the forward-auth gate sidecar's own `/metrics` port (44180), keyed on
`_needs_oauth2_sidecar(s)`. Previously it fired only from `observability.metricsEndpoint`.
That matters more than it sounds, because **no ServiceDefinition in this catalog declares a
`metricsEndpoint`** (measured: zero grep hits across `catalog/*.yaml`). So the function was wired
in at the call site and had **never emitted a single object on any branch** — correct, and
entirely latent. Its presence in the tree read as "scrape ingress is handled". It was not.
Re-rendered `apps/` from that change: five new CNPs, one per forward-auth app
(`jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`), each opening **only** 44180 to **only**
`fromEndpoints: [k8s:io.kubernetes.pod.namespace: monitoring]`.
The bug being fixed: `platform/oauth2-proxy/podmonitor.yaml` landed 2026-10-02T04:58:44Z and
discovered 6 targets that have **never once scraped successfully**. Each app namespace ships a
`default-deny` NetworkPolicy with `policyTypes: [Ingress, Egress]` and **no `ingress` key at all**
— deny-all-ingress in NetworkPolicy semantics — and the only ingress allowance is
`<app>-allow-gateway-ingress`: `fromEntities: [ingress]` to port 4180. Prometheus matched no
clause, so its packets were dropped.
Two deliberate choices, both recorded in the function's docstring:
- **Clause 2 keys on `_needs_oauth2_sidecar`, not on `observability.metrics`.** It has to match
the condition the *scrape* keys on: `podmonitor.yaml` selects pods by the
`homelab.keithjackson.dev/auth-mode: forward-auth` pod label and nothing else, so a target
exists iff the sidecar exists. Keying on `observability.metrics` would let `metrics: false`
produce a discovered-but-blocked target — the same "two halves authored apart" failure this
function exists to prevent, moved one field over.
- **The policy is rendered here, not added to `platform/oauth2-proxy/`.** podmonitor.yaml's own
header records why that tier cannot host it: it would name app namespaces created by the
*sibling* `apps-wave-0/1` tiers, so it would apply on a running cluster and fail on a cold
bootstrap. Rendered per-app it ships in the same tier that creates the namespace.
## What I verified
Measured on prod 2026-10-04, read-only, via the pinned toolbox:
- **The cause, not a guess.** All 6 sidecar targets in pool
`podMonitor/oauth2-proxy/oauth2-proxy-sidecars/0`: `health=down`,
`lastError: context deadline exceeded`, `lastScrapeDuration` exactly `10.0s` — the configured
`scrapeTimeout`. Hitting the timeout precisely means packets **dropped, not refused**.
- **The listener is fine.** jackett's `oauth2-proxy` container carries
`--metrics-address=0.0.0.0:44180` and a named port `metrics: 44180`.
- **The deny is universal.** All six namespaces (`jackett radarr sonarr qbittorrent pihole
crafty`): `default-deny` with `ingress` key absent, `nIngress=0`; the only ingress CNP allowance
is 4180 `fromEntities: [ingress]`. **Zero** namespaces had any `-allow-monitoring-scrape` object.
- **Frequency.** Single continuous episode, no flapping: `ALERTS_FOR_STATE` = 229343s (63.7h),
firing since 2026-10-02 05:09 UTC — 11 minutes after the PodMonitor was created, i.e. TargetDown's
`for: 10m`. A 15-day `query_range` shows **1 episode** per namespace, not many. Alertmanager:
`group_by: [alertname, namespace]`, `repeat_interval: 12h` on `email-warning` → 5 TargetDown
groups = 10 emails/day, plus 6 `OAuth2ProxyGateUnobserved` groups = 12/day, ~22/day total.
0 silences.
- **The re-render is deterministic and the diff is exactly the intended one.** `apps/` is
byte-identical to `render.py --all catalog -o apps --wave-layout --no-image-automation`
(12/12 SAME before my edit; the 11-file "drift" against a default render is solely the
ImageRepository/ImagePolicy pair, which `apps/` is generated without). After the edit:
5 files changed, +21 lines each, nothing else touched.
- `render.py --selftest`: **PASS**.
- **Three L1 goldens needed regenerating, and the diff is +63 lines / -0.** The first gate run
caught `container-workload-render` (R-PLAT-02) and `egress-class-render` (R-NET-05/06/07) as
FAIL with exactly the new CNP as the drift. Regenerated with the same invocations the checks
use (`render.py <fixture>`, no flags). Evidence the regen is faithful rather than a rubber
stamp: `default-deny-baseline-render`'s golden came back **byte-identical** (its fixture is not
forward-auth), and the other three gained only the one CNP, port 44180, ns monitoring.
- Toolbox gate `./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`: **rc 3 — UNDETERMINED,
NOT rc 0, and I am not reporting it as green.** Every stage that ran is green
(spec-immutability, approval-policy, decisions-log, lint, l0, render-golden, l1); `l1` went
FAIL -> pass once the goldens were regenerated. The gate's own words: "Stages that did not run:
gitea-settings" — skipped for want of `GITEA_SETTINGS_TOKEN`, which is an environment gap on
every local run and unrelated to this change. L2/L3 conformance is a `[warn]`, not a not-run
stage: UNVERIFIED for all 8 touched requirements (R-GITOPS-02 R-IDP-02 R-IDP-06 R-NET-01
R-NET-05 R-NET-06 R-NET-07 R-PLAT-02), no cluster.
- `--emit-waves` was **not** run (it re-onboards `vm-demo` + `keycloak`); only the per-service
render path was used, so no wave kustomization changed.
## This PR needs a human approval (sensitive path) — MEASURED, not predicted
CI run #230 on PR #49, head `f9446be3`: **`MERGE GATE: BLOCKED`, and `approval-policy` is the ONLY
failing stage.** Read from the job log, not from the status tick:
[skip] stage spec-immutability: SKIPPED (not run) <- sibling spec repo absent in CI; expected
[ ok ] stage decisions-log: pass
[ ok ] stage lint: pass
[ ok ] stage l0: pass
[ ok ] stage render-golden: pass
[ ok ] stage l1: pass
[fail] stage approval-policy: FAIL
[ ok ] stage gitea-settings: pass <- could NOT run locally; green in CI
[skip] stage conformance: SKIPPED (not run)
sensitive paths touched:
conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml
conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml
conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml
Exactly the three regenerated goldens, nothing else. `l1` and `render-golden` both pass in CI,
which is the independent confirmation that the regenerated goldens are right rather than merely
self-consistent with my local run.
One local-reproduction note worth recording, because it wasted a step: running
`PR_NUMBER=49 SETTINGS_API_TOKEN=... ci/approval-policy.sh` with the credential from
`git credential fill` reports `could not determine PR author — BLOCKING`. That is an **artifact**,
not a finding — `~/.git-credentials` holds a *password*, and `Authorization: token <password>`
returns 401 `user does not exist [uid: 0]`. The stage needs a real API token. I nearly reported
that 401 as the CI failure; the job log is what settled it.
Regenerating the L1 goldens means this branch touches `conformance/**`, which is in
`sensitive_paths` in `infra/gitea/approval-policy.json` ("the tests and the harness; changing them
changes what green MEANS"). So `ci/approval-policy.sh` will require an approving review from a
non-author, and the local gate **cannot** see that: it reports `approval-policy: green` because
offline it evaluates N/A.
The golden update is not optional — a renderer change that adds an object necessarily drifts the
goldens that pin the renderer's output — but it does mean the gate correctly refuses to
self-approve this one. **An agent must not clear this with `/approve-override`**, even though the
`gitea-admin` account holds `owner`: that is precisely the self-approval the stage exists to
prevent. The operator posts the override as a PR *review* of type "Comment", or approves normally.
## What I did NOT verify
- **That the scrape now succeeds.** This is a git-only change and nothing is applied until Flux
reconciles `main`. The 5 CNPs have never existed on the cluster, so `up == 1` for those targets
is **UNVERIFIED** — it is a prediction from the measured drop signature, not an observation.
After merge, confirm `up{container="oauth2-proxy"}` goes to 1 for all 6 targets and that the 11
alerts resolve. `send_resolved: true` is set, so expect ~11 resolved emails.
- **That the CNP `endpointSelector` matches pihole's pods.** It selects
`app.kubernetes.io/name: pihole`, identical to the shape of `pihole-allow-gateway-ingress`
which demonstrably works for 4180 — so this is inference from a working sibling, not a
measurement of the new object.
- **`crafty` is still unobserved and will keep firing `OAuth2ProxyGateUnobserved`.** It is
hand-managed under `platform/crafty`, outside the renderer, and has **no `--metrics-address`
at all** — so there is no port to open and this change cannot help it. Separate work.
- **Whether 44180 is reachable in the Cilium data path once allowed.** Only the policy was
reasoned about; no L2 probe was run (CI has `vars.CLUSTER_AVAILABLE` unset, so L2/L3 did not
run here either — the gate proves the L0/L1 offline subset only).
- **Why `OAuth2ProxyGateUnobserved` started at 2026-10-04 06:44** rather than 10-02 with
TargetDown. Alertmanager did not restart (TargetDown's `startsAt` is still 10-02) and the rule's
`for` is 1h, so neither explains it. Something changed on the monitoring side that morning which
I did not identify. It does not affect this fix.
- I did not re-run a port-forward to 44180 to prove the endpoint serves *today*; I confirmed the
flag and named port on the live pod spec. The 200/10208-byte measurement is podmonitor.yaml's,
from 2026-10-01.