fix(obs): open 44180 to ns monitoring so the forward-auth gate sidecars are actually scrapable
gate / gate (pull_request) Successful in 1m51s
gate / conformance (pull_request) Has been skipped

platform/oauth2-proxy/podmonitor.yaml landed 2026-10-02 and discovered six targets that
have never once scraped. Measured on prod 2026-10-04: all six read health=down with
lastError "context deadline exceeded" and lastScrapeDuration exactly 10.0s — the
configured scrapeTimeout, i.e. packets DROPPED rather than refused.

Cause: every app namespace ships a default-deny NetworkPolicy with
policyTypes: [Ingress, Egress] and NO ingress key at all (deny-all-ingress), and the only
ingress allowance is <app>-allow-gateway-ingress: fromEntities: [ingress] -> 4180.
Prometheus matched no clause. Verified identical in all six of jackett, radarr, sonarr,
qbittorrent, pihole, crafty; zero namespaces had any -allow-monitoring-scrape object.

render.py already had the right idiom — r_monitoring_ingress_policy — but it returned None
unless the service declared observability.metricsEndpoint, and NO ServiceDefinition in this
catalog declares one. It was wired in at the call site and had never emitted a single
object on any branch: correct, and entirely latent.

It now also fires for the forward-auth gate sidecar's own metrics port, keyed on
_needs_oauth2_sidecar — deliberately the same condition the scrape keys on, since
podmonitor.yaml selects pods purely by the auth-mode label, so a target exists iff the
sidecar exists. Keying on observability.metrics instead would let metrics: false produce a
discovered-but-blocked target: the same "two halves authored apart" failure this function
exists to prevent.

Rendered here rather than in platform/oauth2-proxy because that tier would have to name app
namespaces created by the sibling apps-wave-0/1 tiers — it would apply on a running cluster
and fail on a cold bootstrap. podmonitor.yaml's header records that objection; rendering
per-app dissolves it.

Five CNPs result, one per forward-auth app, each opening only 44180 to only ns monitoring.
apps/ re-rendered with the invocation it was generated by (--no-image-automation, verified
byte-identical beforehand); --emit-waves deliberately not run, so no wave kustomization moved.

Three L1 goldens are regenerated with it (+63 lines, -0): a renderer change that adds an
object necessarily drifts the goldens that pin the renderer's output. The first gate run
caught container-workload-render and egress-class-render as FAIL with exactly this CNP as
the drift. default-deny-baseline-render's golden regenerated byte-identical, which is the
evidence that the regen is faithful and not a rubber stamp.

That puts conformance/** in the diff, which is a sensitive path: this PR needs a non-author
approving review. The local gate cannot see that — offline it reports approval-policy green
(N/A).

Local toolbox gate: rc 3 UNDETERMINED, not 0. Every stage that ran is green and l1 went
FAIL -> pass with the goldens; the only not-run stage is gitea-settings (no
GITEA_SETTINGS_TOKEN). L2/L3 conformance is UNVERIFIED for all 8 touched requirements.

Not verified: that the scrape now succeeds. Nothing is applied until Flux reconciles main.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-10-04 16:05:44 -05:00
1 parent 422834d4e2
commit b156ee6cfa
10 files changed
+365 -5

No files matched your search

+21
View File
@@ -244,6 +244,27 @@ spec:
- port: '4180'
protocol: TCP
---
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): jackett-allow-monitoring-scrape
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: jackett-allow-monitoring-scrape
namespace: jackett
labels:
app.kubernetes.io/name: jackett
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: jackett
ingress:
- fromEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: monitoring
toPorts:
- ports:
- port: '44180'
protocol: TCP
---
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): jackett-allow-internet-egress
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
+21
View File
@@ -232,6 +232,27 @@ spec:
- port: '4180'
protocol: TCP
---
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): pihole-allow-monitoring-scrape
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: pihole-allow-monitoring-scrape
namespace: pihole
labels:
app.kubernetes.io/name: pihole
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: pihole
ingress:
- fromEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: monitoring
toPorts:
- ports:
- port: '44180'
protocol: TCP
---
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): pihole-allow-internet-egress
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
+21
View File
@@ -319,6 +319,27 @@ spec:
- port: '4180'
protocol: TCP
---
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): qbittorrent-allow-monitoring-scrape
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: qbittorrent-allow-monitoring-scrape
namespace: qbittorrent
labels:
app.kubernetes.io/name: qbittorrent
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: qbittorrent
ingress:
- fromEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: monitoring
toPorts:
- ports:
- port: '44180'
protocol: TCP
---
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): qbittorrent-allow-internet-egress
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
+21
View File
@@ -347,6 +347,27 @@ spec:
- port: '4180'
protocol: TCP
---
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): radarr-allow-monitoring-scrape
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: radarr-allow-monitoring-scrape
namespace: radarr
labels:
app.kubernetes.io/name: radarr
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: radarr
ingress:
- fromEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: monitoring
toPorts:
- ports:
- port: '44180'
protocol: TCP
---
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): radarr-allow-internet-egress
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
+21
View File
@@ -345,6 +345,27 @@ spec:
- port: '4180'
protocol: TCP
---
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: sonarr-allow-monitoring-scrape
namespace: sonarr
labels:
app.kubernetes.io/name: sonarr
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: sonarr
ingress:
- fromEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: monitoring
toPorts:
- ports:
- port: '44180'
protocol: TCP
---
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy