--- # Allow Prometheus to reach longhorn-manager's metrics port (R-OBS-04, R-STATE-02). # # WITHOUT THIS FILE, `metrics.serviceMonitor.enabled: true` IN helmrelease.yaml IS NOT ENOUGH and the # four Longhorn alert rules in platform/monitoring/alerts.yaml STAY INERT. This was measured, not # predicted — prod, 2026-09-13, with the ServiceMonitor in place: # # count by (job) (up) => {job="longhorn-backend"} => 3 (targets DISCOVERED) # up{job="longhorn-backend"} => 0, 0, 0 (every scrape FAILING) # count(longhorn_volume_robustness) => (empty) (still nothing in the TSDB) # /api/v1/targets lastError => 'Get "http://10.0.0.169:9500/metrics": context deadline # exceeded' (10.0007s — a DROP, not a refusal) # # The endpoint itself is healthy and fast; only the path from ns monitoring is blocked. Proven by # curling it from inside the manager pod: # http://:9500/metrics -> 200 in 0.031s (the manager binds the POD IP, not localhost — # http://localhost:9500/metrics -> 000 which is why a localhost probe misleads here) # # WHY IT IS BLOCKED: longhorn 1.12.1 ships SIX NetworkPolicies of its own, and the one selecting # `app: longhorn-manager` is Ingress-only with a from-list of nothing but other Longhorn components # (longhorn-manager, longhorn-ui, longhorn-csi-plugin, the recurring-job/job-task pods, # longhorn-driver-deployer) — all bare podSelectors, i.e. same-namespace. Prometheus lives in ns # monitoring, so it matches no clause and its packets are dropped. # # THE CHART OFFERS NO KNOB FOR THIS, and the obvious-looking one is a trap. `networkPolicies.enabled` # (default false) does NOT gate these templates — verified by rendering 1.12.1: `helm template` emits # the same 6 NetworkPolicies with the flag on AND off, because templates/network-policies/* are # guarded on `networkPolicies.restrictInternalTraffic` (default TRUE) instead. So the policies are # effectively unconditional, and nothing in the chart adds a Prometheus allowance when # metrics.serviceMonitor.enabled is set. # # WHY NOT `networkPolicies.restrictInternalTraffic: false`: that is the one-line fix and it is the # wrong one. It deletes Longhorn's entire internal-traffic restriction across all six components to # open one port to one namespace — a straight R-SEC-02 (least network reach) regression, trading a # security control for an observability one when the two are not in conflict. # # THIS IS ADDITIVE AND NARROW. Kubernetes NetworkPolicies are OR'd (Cilium enforces the UNION — see # platform/headlamp/networkpolicy.yaml's header), so this grants exactly one new path on top of # Longhorn's floor: ns monitoring -> app=longhorn-manager, TCP 9500 only. Longhorn's own policy is # left untouched, every other component stays restricted, and nothing else in longhorn-system becomes # reachable. # # TIER: a core networking.k8s.io kind with no CRD dependency, so it belongs to the `platform` # Kustomization alongside the HelmRelease (referenced from platform/longhorn/kustomization.yaml) — # NOT platform-config, which exists for CRs of platform-installed CRDs. apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: longhorn-manager-allow-prometheus-scrape namespace: longhorn-system labels: app.kubernetes.io/name: longhorn # Marks this as an observability path, not a data path — so a future reviewer trimming Longhorn # network reach can see at a glance what breaks (the storage alerts) if they delete it. homelab.keithjackson.dev/purpose: metrics-scrape spec: podSelector: matchLabels: app: longhorn-manager policyTypes: - Ingress ingress: - from: # The whole monitoring namespace rather than the Prometheus pod labels: the operator owns # those labels and rewrites them on every kube-prometheus-stack bump (they carry the chart's # instance name and a controller-revision hash). A pod-label selector here would silently # stop matching on an upgrade and put the storage alerts back to inert — the exact regression # class this whole change exists to close. The namespace holds only the monitoring stack. - namespaceSelector: matchLabels: kubernetes.io/metadata.name: monitoring ports: - protocol: TCP # longhorn-manager's container port `manager` (Service longhorn-backend:9500), which serves # /metrics. NOT 9502/9503 (admission webhook / recovery backend) — those stay closed. port: 9500