jakman85andClaude Opus 5
bce101d03f
PR #28 gave the five fleet oauth2-proxy sidecars --metrics-address=0.0.0.0:44180 and a named `metrics` port. This wires a scraper to them and, in the process, found a real bug in the alert this branch added. PodMonitor, not ServiceMonitor. Measured: no Service exposes 44180 on any of the five (jackett/radarr/sonarr/qbittorrent/pihole all publish only their app port + authproxy 4180), so a ServiceMonitor would select nothing. A PodMonitor targets pod ports directly and needs no change to catalog/render.py or the apps/wave-* renders and goldens. Selector is the pod label homelab.keithjackson.dev/auth-mode=forward-auth — present on exactly those five gate pods, absent from crafty's, which still has no metrics listener. The property, not an app list. THE BUG: `up{container="oauth2-proxy"}` -> `up{container="oauth2-proxy"} == 1`. A target that is discovered but FAILING still publishes `up`, with value 0. The series exists, so count by (namespace) returned 1, `unless` removed the namespace, and OAuth2ProxyGateUnobserved went QUIET while nothing was being scraped at all — this branch's own disease reproduced inside the alert written to cure it. Not hypothetical: the PodMonitor lands five targets whose scrape is dropped, so the bare form would have silently cleared on this very commit. Proven by mutation: reverting the `== 1` fails the two new unit-test scenarios (got: nil) and leaves every other assertion green. The scrape does NOT work yet, and that is documented rather than papered over. From a pod in ns monitoring to the jackett pod, :44180 AND :4180 both time out; the shared proxy's :44180 answers. ns jackett's default-deny is podSelector:{} with policyTypes:[Ingress,Egress] and NO ingress key (deny-all), and the one allowance is fromEntities:[ingress] to 4180 — the Gateway's Envoy only. The endpoint itself is healthy: port-forwarded, /metrics returns http=200, 10208 bytes, all three oauth2_proxy_* families including requests_total{code="302"}. Same bug and fingerprint as platform/longhorn/networkpolicy-metrics.yaml records for Longhorn; the fix is one narrow rule per app, and it cannot ship from this tier because oauth2-proxy and apps-wave-0 are Flux SIBLINGS (both dependsOn platform-config), so naming ns jackett here would apply today and fail on a cold rebuild. Needs an owner in the renderer. Touches conformance/** : one KNOWN_EXTERNAL_MONITORS entry, which is the mechanism the failing check explicitly instructs ("DECLARE it ... with who creates it") and which flux-podmonitor.yaml already uses. It records that the port name was measured on prod instead of resolved from a Service, and that the targets are currently down. That makes this PR require a non-author review on the sensitive-path rule. Also: ratio alert still NOT shipped. oauth2_proxy_requests_total now exists on the endpoint but not in the TSDB, so writing it today would make it inert for exactly the reason it was refused on 2026-09-30. Runbook gains policy-vs-no-target triage and the radarr /signalr/messages/negotiate worked case — one orphaned browser tab, 206 identical 302s, nobody ever re-prompted. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
110 lines
6.9 KiB
YAML
110 lines
6.9 KiB
YAML
---
|
|
# PodMonitor for the PER-APP oauth2-proxy gate sidecars (R-OBS-04).
|
|
#
|
|
# ============================================================================================
|
|
# WHY A PodMonitor AND NOT A ServiceMonitor
|
|
# ============================================================================================
|
|
# PR #28 gave all five fleet sidecars `--metrics-address=0.0.0.0:44180` and a named `metrics`
|
|
# container port. It did NOT add a Service port, and measured on prod 2026-10-01 none of the five
|
|
# Services exposes 44180:
|
|
#
|
|
# jackett jackett: [('port-9117', 9117), ('authproxy', 4180)]
|
|
# radarr radarr: [('port-7878', 7878), ('authproxy', 4180)]
|
|
# sonarr sonarr: [('port-8989', 8989), ('authproxy', 4180)]
|
|
# qbittorrent qbittorrent: [('port-8080', 8080), ('authproxy', 4180)]
|
|
# pihole pihole: [('port-80', 80), ('authproxy', 4180)]
|
|
#
|
|
# A ServiceMonitor selects Service PORTS, so it would have nothing to select. A PodMonitor targets
|
|
# pod ports directly and bypasses Services entirely — so this needs no change to the Services, and
|
|
# therefore no change to `catalog/render.py` or the `apps/wave-*` renders and goldens.
|
|
#
|
|
# SELECTOR: the `homelab.keithjackson.dev/auth-mode: forward-auth` POD LABEL, not a namespace list.
|
|
# Measured 2026-10-01: that label is present on exactly the five fleet gate pods (jackett, radarr,
|
|
# sonarr, qbittorrent, pihole) and absent from crafty's, which is a separately-managed deployment
|
|
# (platform/crafty) that still has no `--metrics-address` at all. So the selector describes the
|
|
# PROPERTY being scraped rather than enumerating today's apps, and a sixth forward-auth app is
|
|
# picked up with no edit here. `namespaceSelector.any: true` because those five namespaces are
|
|
# created by the apps-wave-0/1 tiers, which this tier does not and must not depend on.
|
|
#
|
|
# ============================================================================================
|
|
# READ THIS BEFORE BELIEVING THE TARGETS ARE UP: THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY
|
|
# ============================================================================================
|
|
# This PodMonitor is NECESSARY AND NOT SUFFICIENT. The endpoint serves, and the path from ns
|
|
# monitoring is blocked. Both halves measured on prod 2026-10-01, read-only:
|
|
#
|
|
# THE ENDPOINT IS HEALTHY — port-forward straight to the jackett pod's 44180:
|
|
# GET /metrics -> http=200 bytes=10208
|
|
# # TYPE oauth2_proxy_requests_in_flight gauge
|
|
# # TYPE oauth2_proxy_requests_total counter
|
|
# # TYPE oauth2_proxy_response_duration_seconds histogram
|
|
# oauth2_proxy_requests_total{code="200"} 3
|
|
# oauth2_proxy_requests_total{code="302"} 2
|
|
#
|
|
# THE SCRAPE PATH IS DROPPED — from a pod in ns monitoring (grafana, which sits beside
|
|
# Prometheus and so occupies the same policy position) to the jackett pod IP:
|
|
# http://10.0.0.189:44180/metrics -> wget: download timed out
|
|
# http://10.0.0.189:4180/ping -> wget: download timed out (even the "allowed" port)
|
|
# http://<shared-proxy>:44180/metrics -> 200, oauth2_proxy_* present (control: ns oauth2-proxy
|
|
# has NO NetworkPolicy, which is the only reason the
|
|
# shared instance has ever been scrapable)
|
|
#
|
|
# WHY: ns jackett's `default-deny` NetworkPolicy is `podSelector: {}` with
|
|
# `policyTypes: [Ingress, Egress]` and an `egress` rule ONLY — no `ingress` key, which in
|
|
# NetworkPolicy semantics is deny-all-ingress. The only ingress allowance is the CNP
|
|
# `jackett-allow-gateway-ingress`: `fromEntities: [ingress]` to port 4180 — the Gateway's Envoy
|
|
# and nothing else. Prometheus matches no clause and its packets are dropped, not refused.
|
|
#
|
|
# SO THE TARGETS BELOW WILL READ `up == 0` UNTIL AN INGRESS ALLOWANCE LANDS. That is a known,
|
|
# measured, documented state — NOT a regression introduced here, and NOT something to silence.
|
|
# It is deliberately left visible: `OAuth2ProxyGateUnobserved` in auth-visibility-alerts.yaml
|
|
# requires `up == 1`, precisely so a discovered-but-failing target does not read as "observed"
|
|
# (see that file's header — this exact trap is why the `== 1` is there).
|
|
#
|
|
# THIS IS THE SAME BUG LONGHORN HAD, and the platform has a documented fix idiom for it:
|
|
# platform/longhorn/networkpolicy-metrics.yaml, whose header records the identical fingerprint
|
|
# measured on 2026-09-13 — ServiceMonitor in place, 3 targets DISCOVERED, `up` 0/0/0, TSDB empty,
|
|
# `lastError: context deadline exceeded` (a drop, not a refusal). The fix there is a narrow
|
|
# additive `networking.k8s.io/v1` NetworkPolicy: ns monitoring -> the pod label, that one port.
|
|
#
|
|
# WHY THAT POLICY IS NOT IN THIS DIRECTORY. It would have to be five objects, one per app
|
|
# namespace, and those namespaces are created by apps-wave-0/1. Measured Flux graph, 2026-10-01:
|
|
# oauth2-proxy dependsOn=['platform-config'] <- this tier
|
|
# apps-wave-0 dependsOn=['platform-config'] <- SIBLING, no ordering between them
|
|
# apps-wave-1 dependsOn=['apps-wave-0']
|
|
# They are siblings, so on a cold bootstrap there is no guarantee ns jackett exists when this tier
|
|
# applies, and a NetworkPolicy naming a missing namespace fails to apply. It would work on today's
|
|
# running cluster and break on a rebuild — the "works now, broken on restore" shape this repo
|
|
# keeps paying for. Longhorn's precedent is safe only because longhorn-system is created by the
|
|
# same tier that ships its policy. The allowance therefore belongs WITH THE APPS, rendered next to
|
|
# the rest of each app's network policy — i.e. in the catalog/renderer, which is in flight on
|
|
# another branch. Tracked in the handoff; it is one narrow rule per app, not a redesign.
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PodMonitor
|
|
metadata:
|
|
name: oauth2-proxy-sidecars
|
|
namespace: oauth2-proxy
|
|
labels:
|
|
release: kube-prometheus-stack # kube-prometheus-stack's podMonitorSelector
|
|
app.kubernetes.io/name: oauth2-proxy
|
|
homelab.keithjackson.dev/purpose: metrics-scrape
|
|
spec:
|
|
namespaceSelector:
|
|
# The gate sidecars live in the app namespaces, not this one. `any` rather than a fixed list so
|
|
# a new forward-auth app needs no edit here — the pod label below is what scopes this.
|
|
any: true
|
|
selector:
|
|
matchLabels:
|
|
# The PROPERTY, not the app list. Present on exactly the five fleet gate pods (measured
|
|
# 2026-10-01); absent from crafty, which has no metrics listener yet.
|
|
homelab.keithjackson.dev/auth-mode: forward-auth
|
|
podMetricsEndpoints:
|
|
# By PORT NAME (`metrics`, 44180), never by number: the name is what PR #28 declared on the
|
|
# container, and a numeric port here would silently stop matching if the port were renumbered.
|
|
- port: metrics
|
|
path: /metrics
|
|
interval: 30s
|
|
# 10s, matching the drop signature above: a DENIED path does not refuse, it hangs until the
|
|
# scrape times out. Leaving the default would make each failing target hold a scrape slot for
|
|
# the full interval.
|
|
scrapeTimeout: 10s
|