jakman85andClaude Opus 5
bce101d03f
PR #28 gave the five fleet oauth2-proxy sidecars --metrics-address=0.0.0.0:44180 and a named `metrics` port. This wires a scraper to them and, in the process, found a real bug in the alert this branch added. PodMonitor, not ServiceMonitor. Measured: no Service exposes 44180 on any of the five (jackett/radarr/sonarr/qbittorrent/pihole all publish only their app port + authproxy 4180), so a ServiceMonitor would select nothing. A PodMonitor targets pod ports directly and needs no change to catalog/render.py or the apps/wave-* renders and goldens. Selector is the pod label homelab.keithjackson.dev/auth-mode=forward-auth — present on exactly those five gate pods, absent from crafty's, which still has no metrics listener. The property, not an app list. THE BUG: `up{container="oauth2-proxy"}` -> `up{container="oauth2-proxy"} == 1`. A target that is discovered but FAILING still publishes `up`, with value 0. The series exists, so count by (namespace) returned 1, `unless` removed the namespace, and OAuth2ProxyGateUnobserved went QUIET while nothing was being scraped at all — this branch's own disease reproduced inside the alert written to cure it. Not hypothetical: the PodMonitor lands five targets whose scrape is dropped, so the bare form would have silently cleared on this very commit. Proven by mutation: reverting the `== 1` fails the two new unit-test scenarios (got: nil) and leaves every other assertion green. The scrape does NOT work yet, and that is documented rather than papered over. From a pod in ns monitoring to the jackett pod, :44180 AND :4180 both time out; the shared proxy's :44180 answers. ns jackett's default-deny is podSelector:{} with policyTypes:[Ingress,Egress] and NO ingress key (deny-all), and the one allowance is fromEntities:[ingress] to 4180 — the Gateway's Envoy only. The endpoint itself is healthy: port-forwarded, /metrics returns http=200, 10208 bytes, all three oauth2_proxy_* families including requests_total{code="302"}. Same bug and fingerprint as platform/longhorn/networkpolicy-metrics.yaml records for Longhorn; the fix is one narrow rule per app, and it cannot ship from this tier because oauth2-proxy and apps-wave-0 are Flux SIBLINGS (both dependsOn platform-config), so naming ns jackett here would apply today and fail on a cold rebuild. Needs an owner in the renderer. Touches conformance/** : one KNOWN_EXTERNAL_MONITORS entry, which is the mechanism the failing check explicitly instructs ("DECLARE it ... with who creates it") and which flux-podmonitor.yaml already uses. It records that the port name was measured on prod instead of resolved from a Service, and that the targets are currently down. That makes this PR require a non-author review on the sensitive-path rule. Also: ratio alert still NOT shipped. oauth2_proxy_requests_total now exists on the endpoint but not in the TSDB, so writing it today would make it inert for exactly the reason it was refused on 2026-09-30. Runbook gains policy-vs-no-target triage and the radarr /signalr/messages/negotiate worked case — one orphaned browser tab, 206 identical 302s, nobody ever re-prompted. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
26 lines
1.6 KiB
YAML
26 lines
1.6 KiB
YAML
---
|
|
# oauth2-proxy component. Rendered by its OWN post-platform Flux Kustomization
|
|
# (clusters/base/oauth2-proxy.yaml, dependsOn platform-config, wait:false) — NOT by the gating
|
|
# `platform` tier. The httproute + servicemonitor (Gateway API / monitoring CRs) travel WITH the
|
|
# controller here now, rather than being referenced in-place from platform-config: the CRDs they
|
|
# need (gateway.networking.k8s.io, monitoring.coreos.com) are registered by platform and platform
|
|
# is Ready before this tier's platform-config dependency completes, so ordering holds.
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
resources:
|
|
- namespace.yaml
|
|
- helmrepository.yaml
|
|
- helmrelease.yaml
|
|
- oauth2-proxy-credentials.sops.yaml
|
|
- httproute.yaml # forward-auth sign-in/callback ingress (was referenced in-place by platform-config)
|
|
- servicemonitor.yaml # Prometheus scrape (was referenced in-place by platform-config)
|
|
# Alerting on whether the gates are OBSERVABLE at all. Travels with the servicemonitor above
|
|
# for the same reason it does: monitoring.coreos.com CRDs are registered by the `platform`
|
|
# tier, which is Ready before this tier's platform-config dependency completes.
|
|
- auth-visibility-alerts.yaml
|
|
# Discovers the PER-APP gate sidecars' :44180 metrics endpoints by pod label (PR #28 added the
|
|
# listener; no Service exposes it, so this is a PodMonitor, not a ServiceMonitor). The scrape is
|
|
# still denied by the app namespaces' default-deny ingress — read podmonitor.yaml's header
|
|
# before concluding the targets should be up.
|
|
- podmonitor.yaml
|