---
# PodMonitor for the PER-APP oauth2-proxy gate sidecars (R-OBS-04).
#
# ============================================================================================
# WHY A PodMonitor AND NOT A ServiceMonitor
# ============================================================================================
# PR #28 gave all five fleet sidecars `--metrics-address=0.0.0.0:44180` and a named `metrics`
# container port. It did NOT add a Service port, and measured on prod 2026-10-01 none of the five
# Services exposes 44180:
#
#   jackett      jackett:     [('port-9117', 9117), ('authproxy', 4180)]
#   radarr       radarr:      [('port-7878', 7878), ('authproxy', 4180)]
#   sonarr       sonarr:      [('port-8989', 8989), ('authproxy', 4180)]
#   qbittorrent  qbittorrent: [('port-8080', 8080), ('authproxy', 4180)]
#   pihole       pihole:      [('port-80',     80), ('authproxy', 4180)]
#
# A ServiceMonitor selects Service PORTS, so it would have nothing to select. A PodMonitor targets
# pod ports directly and bypasses Services entirely — so this needs no change to the Services, and
# therefore no change to `catalog/render.py` or the `apps/wave-*` renders and goldens.
#
# SELECTOR: the `homelab.keithjackson.dev/auth-mode: forward-auth` POD LABEL, not a namespace list.
# Measured 2026-10-01: that label is present on exactly the five fleet gate pods (jackett, radarr,
# sonarr, qbittorrent, pihole) and absent from crafty's, which is a separately-managed deployment
# (platform/crafty) that still has no `--metrics-address` at all. So the selector describes the
# PROPERTY being scraped rather than enumerating today's apps, and a sixth forward-auth app is
# picked up with no edit here. `namespaceSelector.any: true` because those five namespaces are
# created by the apps-wave-0/1 tiers, which this tier does not and must not depend on.
#
# ============================================================================================
# STATUS: THE SCRAPE WORKS. Verified on prod 2026-10-05 (PR #49 merged as 21c88b0).
# ============================================================================================
# All 6 targets in pool podMonitor/oauth2-proxy/oauth2-proxy-sidecars/0 read `up == 1`,
# health=up, lastError='', lastScrapeDuration 0.001-0.006s. The TSDB holds oauth2_proxy_* for
# these namespaces for the first time: 5 metric names / 148 series, with live counters
# (oauth2_proxy_requests_total 35072 qbittorrent, 23560 radarr, 1284 sonarr). All 5 TargetDown
# and 5 of 6 OAuth2ProxyGateUnobserved resolved; only namespace="crafty" still fires, and that
# is correct — see CRAFTY below.
#
# The missing piece was an ingress allowance, and it now ships WITH EACH APP, rendered by
# catalog/render.py's r_monitoring_ingress_policy as `<app>-allow-monitoring-scrape`:
# ns monitoring -> the app's endpointSelector -> TCP 44180 only. See WHY NOT IN THIS DIRECTORY
# below for why it is rendered per-app instead of shipped from this tier.
#
# Keep the history below. It is why the fix is shaped the way it is, and `up == 1` in
# auth-visibility-alerts.yaml is load-bearing precisely because of it.
#
# --------------------------------------------------------------------------------------------
# HISTORY: 2026-10-02 -> 2026-10-05, THE SCRAPE WAS DENIED BY NETWORKPOLICY FOR 63.7h
# --------------------------------------------------------------------------------------------
# This PodMonitor was NECESSARY AND NOT SUFFICIENT: the endpoint served, and the path from ns
# monitoring was blocked. Both halves measured on prod 2026-10-01, read-only:
#
#   THE ENDPOINT IS HEALTHY — port-forward straight to the jackett pod's 44180:
#     GET /metrics -> http=200 bytes=10208
#     # TYPE oauth2_proxy_requests_in_flight gauge
#     # TYPE oauth2_proxy_requests_total counter
#     # TYPE oauth2_proxy_response_duration_seconds histogram
#     oauth2_proxy_requests_total{code="200"} 3
#     oauth2_proxy_requests_total{code="302"} 2
#
#   THE SCRAPE PATH IS DROPPED — from a pod in ns monitoring (grafana, which sits beside
#   Prometheus and so occupies the same policy position) to the jackett pod IP:
#     http://10.0.0.189:44180/metrics   -> wget: download timed out
#     http://10.0.0.189:4180/ping       -> wget: download timed out     (even the "allowed" port)
#     http://<shared-proxy>:44180/metrics -> 200, oauth2_proxy_* present (control: ns oauth2-proxy
#                                            has NO NetworkPolicy, which is the only reason the
#                                            shared instance has ever been scrapable)
#
#   WHY: ns jackett's `default-deny` NetworkPolicy is `podSelector: {}` with
#   `policyTypes: [Ingress, Egress]` and an `egress` rule ONLY — no `ingress` key, which in
#   NetworkPolicy semantics is deny-all-ingress. The only ingress allowance is the CNP
#   `jackett-allow-gateway-ingress`: `fromEntities: [ingress]` to port 4180 — the Gateway's Envoy
#   and nothing else. Prometheus matches no clause and its packets are dropped, not refused.
#
# THE TARGETS READ `up == 0` FROM 2026-10-02 05:09 UTC UNTIL THE ALLOWANCE LANDED — one
# continuous 63.7h episode (a 15d query_range shows 1 episode per namespace, not flapping),
# starting 11 minutes after this file was created, i.e. TargetDown's `for: 10m`. It was never
# silenced, and that was the right call: `OAuth2ProxyGateUnobserved` in auth-visibility-alerts.yaml
# requires `up == 1`, precisely so a discovered-but-failing target does not read as "observed"
# (see that file's header — this exact trap is why the `== 1` is there).
#
# The signature to recognise if it ever returns: lastScrapeDuration pinned at EXACTLY the
# scrapeTimeout (10.0s) with `context deadline exceeded`. That is a DROP, not a refusal — a
# refused connection fails in milliseconds. Post-fix the same targets scrape in 1-6ms.
#
# THIS IS THE SAME BUG LONGHORN HAD, and the platform has a documented fix idiom for it:
# platform/longhorn/networkpolicy-metrics.yaml, whose header records the identical fingerprint
# measured on 2026-09-13 — ServiceMonitor in place, 3 targets DISCOVERED, `up` 0/0/0, TSDB empty,
# `lastError: context deadline exceeded` (a drop, not a refusal). The fix there is a narrow
# additive `networking.k8s.io/v1` NetworkPolicy: ns monitoring -> the pod label, that one port.
#
# WHY NOT IN THIS DIRECTORY (the reasoning the fix followed). It would have to be five objects, one per app
# namespace, and those namespaces are created by apps-wave-0/1. Measured Flux graph, 2026-10-01:
#   oauth2-proxy  dependsOn=['platform-config']        <- this tier
#   apps-wave-0   dependsOn=['platform-config']        <- SIBLING, no ordering between them
#   apps-wave-1   dependsOn=['apps-wave-0']
# They are siblings, so on a cold bootstrap there is no guarantee ns jackett exists when this tier
# applies, and a NetworkPolicy naming a missing namespace fails to apply. It would work on today's
# running cluster and break on a rebuild — the "works now, broken on restore" shape this repo
# keeps paying for. Longhorn's precedent is safe only because longhorn-system is created by the
# same tier that ships its policy. The allowance therefore belongs WITH THE APPS, rendered next to
# the rest of each app's network policy — i.e. in the catalog/renderer. THAT IS WHERE IT LANDED:
# `r_monitoring_ingress_policy`, keyed on `_needs_oauth2_sidecar`, one narrow rule per app.
#
# --------------------------------------------------------------------------------------------
# CRAFTY: STILL UNOBSERVED, AND THIS FILE CANNOT HELP IT
# --------------------------------------------------------------------------------------------
# `OAuth2ProxyGateUnobserved{namespace="crafty"}` still fires after the fix, correctly. crafty is
# a hand-maintained Deployment under platform/crafty, OUTSIDE the renderer, and its oauth2-proxy
# container has NO `--metrics-address` at all (measured 2026-10-05) — so its metrics listener
# never starts and there is no port to open. The selector below therefore does not match it, and
# the per-app allowance is never rendered for it. Closing crafty's gap needs the flag plus a
# `metrics` containerPort added to its own manifest FIRST; the allowance is the second half, and
# it would have to be written by hand there rather than rendered. Open follow-up work.
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: oauth2-proxy-sidecars
  namespace: oauth2-proxy
  labels:
    release: kube-prometheus-stack     # kube-prometheus-stack's podMonitorSelector
    app.kubernetes.io/name: oauth2-proxy
    homelab.keithjackson.dev/purpose: metrics-scrape
spec:
  namespaceSelector:
    # The gate sidecars live in the app namespaces, not this one. `any` rather than a fixed list so
    # a new forward-auth app needs no edit here — the pod label below is what scopes this.
    any: true
  selector:
    matchLabels:
      # The PROPERTY, not the app list. Present on exactly the five fleet gate pods (measured
      # 2026-10-01); absent from crafty, which has no metrics listener yet.
      homelab.keithjackson.dev/auth-mode: forward-auth
  podMetricsEndpoints:
    # By PORT NAME (`metrics`, 44180), never by number: the name is what PR #28 declared on the
    # container, and a numeric port here would silently stop matching if the port were renumbered.
    - port: metrics
      path: /metrics
      interval: 30s
      # 10s, matching the drop signature above: a DENIED path does not refuse, it hangs until the
      # scrape times out. Leaving the default would make each failing target hold a scrape slot for
      # the full interval.
      scrapeTimeout: 10s