jakman85andClaude Opus 5
81f627167d
PR #49 (merged 21c88b0) fixed the blocked scrape and it is verified working on prod
2026-10-05: all 6 targets up == 1, health=up, lastError='', lastScrapeDuration 0.001-0.006s
against 10.0s and "context deadline exceeded" before. oauth2_proxy_* is in the TSDB for
these namespaces for the first time (5 metric names, 148 series).
Four places in the tree still asserted the pre-fix state as current, in the present tense,
in the files a reader consults first:
platform/oauth2-proxy/podmonitor.yaml "READ THIS BEFORE BELIEVING THE TARGETS ARE UP:
THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY" and "SO THE TARGETS BELOW WILL READ
up == 0 UNTIL AN INGRESS ALLOWANCE LANDS". The first actively instructs the reader to
disbelieve a correct up == 1. Now a STATUS block with the post-fix numbers, with the
diagnosis kept below it under an explicit HISTORY heading because it is why the fix is
shaped the way it is.
catalog/render.py:159 "THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED".
catalog/render.py:1365 the same claim on the sidecar arg.
platform/oauth2-proxy/README.md its subsections are dated and so are honest records, but
a reader hits the stale body long before any correction. A SUPERSEDED pointer now sits
under the heading and an Update 2026-10-05 section carries the before/after.
The "not scraped" claims are corrected; the "a flag alone is not enough" warning is KEPT
deliberately, because that part is still true and "flag without ingress allowance" is
exactly what caused the 63.7h outage.
Two substantive errors fixed while there, not just tense. The README's "three things are
needed" listed a Service port and a ServiceMonitor — wrong, a PodMonitor targets pod ports
directly, which is why no Service exposes 44180 and the renderer needed no Service change;
following it would have sent someone to edit Service rendering for nothing. And
podmonitor.yaml deferred the fix to "the catalog/renderer, in flight on another branch" —
no such branch existed; it now names r_monitoring_ingress_policy, with a new CRAFTY section
explaining the one namespace this file cannot help.
Comments only in render.py, so the rendered output is byte-identical: no apps/ file and no
golden changes, and therefore no sensitive path and no human approval needed — the opposite
of the PR this documents.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
144 lines
9.4 KiB
YAML
144 lines
9.4 KiB
YAML
---
|
|
# PodMonitor for the PER-APP oauth2-proxy gate sidecars (R-OBS-04).
|
|
#
|
|
# ============================================================================================
|
|
# WHY A PodMonitor AND NOT A ServiceMonitor
|
|
# ============================================================================================
|
|
# PR #28 gave all five fleet sidecars `--metrics-address=0.0.0.0:44180` and a named `metrics`
|
|
# container port. It did NOT add a Service port, and measured on prod 2026-10-01 none of the five
|
|
# Services exposes 44180:
|
|
#
|
|
# jackett jackett: [('port-9117', 9117), ('authproxy', 4180)]
|
|
# radarr radarr: [('port-7878', 7878), ('authproxy', 4180)]
|
|
# sonarr sonarr: [('port-8989', 8989), ('authproxy', 4180)]
|
|
# qbittorrent qbittorrent: [('port-8080', 8080), ('authproxy', 4180)]
|
|
# pihole pihole: [('port-80', 80), ('authproxy', 4180)]
|
|
#
|
|
# A ServiceMonitor selects Service PORTS, so it would have nothing to select. A PodMonitor targets
|
|
# pod ports directly and bypasses Services entirely — so this needs no change to the Services, and
|
|
# therefore no change to `catalog/render.py` or the `apps/wave-*` renders and goldens.
|
|
#
|
|
# SELECTOR: the `homelab.keithjackson.dev/auth-mode: forward-auth` POD LABEL, not a namespace list.
|
|
# Measured 2026-10-01: that label is present on exactly the five fleet gate pods (jackett, radarr,
|
|
# sonarr, qbittorrent, pihole) and absent from crafty's, which is a separately-managed deployment
|
|
# (platform/crafty) that still has no `--metrics-address` at all. So the selector describes the
|
|
# PROPERTY being scraped rather than enumerating today's apps, and a sixth forward-auth app is
|
|
# picked up with no edit here. `namespaceSelector.any: true` because those five namespaces are
|
|
# created by the apps-wave-0/1 tiers, which this tier does not and must not depend on.
|
|
#
|
|
# ============================================================================================
|
|
# STATUS: THE SCRAPE WORKS. Verified on prod 2026-10-05 (PR #49 merged as 21c88b0).
|
|
# ============================================================================================
|
|
# All 6 targets in pool podMonitor/oauth2-proxy/oauth2-proxy-sidecars/0 read `up == 1`,
|
|
# health=up, lastError='', lastScrapeDuration 0.001-0.006s. The TSDB holds oauth2_proxy_* for
|
|
# these namespaces for the first time: 5 metric names / 148 series, with live counters
|
|
# (oauth2_proxy_requests_total 35072 qbittorrent, 23560 radarr, 1284 sonarr). All 5 TargetDown
|
|
# and 5 of 6 OAuth2ProxyGateUnobserved resolved; only namespace="crafty" still fires, and that
|
|
# is correct — see CRAFTY below.
|
|
#
|
|
# The missing piece was an ingress allowance, and it now ships WITH EACH APP, rendered by
|
|
# catalog/render.py's r_monitoring_ingress_policy as `<app>-allow-monitoring-scrape`:
|
|
# ns monitoring -> the app's endpointSelector -> TCP 44180 only. See WHY NOT IN THIS DIRECTORY
|
|
# below for why it is rendered per-app instead of shipped from this tier.
|
|
#
|
|
# Keep the history below. It is why the fix is shaped the way it is, and `up == 1` in
|
|
# auth-visibility-alerts.yaml is load-bearing precisely because of it.
|
|
#
|
|
# --------------------------------------------------------------------------------------------
|
|
# HISTORY: 2026-10-02 -> 2026-10-05, THE SCRAPE WAS DENIED BY NETWORKPOLICY FOR 63.7h
|
|
# --------------------------------------------------------------------------------------------
|
|
# This PodMonitor was NECESSARY AND NOT SUFFICIENT: the endpoint served, and the path from ns
|
|
# monitoring was blocked. Both halves measured on prod 2026-10-01, read-only:
|
|
#
|
|
# THE ENDPOINT IS HEALTHY — port-forward straight to the jackett pod's 44180:
|
|
# GET /metrics -> http=200 bytes=10208
|
|
# # TYPE oauth2_proxy_requests_in_flight gauge
|
|
# # TYPE oauth2_proxy_requests_total counter
|
|
# # TYPE oauth2_proxy_response_duration_seconds histogram
|
|
# oauth2_proxy_requests_total{code="200"} 3
|
|
# oauth2_proxy_requests_total{code="302"} 2
|
|
#
|
|
# THE SCRAPE PATH IS DROPPED — from a pod in ns monitoring (grafana, which sits beside
|
|
# Prometheus and so occupies the same policy position) to the jackett pod IP:
|
|
# http://10.0.0.189:44180/metrics -> wget: download timed out
|
|
# http://10.0.0.189:4180/ping -> wget: download timed out (even the "allowed" port)
|
|
# http://<shared-proxy>:44180/metrics -> 200, oauth2_proxy_* present (control: ns oauth2-proxy
|
|
# has NO NetworkPolicy, which is the only reason the
|
|
# shared instance has ever been scrapable)
|
|
#
|
|
# WHY: ns jackett's `default-deny` NetworkPolicy is `podSelector: {}` with
|
|
# `policyTypes: [Ingress, Egress]` and an `egress` rule ONLY — no `ingress` key, which in
|
|
# NetworkPolicy semantics is deny-all-ingress. The only ingress allowance is the CNP
|
|
# `jackett-allow-gateway-ingress`: `fromEntities: [ingress]` to port 4180 — the Gateway's Envoy
|
|
# and nothing else. Prometheus matches no clause and its packets are dropped, not refused.
|
|
#
|
|
# THE TARGETS READ `up == 0` FROM 2026-10-02 05:09 UTC UNTIL THE ALLOWANCE LANDED — one
|
|
# continuous 63.7h episode (a 15d query_range shows 1 episode per namespace, not flapping),
|
|
# starting 11 minutes after this file was created, i.e. TargetDown's `for: 10m`. It was never
|
|
# silenced, and that was the right call: `OAuth2ProxyGateUnobserved` in auth-visibility-alerts.yaml
|
|
# requires `up == 1`, precisely so a discovered-but-failing target does not read as "observed"
|
|
# (see that file's header — this exact trap is why the `== 1` is there).
|
|
#
|
|
# The signature to recognise if it ever returns: lastScrapeDuration pinned at EXACTLY the
|
|
# scrapeTimeout (10.0s) with `context deadline exceeded`. That is a DROP, not a refusal — a
|
|
# refused connection fails in milliseconds. Post-fix the same targets scrape in 1-6ms.
|
|
#
|
|
# THIS IS THE SAME BUG LONGHORN HAD, and the platform has a documented fix idiom for it:
|
|
# platform/longhorn/networkpolicy-metrics.yaml, whose header records the identical fingerprint
|
|
# measured on 2026-09-13 — ServiceMonitor in place, 3 targets DISCOVERED, `up` 0/0/0, TSDB empty,
|
|
# `lastError: context deadline exceeded` (a drop, not a refusal). The fix there is a narrow
|
|
# additive `networking.k8s.io/v1` NetworkPolicy: ns monitoring -> the pod label, that one port.
|
|
#
|
|
# WHY NOT IN THIS DIRECTORY (the reasoning the fix followed). It would have to be five objects, one per app
|
|
# namespace, and those namespaces are created by apps-wave-0/1. Measured Flux graph, 2026-10-01:
|
|
# oauth2-proxy dependsOn=['platform-config'] <- this tier
|
|
# apps-wave-0 dependsOn=['platform-config'] <- SIBLING, no ordering between them
|
|
# apps-wave-1 dependsOn=['apps-wave-0']
|
|
# They are siblings, so on a cold bootstrap there is no guarantee ns jackett exists when this tier
|
|
# applies, and a NetworkPolicy naming a missing namespace fails to apply. It would work on today's
|
|
# running cluster and break on a rebuild — the "works now, broken on restore" shape this repo
|
|
# keeps paying for. Longhorn's precedent is safe only because longhorn-system is created by the
|
|
# same tier that ships its policy. The allowance therefore belongs WITH THE APPS, rendered next to
|
|
# the rest of each app's network policy — i.e. in the catalog/renderer. THAT IS WHERE IT LANDED:
|
|
# `r_monitoring_ingress_policy`, keyed on `_needs_oauth2_sidecar`, one narrow rule per app.
|
|
#
|
|
# --------------------------------------------------------------------------------------------
|
|
# CRAFTY: STILL UNOBSERVED, AND THIS FILE CANNOT HELP IT
|
|
# --------------------------------------------------------------------------------------------
|
|
# `OAuth2ProxyGateUnobserved{namespace="crafty"}` still fires after the fix, correctly. crafty is
|
|
# a hand-maintained Deployment under platform/crafty, OUTSIDE the renderer, and its oauth2-proxy
|
|
# container has NO `--metrics-address` at all (measured 2026-10-05) — so its metrics listener
|
|
# never starts and there is no port to open. The selector below therefore does not match it, and
|
|
# the per-app allowance is never rendered for it. Closing crafty's gap needs the flag plus a
|
|
# `metrics` containerPort added to its own manifest FIRST; the allowance is the second half, and
|
|
# it would have to be written by hand there rather than rendered. Open follow-up work.
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PodMonitor
|
|
metadata:
|
|
name: oauth2-proxy-sidecars
|
|
namespace: oauth2-proxy
|
|
labels:
|
|
release: kube-prometheus-stack # kube-prometheus-stack's podMonitorSelector
|
|
app.kubernetes.io/name: oauth2-proxy
|
|
homelab.keithjackson.dev/purpose: metrics-scrape
|
|
spec:
|
|
namespaceSelector:
|
|
# The gate sidecars live in the app namespaces, not this one. `any` rather than a fixed list so
|
|
# a new forward-auth app needs no edit here — the pod label below is what scopes this.
|
|
any: true
|
|
selector:
|
|
matchLabels:
|
|
# The PROPERTY, not the app list. Present on exactly the five fleet gate pods (measured
|
|
# 2026-10-01); absent from crafty, which has no metrics listener yet.
|
|
homelab.keithjackson.dev/auth-mode: forward-auth
|
|
podMetricsEndpoints:
|
|
# By PORT NAME (`metrics`, 44180), never by number: the name is what PR #28 declared on the
|
|
# container, and a numeric port here would silently stop matching if the port were renumbered.
|
|
- port: metrics
|
|
path: /metrics
|
|
interval: 30s
|
|
# 10s, matching the drop signature above: a DENIED path does not refuse, it hangs until the
|
|
# scrape times out. Leaving the default would make each failing target hold a scrape slot for
|
|
# the full interval.
|
|
scrapeTimeout: 10s
|