2 Commits

Include renames

Author	SHA1	Message	Date
jakman85andClaude Opus 5	81f627167d	docs(obs): record that the gate-sidecar scrape works — four places still said it did not ... PR #49 (merged 21c88b0) fixed the blocked scrape and it is verified working on prod 2026-10-05: all 6 targets up == 1, health=up, lastError='', lastScrapeDuration 0.001-0.006s against 10.0s and "context deadline exceeded" before. oauth2_proxy_* is in the TSDB for these namespaces for the first time (5 metric names, 148 series). Four places in the tree still asserted the pre-fix state as current, in the present tense, in the files a reader consults first: platform/oauth2-proxy/podmonitor.yaml "READ THIS BEFORE BELIEVING THE TARGETS ARE UP: THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY" and "SO THE TARGETS BELOW WILL READ up == 0 UNTIL AN INGRESS ALLOWANCE LANDS". The first actively instructs the reader to disbelieve a correct up == 1. Now a STATUS block with the post-fix numbers, with the diagnosis kept below it under an explicit HISTORY heading because it is why the fix is shaped the way it is. catalog/render.py:159 "THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED". catalog/render.py:1365 the same claim on the sidecar arg. platform/oauth2-proxy/README.md its subsections are dated and so are honest records, but a reader hits the stale body long before any correction. A SUPERSEDED pointer now sits under the heading and an Update 2026-10-05 section carries the before/after. The "not scraped" claims are corrected; the "a flag alone is not enough" warning is KEPT deliberately, because that part is still true and "flag without ingress allowance" is exactly what caused the 63.7h outage. Two substantive errors fixed while there, not just tense. The README's "three things are needed" listed a Service port and a ServiceMonitor — wrong, a PodMonitor targets pod ports directly, which is why no Service exposes 44180 and the renderer needed no Service change; following it would have sent someone to edit Service rendering for nothing. And podmonitor.yaml deferred the fix to "the catalog/renderer, in flight on another branch" — no such branch existed; it now names r_monitoring_ingress_policy, with a new CRAFTY section explaining the one namespace this file cannot help. Comments only in render.py, so the rendered output is byte-identical: no apps/ file and no golden changes, and therefore no sensitive path and no human approval needed — the opposite of the PR this documents. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-05 21:18:33 -05:00
jakman85andClaude Opus 5	bce101d03f	feat(obs): wire the gate-sidecar scrape, and fix the alert that would have gone silent doing it ... PR #28 gave the five fleet oauth2-proxy sidecars --metrics-address=0.0.0.0:44180 and a named `metrics` port. This wires a scraper to them and, in the process, found a real bug in the alert this branch added. PodMonitor, not ServiceMonitor. Measured: no Service exposes 44180 on any of the five (jackett/radarr/sonarr/qbittorrent/pihole all publish only their app port + authproxy 4180), so a ServiceMonitor would select nothing. A PodMonitor targets pod ports directly and needs no change to catalog/render.py or the apps/wave-* renders and goldens. Selector is the pod label homelab.keithjackson.dev/auth-mode=forward-auth — present on exactly those five gate pods, absent from crafty's, which still has no metrics listener. The property, not an app list. THE BUG: `up{container="oauth2-proxy"}` -> `up{container="oauth2-proxy"} == 1`. A target that is discovered but FAILING still publishes `up`, with value 0. The series exists, so count by (namespace) returned 1, `unless` removed the namespace, and OAuth2ProxyGateUnobserved went QUIET while nothing was being scraped at all — this branch's own disease reproduced inside the alert written to cure it. Not hypothetical: the PodMonitor lands five targets whose scrape is dropped, so the bare form would have silently cleared on this very commit. Proven by mutation: reverting the `== 1` fails the two new unit-test scenarios (got: nil) and leaves every other assertion green. The scrape does NOT work yet, and that is documented rather than papered over. From a pod in ns monitoring to the jackett pod, :44180 AND :4180 both time out; the shared proxy's :44180 answers. ns jackett's default-deny is podSelector:{} with policyTypes:[Ingress,Egress] and NO ingress key (deny-all), and the one allowance is fromEntities:[ingress] to 4180 — the Gateway's Envoy only. The endpoint itself is healthy: port-forwarded, /metrics returns http=200, 10208 bytes, all three oauth2_proxy_* families including requests_total{code="302"}. Same bug and fingerprint as platform/longhorn/networkpolicy-metrics.yaml records for Longhorn; the fix is one narrow rule per app, and it cannot ship from this tier because oauth2-proxy and apps-wave-0 are Flux SIBLINGS (both dependsOn platform-config), so naming ns jackett here would apply today and fail on a cold rebuild. Needs an owner in the renderer. Touches conformance/** : one KNOWN_EXTERNAL_MONITORS entry, which is the mechanism the failing check explicitly instructs ("DECLARE it ... with who creates it") and which flux-podmonitor.yaml already uses. It records that the port name was measured on prod instead of resolved from a Service, and that the targets are currently down. That makes this PR require a non-author review on the sensitive-path rule. Also: ratio alert still NOT shipped. oauth2_proxy_requests_total now exists on the endpoint but not in the TSDB, so writing it today would make it inert for exactly the reason it was refused on 2026-09-30. Runbook gains policy-vs-no-target triage and the radarr /signalr/messages/negotiate worked case — one orphaned browser tab, 206 identical 302s, nobody ever re-prompted. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-01 21:56:07 -05:00