c3ce4db767d71a96e93944a571fa53587e07ffb7
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
bce101d03f |
feat(obs): wire the gate-sidecar scrape, and fix the alert that would have gone silent doing it
PR #28 gave the five fleet oauth2-proxy sidecars --metrics-address=0.0.0.0:44180 and a named `metrics` port. This wires a scraper to them and, in the process, found a real bug in the alert this branch added. PodMonitor, not ServiceMonitor. Measured: no Service exposes 44180 on any of the five (jackett/radarr/sonarr/qbittorrent/pihole all publish only their app port + authproxy 4180), so a ServiceMonitor would select nothing. A PodMonitor targets pod ports directly and needs no change to catalog/render.py or the apps/wave-* renders and goldens. Selector is the pod label homelab.keithjackson.dev/auth-mode=forward-auth — present on exactly those five gate pods, absent from crafty's, which still has no metrics listener. The property, not an app list. THE BUG: `up{container="oauth2-proxy"}` -> `up{container="oauth2-proxy"} == 1`. A target that is discovered but FAILING still publishes `up`, with value 0. The series exists, so count by (namespace) returned 1, `unless` removed the namespace, and OAuth2ProxyGateUnobserved went QUIET while nothing was being scraped at all — this branch's own disease reproduced inside the alert written to cure it. Not hypothetical: the PodMonitor lands five targets whose scrape is dropped, so the bare form would have silently cleared on this very commit. Proven by mutation: reverting the `== 1` fails the two new unit-test scenarios (got: nil) and leaves every other assertion green. The scrape does NOT work yet, and that is documented rather than papered over. From a pod in ns monitoring to the jackett pod, :44180 AND :4180 both time out; the shared proxy's :44180 answers. ns jackett's default-deny is podSelector:{} with policyTypes:[Ingress,Egress] and NO ingress key (deny-all), and the one allowance is fromEntities:[ingress] to 4180 — the Gateway's Envoy only. The endpoint itself is healthy: port-forwarded, /metrics returns http=200, 10208 bytes, all three oauth2_proxy_* families including requests_total{code="302"}. Same bug and fingerprint as platform/longhorn/networkpolicy-metrics.yaml records for Longhorn; the fix is one narrow rule per app, and it cannot ship from this tier because oauth2-proxy and apps-wave-0 are Flux SIBLINGS (both dependsOn platform-config), so naming ns jackett here would apply today and fail on a cold rebuild. Needs an owner in the renderer. Touches conformance/** : one KNOWN_EXTERNAL_MONITORS entry, which is the mechanism the failing check explicitly instructs ("DECLARE it ... with who creates it") and which flux-podmonitor.yaml already uses. It records that the port name was measured on prod instead of resolved from a Service, and that the targets are currently down. That makes this PR require a non-author review on the sensitive-path rule. Also: ratio alert still NOT shipped. oauth2_proxy_requests_total now exists on the endpoint but not in the TSDB, so writing it today would make it inert for exactly the reason it was refused on 2026-09-30. Runbook gains policy-vs-no-target triage and the radarr /signalr/messages/negotiate worked case — one orphaned browser tab, 206 identical 302s, nobody ever re-prompted. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
fc7ec01628 |
feat(obs): make oauth2-proxy auth failures visible — alert on the gap, not on absent metrics
Jackett failed on 2026-09-30 in the most expensive way available: HTTP 200 and a
plausible page. oauth2-proxy answers an unauthenticated XHR with a 302 to Keycloak
rather than a 401; the browser follows it silently against a live SSO session and
the call resolves 200 carrying a 44 KB HTML dashboard where the app's JavaScript
wanted JSON. The user is never re-prompted, reloading never helps, and every uptime
check stays green. `jackett-down` watches replicas_available, which was 1 throughout.
The reason nothing alerted turns out not to be a missing rule. Measured on prod:
count by (__name__) ({__name__=~"oauth2_proxy.*"})
oauth2_proxy_requests_in_flight => 2
Two series, both from the SHARED forward-auth Deployment. The six per-app gate
sidecars (crafty, jackett, pihole, qbittorrent, radarr, sonarr) publish nothing at
all: no --metrics-address, so the metrics listener never starts; no `metrics`
container port; no ServiceMonitor or PodMonitor selects them. The alert the incident
calls for — sustained auth redirects to an authenticated user — is therefore NOT
WRITABLE, and writing it anyway would have shipped a rule that is present,
evaluating, permanently silent and indistinguishable from health.
So alert on the condition that made the incident invisible, from families that ARE
scraped:
OAuth2ProxyGateUnobserved
count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
unless
count by (namespace) (up{container="oauth2-proxy"})
kube_pod_container_info is kube-state-metrics (a declared scrape owner) and sees the
container whether or not it exports anything; `up` is UNIVERSAL and carries a
`container` label for any target, so the rule needs no advance knowledge of what job
name a future per-app scrape picks. Nothing in it touches oauth2_proxy_*, so it
cannot be inert for the reason it is reporting. Run verbatim against live prod it
returns six namespaces and correctly excludes oauth2-proxy. It clears per namespace
as each gate gains a real scrape.
Also:
- promtool companion test pinning all four states: fires unscraped, quiet when
scraped, honours for:1h, and STOPS once a scrape appears under a different job name.
- runbooks/oauth2-proxy-auth-failures.md — tell the gate's refusal from the group
check from the app's own login from an XHR that silently got HTML, with the
2026-09-30 jackett trace as the worked example and the Loki queries to use while
there are no metrics. Registered in the configMapGenerator and README.
- runbooks/README.md gains an inert-alerts row for the unscraped sidecars.
Rejected: an alert on the shared proxy's oauth2_proxy_requests_total ratio. It needs
a conformance/ SCRAPE_OWNERS entry (human approval) and, decisively, would not have
fired on this incident — the shared instance is not in any sidecar-gated app's data
path and its current pods have proxied nothing. It would have looked like coverage
and been none.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
960cab937b |
fix(flux): move oauth2-proxy to a post-platform tier (dependsOn platform-config)
oauth2-proxy is a leaf forward-auth service that depends on Keycloak + trusted wildcard TLS + split-horizon DNS. It lived in the gating `platform` Flux Kustomization (wait:true), so on a cold start it Failed until those dependencies existed, keeping the whole `platform` tier not-Ready. That in turn blocked `platform-config` — which brings up the very Keycloak instance, wildcard Certificate and L2 VIP oauth2-proxy needs (and previously carried oauth2-proxy's own httproute/servicemonitor) — from ever reconciling. A self-gating cold-start deadlock, twice-proven (DECISIONS-LOG sessions 7 & 9). Break the cycle by moving the whole oauth2-proxy component out of the `platform` tier into its own non-gating post-platform Flux Kustomization: - clusters/base/oauth2-proxy.yaml: new Kustomization -> ./platform/oauth2-proxy, dependsOn platform-config, wait:false, substituteFrom cluster-vars. - clusters/base/kustomization.yaml: reference the new Kustomization. - platform/kustomization.yaml: drop the oauth2-proxy component (no longer in the gating tier). - platform/oauth2-proxy/kustomization.yaml: pull in httproute + servicemonitor so the full component travels together. - platform-config/kustomization.yaml: drop the in-place oauth2-proxy httproute/servicemonitor refs (moved with the controller). - clusters/dev/kustomization.yaml: add the SOPS decryption patch for the new oauth2-proxy Kustomization (it now owns the encrypted credentials Secret that the `platform` Kustomization used to decrypt). Now `platform` reaches Ready without oauth2-proxy; platform-config brings up Keycloak/TLS/DNS; then oauth2-proxy reconciles against dependencies that already exist. wait:false guarantees its health can never gate anything above it. No config of oauth2-proxy itself changed — only its reconciliation tier. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
ed6a772599 |
feat(keycloak): provision SSO client secrets + re-enable oauth2-proxy (R-IDP-01/02/04)
Generate the 3 OIDC client secrets (SOPS-encrypted, minimal manifests) so the realm import can inject the grafana + oauth2-proxy client secrets and both consumers match: - platform/keycloak/keycloak-realm-clients.sops.yaml (grafana + oauth2-proxy client secrets) - platform/oauth2-proxy/oauth2-proxy-credentials.sops.yaml (client-id/secret/cookie) - platform/monitoring/grafana-oidc-credentials.sops.yaml (grafana client secret) Wire each into its component kustomization and re-enable oauth2-proxy in platform + platform-config (reverses the temp deferral now that its secret exists). Values generated by PM per operator instruction (random; recorded out-of-repo at ~/.config/home-k3s/keycloak-sso-secrets.txt for later rotation). Encrypted via sops in the dev container to the age recipient in .sops.yaml; plaintext never committed. Serves: R-IDP-01, R-IDP-02, R-IDP-04, R-SEC-01. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
af4f6c07ae |
T-0.1 Flux GitOps layout + dev/prod overlays + platform component skeletons (R-GITOPS-01/03, R-ENV-02, R-PLAT-01)
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |