jakman85andClaude Opus 5
bce101d03f
PR #28 gave the five fleet oauth2-proxy sidecars --metrics-address=0.0.0.0:44180 and a named `metrics` port. This wires a scraper to them and, in the process, found a real bug in the alert this branch added. PodMonitor, not ServiceMonitor. Measured: no Service exposes 44180 on any of the five (jackett/radarr/sonarr/qbittorrent/pihole all publish only their app port + authproxy 4180), so a ServiceMonitor would select nothing. A PodMonitor targets pod ports directly and needs no change to catalog/render.py or the apps/wave-* renders and goldens. Selector is the pod label homelab.keithjackson.dev/auth-mode=forward-auth — present on exactly those five gate pods, absent from crafty's, which still has no metrics listener. The property, not an app list. THE BUG: `up{container="oauth2-proxy"}` -> `up{container="oauth2-proxy"} == 1`. A target that is discovered but FAILING still publishes `up`, with value 0. The series exists, so count by (namespace) returned 1, `unless` removed the namespace, and OAuth2ProxyGateUnobserved went QUIET while nothing was being scraped at all — this branch's own disease reproduced inside the alert written to cure it. Not hypothetical: the PodMonitor lands five targets whose scrape is dropped, so the bare form would have silently cleared on this very commit. Proven by mutation: reverting the `== 1` fails the two new unit-test scenarios (got: nil) and leaves every other assertion green. The scrape does NOT work yet, and that is documented rather than papered over. From a pod in ns monitoring to the jackett pod, :44180 AND :4180 both time out; the shared proxy's :44180 answers. ns jackett's default-deny is podSelector:{} with policyTypes:[Ingress,Egress] and NO ingress key (deny-all), and the one allowance is fromEntities:[ingress] to 4180 — the Gateway's Envoy only. The endpoint itself is healthy: port-forwarded, /metrics returns http=200, 10208 bytes, all three oauth2_proxy_* families including requests_total{code="302"}. Same bug and fingerprint as platform/longhorn/networkpolicy-metrics.yaml records for Longhorn; the fix is one narrow rule per app, and it cannot ship from this tier because oauth2-proxy and apps-wave-0 are Flux SIBLINGS (both dependsOn platform-config), so naming ns jackett here would apply today and fail on a cold rebuild. Needs an owner in the renderer. Touches conformance/** : one KNOWN_EXTERNAL_MONITORS entry, which is the mechanism the failing check explicitly instructs ("DECLARE it ... with who creates it") and which flux-podmonitor.yaml already uses. It records that the port name was measured on prod instead of resolved from a Service, and that the targets are currently down. That makes this PR require a non-author review on the sensitive-path rule. Also: ratio alert still NOT shipped. oauth2_proxy_requests_total now exists on the endpoint but not in the TSDB, so writing it today would make it inert for exactly the reason it was refused on 2026-09-30. Runbook gains policy-vs-no-target triage and the radarr /signalr/messages/negotiate worked case — one orphaned browser tab, 206 identical 302s, nobody ever re-prompted. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
160 lines
11 KiB
Markdown
160 lines
11 KiB
Markdown
# platform/oauth2-proxy
|
|
|
|
**oauth2-proxy** — forward-auth at the Gateway for apps **without usable OIDC**, authenticating
|
|
against Keycloak.
|
|
|
|
- **Governing ADR:** [ADR-0014 — IdP: Keycloak](../../../home-network-k3s/decisions/adr-0014-idp-keycloak.md)
|
|
- **Spec:** [docs/15 — Identity & Access](../../../home-network-k3s/docs/15-identity-and-access.md)
|
|
- **Serves:** R-IDP-01..06 (esp. R-IDP-02: management UIs never unauthenticated)
|
|
- **Owner:** WS-SEC (Gateway wiring shared with WS-NET)
|
|
|
|
## What this stub is
|
|
|
|
Flux `HelmRelease` for oauth2-proxy as the forward-auth provider for non-OIDC apps (PiHole, *arr UIs,
|
|
qBittorrent, Homepage, Prometheus, Alertmanager, Hubble UI — ADR-0014). Replica count is an overlay
|
|
knob (`${OAUTH2_PROXY_REPLICAS}`, R-ENV-02). Client/cookie secrets referenced **by name**,
|
|
**SOPS-encrypted** (WS-SEC).
|
|
|
|
## Contract (SSO/auth + Gateway)
|
|
|
|
Pairs with [`../keycloak`](../keycloak) to realize the "SSO/auth contract": a ServiceDefinition `auth:
|
|
forward-auth` service is protected by oauth2-proxy at the Gateway (ext-auth); `auth: oidc` services go
|
|
native to Keycloak. Gateway ext-auth wiring is a **WS-NET** contract.
|
|
|
|
## Enforcement (ext_authz forward-auth)
|
|
|
|
The **authenticator** (oauth2-proxy) is deployed and wired to Keycloak: `provider=oidc`,
|
|
`oidc-issuer-url=.../realms/homelab`, `--reverse-proxy`, `--upstream=static://202`,
|
|
`--set-xauthrequest`, cookie/redirect domain `.home.keithjackson.dev`, and — added here —
|
|
`--allowed-groups=platform-admins` (R-IDP-06 floor) + `--skip-provider-button` (302 straight to
|
|
Keycloak). The Keycloak `oauth2-proxy` client already carries the `auth.home…/oauth2/callback`
|
|
redirect URI and a `groups` claim with `full.path=false` (bare `platform-admins`) — no realm change
|
|
needed.
|
|
|
|
**Mechanism.** Forward-auth is an Envoy `ext_authz` HTTP filter that calls oauth2-proxy's
|
|
`/oauth2/auth` decision endpoint: **202** for a valid session in `platform-admins` (request passes to
|
|
the upstream, `X-Auth-Request-*` identity headers forwarded), **401/403** otherwise. With
|
|
`failure_mode_allow: false` the gateway **fails closed** — a denied or errored check never reaches the
|
|
upstream UI (R-IDP-02). The filter + oauth2-proxy cluster live in
|
|
[`forward-auth-cec.yaml`](./forward-auth-cec.yaml).
|
|
|
|
### Why this is NOT wired into the shared Gateway (Cilium 1.16.5 constraint)
|
|
|
|
The shared Gateway's Envoy program is a **machine-owned, constantly-regenerated**
|
|
`CiliumEnvoyConfig` (`gateway/cilium-gateway-shared`, `ownerReferences->Gateway`, regenerated on every
|
|
HTTPRoute change). Cilium 1.16.5 exposes **no supported hook** to inject an `ext_authz` filter into
|
|
that generated listener, and native Gateway-API **ExternalAuth (GEP-1494) ships in no released Cilium
|
|
version yet** (PoC only). So there is **no clean per-route ext_authz on the shared Gateway today**.
|
|
|
|
**Options (PM decision):**
|
|
|
|
| # | Approach | Pros | Cons |
|
|
|---|---|---|---|
|
|
| A | **Interim self-authored CEC** — a dedicated ext_authz HTTPS listener on its own L2 VIP (`forward-auth-cec.yaml`), protected hosts moved off the shared Gateway and DNS repointed. | Works on 1.16.5 **today**; per-route opt-in; reuses the wildcard TLS via SDS; **verified below**. | Hand-authored TLS+routes; a 2nd ingress VIP + DNS; per-app maintenance; not Gateway-API-native. |
|
|
| B | **oauth2-proxy in reverse-proxy mode per app** — app HTTPRoute `backendRef` -> a per-app oauth2-proxy (`--upstream=http://app.svc`). | Fully Gateway-API-native (just a backendRef swap); robust; no CEC. | One oauth2-proxy instance/config per app; more pods; not the single-proxy ext_authz model. |
|
|
| C | **Upgrade Cilium** to a release with native `ExternalAuth` (GEP-1494) and gate routes with an HTTPRoute filter. | Cleanest, truly per-route, Crafty-friendly. | Not released yet; large jump from 1.16.5; must keep the Ansible-bootstrap/Flux **parity contract** in lock-step. |
|
|
|
|
**Recommendation:** ship the oauth2-proxy hardening now (done); adopt **A** as the interim enforcement
|
|
for the sensitive UIs when the Phase-2 L2 VIP pool lands; track **C** as the long-term target and drop
|
|
A once released.
|
|
|
|
### Opt-in (per route)
|
|
|
|
A route is protected **only** by being listed as a virtualHost on the ext_authz listener — apps not
|
|
listed keep hitting the shared Gateway. To protect an app (pihole today; **Crafty next**): add its
|
|
Service + EDS cluster + virtualHost to `forward-auth-cec.yaml`, **remove its HTTPRoute from the shared
|
|
Gateway**, and repoint its DNS at the ext_authz VIP. The ServiceDefinition already declares
|
|
`auth.mode: forward-auth` + `groups: [platform-admins]` (rendered onto the HTTPRoute annotations) — the
|
|
opt-in signal. Ready to extend to Crafty and the other non-OIDC UIs (PiHole/*arr/qBittorrent/Homepage).
|
|
|
|
### Proof (in-cluster, cilium-envoy v1.30.8, 2026-08-28)
|
|
|
|
A throwaway standalone `cilium-envoy` pod ran the exact `ext_authz` stanza from `forward-auth-cec.yaml`
|
|
against the live oauth2-proxy Service, with a no-filter **control** route on the same backend:
|
|
|
|
- **Control** (`:8081`, no ext_authz) -> **200** `CONTROL-OK`.
|
|
- **Protected** (`:8080`, ext_authz), unauthenticated -> **403**; body is oauth2-proxy's sign-in page,
|
|
**not** the upstream (`UPSTREAM-OK` never served). oauth2-proxy's own access log recorded the
|
|
`/oauth2/auth` call from the envoy pod. `failure_mode_allow: false` confirmed fail-closed.
|
|
|
|
This proves the R-IDP-02 property (unauthenticated cannot reach the upstream) on this exact Envoy. The
|
|
**allow path** (valid `platform-admins` session -> 202 -> upstream) needs a real browser round-trip.
|
|
|
|
### Browser test (for the human, once `forward-auth-cec.yaml` is deployed)
|
|
|
|
1. Incognito -> `https://pihole.home.keithjackson.dev` -> expect a 302 to
|
|
`https://keycloak.home.keithjackson.dev/realms/homelab/...` (not the PiHole UI).
|
|
2. Log in as a **platform-admins** user (MFA/OTP enforced) -> land back on the PiHole UI.
|
|
3. Log in as a **non-platform-admins** user -> expect **403** (authenticated but not authorized).
|
|
4. Confirm the session cookie is `.home.keithjackson.dev`-scoped (single sign-on across the other
|
|
protected UIs).
|
|
|
|
## Observability: what this component can and cannot see (measured 2026-09-30)
|
|
|
|
`servicemonitor.yaml` scrapes **this namespace's** deployment and nothing else. On prod that is
|
|
two targets, `job="oauth2-proxy"`, both `up=1`. The complete set of oauth2-proxy-specific series
|
|
in the TSDB is:
|
|
|
|
count by (__name__) ({__name__=~"oauth2_proxy.*"})
|
|
oauth2_proxy_requests_in_flight => 2
|
|
|
|
**The per-app gate sidecars are not scraped, and there is nothing to scrape.** Six apps run an
|
|
`oauth2-proxy` container of their own — crafty, jackett, pihole, qbittorrent, radarr, sonarr —
|
|
each with `--http-address=0.0.0.0:4180` and **no `--metrics-address`**, so oauth2-proxy's metrics
|
|
listener never starts. They declare only a `http:4180` container port, no `metrics` port exists on
|
|
their Services, and `kubectl get servicemonitors,podmonitors -A` selects none of them. Any alert
|
|
written against `oauth2_proxy_requests_total{namespace="jackett"}` would be structurally inert.
|
|
|
|
This matters because the gate's characteristic failure is invisible by construction: an
|
|
unauthenticated **XHR** gets a `302` to Keycloak rather than a `401`, the browser follows it
|
|
against a live SSO session with no prompt, and the call resolves **200 carrying an HTML page**
|
|
where the app expected JSON. Nothing in the request path reports an error. On 2026-09-30 jackett
|
|
spent a burst of requests in exactly that state, re-established a session, and lost it 26 seconds
|
|
into a 5-minute session, while `jackett-down` sat at 1 replica available the whole time.
|
|
|
|
`auth-visibility-alerts.yaml` therefore alerts on **the gap**, not on the incident:
|
|
`OAuth2ProxyGateUnobserved` compares `kube_pod_container_info{container="oauth2-proxy"}` against
|
|
`up{container="oauth2-proxy"}` — two already-scraped families — and names every namespace running
|
|
a gate that nothing observes. It fires for six namespaces today and clears per-namespace as each
|
|
one gains a scrape. `runbooks/oauth2-proxy-auth-failures.md` is the by-hand substitute meanwhile,
|
|
and it reads the sidecar logs out of Loki (Alloy collects them; 72h retention).
|
|
|
|
**To close the gap for an app**, three things are needed together, on that app's own manifest:
|
|
`--metrics-address=0.0.0.0:44180` on the sidecar, a `metrics` containerPort plus the matching
|
|
Service port, and a ServiceMonitor selecting it. Only then does the auth-churn rule — sustained
|
|
`code="302"` as a share of `oauth2_proxy_requests_total`, held across a window longer than the
|
|
session lifetime — become writable rather than decorative.
|
|
|
|
### Update 2026-10-01: the listener exists; the scrape path does not
|
|
|
|
PR #28 added `--metrics-address=0.0.0.0:44180` and a named `metrics` container port to the five
|
|
fleet sidecars (jackett, radarr, sonarr, qbittorrent, pihole). `podmonitor.yaml` in this directory
|
|
discovers them by the pod label `homelab.keithjackson.dev/auth-mode: forward-auth` — a **PodMonitor**
|
|
rather than a ServiceMonitor because no Service exposes 44180 and adding one would mean changing the
|
|
renderer.
|
|
|
|
The endpoint is real: port-forwarded straight to the jackett pod, `/metrics` returns **http=200,
|
|
10208 bytes**, exporting `oauth2_proxy_requests_in_flight`, `oauth2_proxy_requests_total` (with
|
|
`code` labels) and `oauth2_proxy_response_duration_seconds`. That is verified by payload, not by the
|
|
flag appearing in `args` — a distinction worth keeping, because the two claims came apart here.
|
|
|
|
**The scrape is nevertheless dropped.** From a pod in ns `monitoring`, both `:44180` and `:4180` on
|
|
the jackett pod time out; the shared proxy's `:44180` answers. The only difference is that ns
|
|
`oauth2-proxy` has **no NetworkPolicy at all**, while each app namespace has a `default-deny` whose
|
|
`policyTypes` include `Ingress` with no `ingress` rules (deny-all) plus a single CNP admitting
|
|
`fromEntities: [ingress]` to port 4180 — the Gateway's Envoy and nothing else. Prometheus matches no
|
|
clause.
|
|
|
|
So the targets read `up == 0` and the TSDB is still empty for those namespaces. This is the same bug
|
|
and the same fingerprint as `platform/longhorn/networkpolicy-metrics.yaml` documents for Longhorn,
|
|
and the same narrow fix applies: ns `monitoring` → the gate pod label → TCP 44180 only. It cannot
|
|
ship from this directory, because `oauth2-proxy` and `apps-wave-0` are **siblings** in the Flux graph
|
|
(both `dependsOn: [platform-config]`), so a NetworkPolicy naming ns `jackett` would apply on today's
|
|
cluster and fail on a cold rebuild. It belongs with each app's own rendered policy.
|
|
|
|
Consequence for `auth-visibility-alerts.yaml`: its `unless` clause requires `up == 1`, not merely a
|
|
present `up` series. A discovered-but-failing target publishes `up` with value 0, so the bare form
|
|
would have cleared `OAuth2ProxyGateUnobserved` the moment the PodMonitor landed — reporting the gates
|
|
as observed while nothing was being scraped. The unit test pins that scenario; deleting the `== 1`
|
|
fails it and nothing else.
|