jakman85andClaude Opus 5
81f627167d
PR #49 (merged 21c88b0) fixed the blocked scrape and it is verified working on prod
2026-10-05: all 6 targets up == 1, health=up, lastError='', lastScrapeDuration 0.001-0.006s
against 10.0s and "context deadline exceeded" before. oauth2_proxy_* is in the TSDB for
these namespaces for the first time (5 metric names, 148 series).
Four places in the tree still asserted the pre-fix state as current, in the present tense,
in the files a reader consults first:
platform/oauth2-proxy/podmonitor.yaml "READ THIS BEFORE BELIEVING THE TARGETS ARE UP:
THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY" and "SO THE TARGETS BELOW WILL READ
up == 0 UNTIL AN INGRESS ALLOWANCE LANDS". The first actively instructs the reader to
disbelieve a correct up == 1. Now a STATUS block with the post-fix numbers, with the
diagnosis kept below it under an explicit HISTORY heading because it is why the fix is
shaped the way it is.
catalog/render.py:159 "THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED".
catalog/render.py:1365 the same claim on the sidecar arg.
platform/oauth2-proxy/README.md its subsections are dated and so are honest records, but
a reader hits the stale body long before any correction. A SUPERSEDED pointer now sits
under the heading and an Update 2026-10-05 section carries the before/after.
The "not scraped" claims are corrected; the "a flag alone is not enough" warning is KEPT
deliberately, because that part is still true and "flag without ingress allowance" is
exactly what caused the 63.7h outage.
Two substantive errors fixed while there, not just tense. The README's "three things are
needed" listed a Service port and a ServiceMonitor — wrong, a PodMonitor targets pod ports
directly, which is why no Service exposes 44180 and the renderer needed no Service change;
following it would have sent someone to edit Service rendering for nothing. And
podmonitor.yaml deferred the fix to "the catalog/renderer, in flight on another branch" —
no such branch existed; it now names r_monitoring_ingress_policy, with a new CRAFTY section
explaining the one namespace this file cannot help.
Comments only in render.py, so the rendered output is byte-identical: no apps/ file and no
golden changes, and therefore no sensitive path and no human approval needed — the opposite
of the PR this documents.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
216 lines
14 KiB
Markdown
216 lines
14 KiB
Markdown
# platform/oauth2-proxy
|
||
|
||
**oauth2-proxy** — forward-auth at the Gateway for apps **without usable OIDC**, authenticating
|
||
against Keycloak.
|
||
|
||
- **Governing ADR:** [ADR-0014 — IdP: Keycloak](../../../home-network-k3s/decisions/adr-0014-idp-keycloak.md)
|
||
- **Spec:** [docs/15 — Identity & Access](../../../home-network-k3s/docs/15-identity-and-access.md)
|
||
- **Serves:** R-IDP-01..06 (esp. R-IDP-02: management UIs never unauthenticated)
|
||
- **Owner:** WS-SEC (Gateway wiring shared with WS-NET)
|
||
|
||
## What this stub is
|
||
|
||
Flux `HelmRelease` for oauth2-proxy as the forward-auth provider for non-OIDC apps (PiHole, *arr UIs,
|
||
qBittorrent, Homepage, Prometheus, Alertmanager, Hubble UI — ADR-0014). Replica count is an overlay
|
||
knob (`${OAUTH2_PROXY_REPLICAS}`, R-ENV-02). Client/cookie secrets referenced **by name**,
|
||
**SOPS-encrypted** (WS-SEC).
|
||
|
||
## Contract (SSO/auth + Gateway)
|
||
|
||
Pairs with [`../keycloak`](../keycloak) to realize the "SSO/auth contract": a ServiceDefinition `auth:
|
||
forward-auth` service is protected by oauth2-proxy at the Gateway (ext-auth); `auth: oidc` services go
|
||
native to Keycloak. Gateway ext-auth wiring is a **WS-NET** contract.
|
||
|
||
## Enforcement (ext_authz forward-auth)
|
||
|
||
The **authenticator** (oauth2-proxy) is deployed and wired to Keycloak: `provider=oidc`,
|
||
`oidc-issuer-url=.../realms/homelab`, `--reverse-proxy`, `--upstream=static://202`,
|
||
`--set-xauthrequest`, cookie/redirect domain `.home.keithjackson.dev`, and — added here —
|
||
`--allowed-groups=platform-admins` (R-IDP-06 floor) + `--skip-provider-button` (302 straight to
|
||
Keycloak). The Keycloak `oauth2-proxy` client already carries the `auth.home…/oauth2/callback`
|
||
redirect URI and a `groups` claim with `full.path=false` (bare `platform-admins`) — no realm change
|
||
needed.
|
||
|
||
**Mechanism.** Forward-auth is an Envoy `ext_authz` HTTP filter that calls oauth2-proxy's
|
||
`/oauth2/auth` decision endpoint: **202** for a valid session in `platform-admins` (request passes to
|
||
the upstream, `X-Auth-Request-*` identity headers forwarded), **401/403** otherwise. With
|
||
`failure_mode_allow: false` the gateway **fails closed** — a denied or errored check never reaches the
|
||
upstream UI (R-IDP-02). The filter + oauth2-proxy cluster live in
|
||
[`forward-auth-cec.yaml`](./forward-auth-cec.yaml).
|
||
|
||
### Why this is NOT wired into the shared Gateway (Cilium 1.16.5 constraint)
|
||
|
||
The shared Gateway's Envoy program is a **machine-owned, constantly-regenerated**
|
||
`CiliumEnvoyConfig` (`gateway/cilium-gateway-shared`, `ownerReferences->Gateway`, regenerated on every
|
||
HTTPRoute change). Cilium 1.16.5 exposes **no supported hook** to inject an `ext_authz` filter into
|
||
that generated listener, and native Gateway-API **ExternalAuth (GEP-1494) ships in no released Cilium
|
||
version yet** (PoC only). So there is **no clean per-route ext_authz on the shared Gateway today**.
|
||
|
||
**Options (PM decision):**
|
||
|
||
| # | Approach | Pros | Cons |
|
||
|---|---|---|---|
|
||
| A | **Interim self-authored CEC** — a dedicated ext_authz HTTPS listener on its own L2 VIP (`forward-auth-cec.yaml`), protected hosts moved off the shared Gateway and DNS repointed. | Works on 1.16.5 **today**; per-route opt-in; reuses the wildcard TLS via SDS; **verified below**. | Hand-authored TLS+routes; a 2nd ingress VIP + DNS; per-app maintenance; not Gateway-API-native. |
|
||
| B | **oauth2-proxy in reverse-proxy mode per app** — app HTTPRoute `backendRef` -> a per-app oauth2-proxy (`--upstream=http://app.svc`). | Fully Gateway-API-native (just a backendRef swap); robust; no CEC. | One oauth2-proxy instance/config per app; more pods; not the single-proxy ext_authz model. |
|
||
| C | **Upgrade Cilium** to a release with native `ExternalAuth` (GEP-1494) and gate routes with an HTTPRoute filter. | Cleanest, truly per-route, Crafty-friendly. | Not released yet; large jump from 1.16.5; must keep the Ansible-bootstrap/Flux **parity contract** in lock-step. |
|
||
|
||
**Recommendation:** ship the oauth2-proxy hardening now (done); adopt **A** as the interim enforcement
|
||
for the sensitive UIs when the Phase-2 L2 VIP pool lands; track **C** as the long-term target and drop
|
||
A once released.
|
||
|
||
### Opt-in (per route)
|
||
|
||
A route is protected **only** by being listed as a virtualHost on the ext_authz listener — apps not
|
||
listed keep hitting the shared Gateway. To protect an app (pihole today; **Crafty next**): add its
|
||
Service + EDS cluster + virtualHost to `forward-auth-cec.yaml`, **remove its HTTPRoute from the shared
|
||
Gateway**, and repoint its DNS at the ext_authz VIP. The ServiceDefinition already declares
|
||
`auth.mode: forward-auth` + `groups: [platform-admins]` (rendered onto the HTTPRoute annotations) — the
|
||
opt-in signal. Ready to extend to Crafty and the other non-OIDC UIs (PiHole/*arr/qBittorrent/Homepage).
|
||
|
||
### Proof (in-cluster, cilium-envoy v1.30.8, 2026-08-28)
|
||
|
||
A throwaway standalone `cilium-envoy` pod ran the exact `ext_authz` stanza from `forward-auth-cec.yaml`
|
||
against the live oauth2-proxy Service, with a no-filter **control** route on the same backend:
|
||
|
||
- **Control** (`:8081`, no ext_authz) -> **200** `CONTROL-OK`.
|
||
- **Protected** (`:8080`, ext_authz), unauthenticated -> **403**; body is oauth2-proxy's sign-in page,
|
||
**not** the upstream (`UPSTREAM-OK` never served). oauth2-proxy's own access log recorded the
|
||
`/oauth2/auth` call from the envoy pod. `failure_mode_allow: false` confirmed fail-closed.
|
||
|
||
This proves the R-IDP-02 property (unauthenticated cannot reach the upstream) on this exact Envoy. The
|
||
**allow path** (valid `platform-admins` session -> 202 -> upstream) needs a real browser round-trip.
|
||
|
||
### Browser test (for the human, once `forward-auth-cec.yaml` is deployed)
|
||
|
||
1. Incognito -> `https://pihole.home.keithjackson.dev` -> expect a 302 to
|
||
`https://keycloak.home.keithjackson.dev/realms/homelab/...` (not the PiHole UI).
|
||
2. Log in as a **platform-admins** user (MFA/OTP enforced) -> land back on the PiHole UI.
|
||
3. Log in as a **non-platform-admins** user -> expect **403** (authenticated but not authorized).
|
||
4. Confirm the session cookie is `.home.keithjackson.dev`-scoped (single sign-on across the other
|
||
protected UIs).
|
||
|
||
## Observability: what this component can and cannot see (measured 2026-09-30)
|
||
|
||
> **SUPERSEDED — read "Update 2026-10-05" at the end of this section first.** The gate sidecars
|
||
> **are** scraped now and the TSDB **does** hold `oauth2_proxy_*` for them. The two subsections
|
||
> below are kept as the dated record of how this was diagnosed, not as current state.
|
||
|
||
`servicemonitor.yaml` scrapes **this namespace's** deployment and nothing else. On prod that is
|
||
two targets, `job="oauth2-proxy"`, both `up=1`. The complete set of oauth2-proxy-specific series
|
||
in the TSDB is:
|
||
|
||
count by (__name__) ({__name__=~"oauth2_proxy.*"})
|
||
oauth2_proxy_requests_in_flight => 2
|
||
|
||
**The per-app gate sidecars are not scraped, and there is nothing to scrape.** Six apps run an
|
||
`oauth2-proxy` container of their own — crafty, jackett, pihole, qbittorrent, radarr, sonarr —
|
||
each with `--http-address=0.0.0.0:4180` and **no `--metrics-address`**, so oauth2-proxy's metrics
|
||
listener never starts. They declare only a `http:4180` container port, no `metrics` port exists on
|
||
their Services, and `kubectl get servicemonitors,podmonitors -A` selects none of them. Any alert
|
||
written against `oauth2_proxy_requests_total{namespace="jackett"}` would be structurally inert.
|
||
|
||
This matters because the gate's characteristic failure is invisible by construction: an
|
||
unauthenticated **XHR** gets a `302` to Keycloak rather than a `401`, the browser follows it
|
||
against a live SSO session with no prompt, and the call resolves **200 carrying an HTML page**
|
||
where the app expected JSON. Nothing in the request path reports an error. On 2026-09-30 jackett
|
||
spent a burst of requests in exactly that state, re-established a session, and lost it 26 seconds
|
||
into a 5-minute session, while `jackett-down` sat at 1 replica available the whole time.
|
||
|
||
`auth-visibility-alerts.yaml` therefore alerts on **the gap**, not on the incident:
|
||
`OAuth2ProxyGateUnobserved` compares `kube_pod_container_info{container="oauth2-proxy"}` against
|
||
`up{container="oauth2-proxy"}` — two already-scraped families — and names every namespace running
|
||
a gate that nothing observes. It fires for six namespaces today and clears per-namespace as each
|
||
one gains a scrape. `runbooks/oauth2-proxy-auth-failures.md` is the by-hand substitute meanwhile,
|
||
and it reads the sidecar logs out of Loki (Alloy collects them; 72h retention).
|
||
|
||
**To close the gap for an app**, three things are needed together, on that app's own manifest:
|
||
`--metrics-address=0.0.0.0:44180` on the sidecar, a `metrics` containerPort plus the matching
|
||
Service port, and a ServiceMonitor selecting it. Only then does the auth-churn rule — sustained
|
||
`code="302"` as a share of `oauth2_proxy_requests_total`, held across a window longer than the
|
||
session lifetime — become writable rather than decorative.
|
||
|
||
### Update 2026-10-01: the listener exists; the scrape path does not
|
||
|
||
PR #28 added `--metrics-address=0.0.0.0:44180` and a named `metrics` container port to the five
|
||
fleet sidecars (jackett, radarr, sonarr, qbittorrent, pihole). `podmonitor.yaml` in this directory
|
||
discovers them by the pod label `homelab.keithjackson.dev/auth-mode: forward-auth` — a **PodMonitor**
|
||
rather than a ServiceMonitor because no Service exposes 44180 and adding one would mean changing the
|
||
renderer.
|
||
|
||
The endpoint is real: port-forwarded straight to the jackett pod, `/metrics` returns **http=200,
|
||
10208 bytes**, exporting `oauth2_proxy_requests_in_flight`, `oauth2_proxy_requests_total` (with
|
||
`code` labels) and `oauth2_proxy_response_duration_seconds`. That is verified by payload, not by the
|
||
flag appearing in `args` — a distinction worth keeping, because the two claims came apart here.
|
||
|
||
**The scrape is nevertheless dropped.** From a pod in ns `monitoring`, both `:44180` and `:4180` on
|
||
the jackett pod time out; the shared proxy's `:44180` answers. The only difference is that ns
|
||
`oauth2-proxy` has **no NetworkPolicy at all**, while each app namespace has a `default-deny` whose
|
||
`policyTypes` include `Ingress` with no `ingress` rules (deny-all) plus a single CNP admitting
|
||
`fromEntities: [ingress]` to port 4180 — the Gateway's Envoy and nothing else. Prometheus matches no
|
||
clause.
|
||
|
||
So the targets read `up == 0` and the TSDB is still empty for those namespaces. This is the same bug
|
||
and the same fingerprint as `platform/longhorn/networkpolicy-metrics.yaml` documents for Longhorn,
|
||
and the same narrow fix applies: ns `monitoring` → the gate pod label → TCP 44180 only. It cannot
|
||
ship from this directory, because `oauth2-proxy` and `apps-wave-0` are **siblings** in the Flux graph
|
||
(both `dependsOn: [platform-config]`), so a NetworkPolicy naming ns `jackett` would apply on today's
|
||
cluster and fail on a cold rebuild. It belongs with each app's own rendered policy.
|
||
|
||
Consequence for `auth-visibility-alerts.yaml`: its `unless` clause requires `up == 1`, not merely a
|
||
present `up` series. A discovered-but-failing target publishes `up` with value 0, so the bare form
|
||
would have cleared `OAuth2ProxyGateUnobserved` the moment the PodMonitor landed — reporting the gates
|
||
as observed while nothing was being scraped. The unit test pins that scenario; deleting the `== 1`
|
||
fails it and nothing else.
|
||
|
||
### Update 2026-10-05: the scrape works
|
||
|
||
PR #49 (merged `21c88b0`) added the missing third piece. `catalog/render.py`'s
|
||
`r_monitoring_ingress_policy` now also fires for the gate sidecar's metrics port, keyed on
|
||
`_needs_oauth2_sidecar`, rendering `<app>-allow-monitoring-scrape` per app: ns `monitoring` → that
|
||
app's `endpointSelector` → TCP 44180 only. Five objects, one each for jackett, pihole, qbittorrent,
|
||
radarr, sonarr.
|
||
|
||
Measured on prod 2026-10-05 after Flux applied it (`apps-wave-0` 21:37:01Z, `apps-wave-1`
|
||
21:37:46Z):
|
||
|
||
| | before | after |
|
||
|---|---|---|
|
||
| target health (6) | `down` | `up` |
|
||
| `lastError` | `context deadline exceeded` | `''` |
|
||
| `lastScrapeDuration` | **10.0s** — exactly the `scrapeTimeout` | **0.001–0.006s** |
|
||
| `up` | `0` × 6 | `1` × 6 |
|
||
|
||
count by (__name__) ({__name__=~"oauth2_proxy_.*",
|
||
namespace=~"jackett|radarr|sonarr|qbittorrent|pihole"})
|
||
oauth2_proxy_requests_in_flight => 6
|
||
oauth2_proxy_requests_total => 16
|
||
oauth2_proxy_response_duration_seconds_bucket => 108
|
||
oauth2_proxy_response_duration_seconds_sum => 9
|
||
oauth2_proxy_response_duration_seconds_count => 9
|
||
|
||
sum by (namespace) (oauth2_proxy_requests_total)
|
||
qbittorrent => 35072 radarr => 23560 sonarr => 1284
|
||
|
||
`oauth2_proxy_requests_total` is absent for jackett and pihole simply because that counter is not
|
||
created until the gate has served its first request.
|
||
|
||
**So the auth-churn rule is now writable rather than decorative** — sustained `code="302"` as a
|
||
share of `oauth2_proxy_requests_total`, held across a window longer than the session lifetime. That
|
||
is the rule the 2026-09-30 incident above could not be caught by. It is still unwritten.
|
||
|
||
**Alert outcome:** all 5 `TargetDown` resolved, and 5 of 6 `OAuth2ProxyGateUnobserved` with them.
|
||
At `repeat_interval: 12h` with `group_by: [alertname, namespace]`, those 11 groups had been
|
||
re-notifying at roughly **22 emails/day** since 2026-10-02; that drops to 2/day.
|
||
|
||
**`crafty` still fires, and that is correct.** It is a hand-maintained Deployment under
|
||
`platform/crafty`, outside the renderer, whose `oauth2-proxy` container has **no
|
||
`--metrics-address` at all** — no listener, so no port to open and nothing for the PodMonitor's
|
||
pod-label selector to match. Closing it needs the flag and a `metrics` containerPort on crafty's
|
||
own manifest first, then a hand-written allowance beside it. Open follow-up work.
|
||
|
||
**The paragraph above headed "To close the gap for an app" is superseded**: a *Service* port and a
|
||
ServiceMonitor are **not** required. A PodMonitor targets pod ports directly, which is why no
|
||
Service exposes 44180 and why the renderer needed no Service change. The three things actually
|
||
needed are `--metrics-address` + named container port, a PodMonitor, and the monitoring-ingress
|
||
allowance.
|