Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
  jakman85andClaude Opus 5
|
1e71b578d5 | ||
|
jakman85andClaude Opus 5
|
bce101d03f | ||
|
jakman85
|
ba79a1b59b | ||
|
jakman85
|
cdf3e14e27 | ||
|
jakman85
|
41a20dfd20 | ||
|
jakman85andClaude Opus 5
|
df1fd1b3d1 | ||
|
jakman85andClaude Opus 5
|
0fee584e3a | ||
|
jakman85andClaude Opus 5
|
fc7ec01628 |
No files matched your search
@@ -143,6 +143,8 @@ spec:
|
||||
- name: media
|
||||
mountPath: /media
|
||||
readOnly: true
|
||||
- name: dri
|
||||
mountPath: /dev/dri
|
||||
volumes:
|
||||
- name: config
|
||||
persistentVolumeClaim:
|
||||
@@ -150,6 +152,10 @@ spec:
|
||||
- name: media
|
||||
persistentVolumeClaim:
|
||||
claimName: plex-media
|
||||
- name: dri
|
||||
hostPath:
|
||||
path: /dev/dri
|
||||
type: DirectoryOrCreate
|
||||
---
|
||||
# Service
|
||||
apiVersion: v1
|
||||
|
||||
@@ -80,6 +80,58 @@ env:
|
||||
# remote access are UNVERIFIED. Advertising a wrong/internal-only URL breaks remote access for ALL
|
||||
# clients, so per "prefer safety over completeness" it is left unset. FOLLOW-UP: once remote
|
||||
# reachability through the Gateway is confirmed, set ADVERTISE_IP: "https://plex.home.keithjackson.dev:443".
|
||||
# HARDWARE TRANSCODING — the host iGPU's render device (schema 1.8.0 `hardware`; R-PLAT-02).
|
||||
# INTENT ONLY. This says "this workload needs an Intel Quick Sync render device"; it does NOT name a
|
||||
# device path. The renderer currently satisfies it with exactly two things — a `/dev/dri` hostPath
|
||||
# volume (`type: DirectoryOrCreate`) and a read-write `/dev/dri` volumeMount on the app container —
|
||||
# and nothing else: no nodeSelector, no privileged, no resources (schemas/renderer-contract.md
|
||||
# `### hardware`). If that mechanism is later replaced by the Intel GPU device plugin
|
||||
# (`gpu.intel.com/i915`), it is a `catalog/render.py` change and THIS BLOCK DOES NOT MOVE. Do not
|
||||
# "simplify" it into a raw `/dev/...` path; that is the whole point of the enum.
|
||||
#
|
||||
# THE DEVICE IS PRESENT ON EVERY PROD NODE, so nothing had to be installed and Plex can land
|
||||
# anywhere. Measured on all three nodes 2026-09-30: `/dev/dri/renderD128` present, `root:render`,
|
||||
# GID 992, mode 0660, driver `i915` bound (NOT `xe`), UHD Graphics 630 on an i7-9700T, media engines
|
||||
# `vcs0`/`vecs0` live, iGPU enabled in BIOS on each. Plex happens to run on k3s-prod-2 today, but
|
||||
# that is scheduler history rather than a requirement — this field deliberately renders no
|
||||
# nodeSelector, because placement is a separate concern with its own justification.
|
||||
#
|
||||
# NO `podSecurityContext.supplementalGroups`, AND ADDING ONE WOULD BE INERT — do not "fix" this.
|
||||
# The device is `root:render` 0660 and this process is uid/gid 1000, so the obvious move is to push
|
||||
# the host `render` GID in as a supplemental group. It cannot work here. MEASURED IN THE RUNNING
|
||||
# CONTAINER, 2026-09-30: the image's own
|
||||
# `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` stats every char device under
|
||||
# `/dev/dri`, `groupadd`s a matching group INSIDE the container when none exists, and
|
||||
# `usermod -a -G`s the `plex` user into it — every start, nothing hardcoded, which also makes a host
|
||||
# GID renumbering self-healing (no GID is pinned anywhere in this repo). The service then execs via
|
||||
# `s6-setuidgid plex`, and `s6-setuidgid` REPLACES the process group list from `/etc/group` rather
|
||||
# than appending to it: `Plex Media Server` runs with `Groups: 100 1000`, and root's gid `0` — which
|
||||
# PID 1 (`s6-svscan`) does hold — is GONE. Anything Kubernetes supplies therefore reaches s6-svscan
|
||||
# and is discarded before Plex ever starts. Mounting the device is sufficient; the group is the
|
||||
# image's job and it does it correctly. `render.py --selftest` asserts the ABSENCE of a pod
|
||||
# securityContext for this field precisely so nobody adds an inert one back later.
|
||||
#
|
||||
# `DirectoryOrCreate` IS THE SAFE FAILURE MODE, not laziness. A node that comes back without the
|
||||
# device (rebuild, BIOS reset, driver regression, hardware swap) gets an empty directory and Plex
|
||||
# degrades to SOFTWARE transcoding. `Directory` would fail the mount instead — and because
|
||||
# `apps-wave-0` is `wait: true` with `apps-wave-1` `dependsOn` it, that would stall the WHOLE app
|
||||
# tier for a feature that is not load-bearing. The price, stated plainly: a missing device is
|
||||
# SILENT. This SD renders no `metricsEndpoint`, so there is no ServiceMonitor, no transcode-mode
|
||||
# metric, and no alert that could ever fire. The only proof is an operator reading the Plex
|
||||
# dashboard.
|
||||
#
|
||||
# THE LAST MILE IS A MANUAL OPERATOR STEP AND CANNOT LIVE IN THIS REPO. Mounting the device does
|
||||
# not enable hardware transcoding — Plex's own `HardwareAcceleratedCodecs` preference does, and it
|
||||
# is ABSENT from `Preferences.xml` today (the toggle has never been on). NO environment variable can
|
||||
# set it: a grep of the running image's entire init surface (`/etc/cont-init.d`, `/etc/services.d`,
|
||||
# `/etc/fix-attrs.d`, `/plex-common.sh`, `/usr/local/bin`) found ZERO `PLEX_PREFERENCE*` support,
|
||||
# and the six prefs the image does write (`MachineIdentifier`, `ProcessedMachineIdentifier`,
|
||||
# `PlexOnlineToken`, `customConnections`, `allowedNetworks`, `TranscoderTempDirectory`) are all
|
||||
# unrelated. So there is no GitOps path to the toggle; an operator clicks it once in the web UI.
|
||||
# `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took and the
|
||||
# device-cgroup failure mode to expect first.
|
||||
hardware:
|
||||
gpu: intel-quicksync
|
||||
exposure: public # edge port-forward + public DNS; clients + Seerr reach it (docs/09; R-DNS/R-NET)
|
||||
egress: internet # normal WAN egress (docs/03; R-NET-05)
|
||||
storage:
|
||||
|
||||
@@ -129,6 +129,22 @@ KNOWN_EXTERNAL_MONITORS: dict[str, str] = {
|
||||
"a PodMonitor, so it resolves against POD ports, not Service ports — the Flux controller "
|
||||
"Deployments (and their `http-prom` container port) are installed by the Flux bootstrap "
|
||||
"manifests, not authored here. Live on prod: job flux-system/flux-system, 6 targets up.",
|
||||
"platform/oauth2-proxy/podmonitor.yaml:oauth2-proxy-sidecars":
|
||||
"a PodMonitor, so it resolves against POD ports, not Service ports. Unlike the other "
|
||||
"entries here the target is NOT chart-owned — the `metrics` (44180) container port is "
|
||||
"authored in this repo, on the oauth2-proxy sidecar in apps/wave-0/{jackett,qbittorrent,"
|
||||
"pihole}.yaml and apps/wave-1/{radarr,sonarr}.yaml. It is unauditable here because NO "
|
||||
"Service exposes 44180 BY DESIGN: adding one would mean changing catalog/render.py and "
|
||||
"every apps/wave-* render and golden, to publish an endpoint only Prometheus should dial. "
|
||||
"A PodMonitor needs no Service, which is the whole reason it was chosen. What assertion "
|
||||
"(2) would have checked was instead MEASURED on prod 2026-10-01, read-only: the pods "
|
||||
"declare ports [('http', 4180), ('metrics', 44180)], the selector label "
|
||||
"homelab.keithjackson.dev/auth-mode=forward-auth is on exactly those five gate pods, and "
|
||||
"port-forwarding 44180 returns http=200 with 10208 bytes of oauth2_proxy_* metrics. "
|
||||
"CAVEAT, deliberately recorded: those targets are DOWN on prod, because the app "
|
||||
"namespaces' default-deny NetworkPolicy drops the scrape from ns monitoring (same bug and "
|
||||
"fingerprint as platform/longhorn/networkpolicy-metrics.yaml documents). The port name "
|
||||
"resolves; the network path does not yet.",
|
||||
}
|
||||
|
||||
MONITOR_KINDS = ("ServiceMonitor", "PodMonitor")
|
||||
|
||||
@@ -0,0 +1,289 @@
|
||||
---
|
||||
date: 2026-09-30
|
||||
branch: feat/oauth2-proxy-auth-failure-visible
|
||||
title: The Jackett auth failure was invisible because the gates emit nothing — alert on the gap, not on a metric that does not exist
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Jackett's UI broke in the way that costs the most: **HTTP 200 and a plausible page**. The
|
||||
operator's report named the property that matters — *"reloading the page never fixes the issue and
|
||||
the user is never re-prompted for login"*. Two sibling branches fix the auth defects. This one is
|
||||
about the fact that **no signal existed anywhere**, and it turns out that is not an oversight in
|
||||
the alert rules. It is an absence of data.
|
||||
|
||||
**The measurement that decided the whole design** (prod, live Prometheus, 2026-09-30):
|
||||
|
||||
count by (__name__) ({__name__=~"oauth2_proxy.*"})
|
||||
oauth2_proxy_requests_in_flight => 2
|
||||
|
||||
Two series. Both from the **shared** forward-auth Deployment in the `oauth2-proxy` namespace.
|
||||
That is every oauth2-proxy metric this cluster holds. The six per-app gate sidecars — crafty,
|
||||
jackett, pihole, qbittorrent, radarr, sonarr — publish **nothing at all**: they run with
|
||||
`--http-address=0.0.0.0:4180` and no `--metrics-address`, so oauth2-proxy's metrics listener never
|
||||
starts; they declare only a `http:4180` container port; and no ServiceMonitor or PodMonitor in the
|
||||
cluster selects them.
|
||||
|
||||
So the alert the brief asked for — *"a gated app is serving a sustained rate of auth redirects to
|
||||
a user who is authenticated"* — **is not writable**, and writing it anyway would have produced
|
||||
exactly the artefact this repo keeps getting burned by: a rule that is present, evaluating,
|
||||
permanently silent, and indistinguishable from health. It was not written.
|
||||
|
||||
What was written instead alerts on the condition that made the incident invisible, from families
|
||||
that **are** scraped:
|
||||
|
||||
- **`platform/oauth2-proxy/auth-visibility-alerts.yaml`** — `OAuth2ProxyGateUnobserved`:
|
||||
|
||||
count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
|
||||
unless
|
||||
count by (namespace) (up{container="oauth2-proxy"})
|
||||
|
||||
`for: 1h`, `severity: warning`. `kube_pod_container_info` is kube-state-metrics (a declared
|
||||
owner in `alert-metrics-have-scrape-targets`) and sees the container whether or not it exports
|
||||
anything; `up` is UNIVERSAL and carries a `container` label for any scrape target, so the rule
|
||||
needs no advance knowledge of what job name a future per-app scrape will use. Nothing in it
|
||||
touches `oauth2_proxy_*`, so it cannot be inert for the reason it is reporting. **It returns six
|
||||
namespaces against live prod right now** and excludes `oauth2-proxy`, which is scraped. It
|
||||
clears per-namespace as each gate gains a real scrape.
|
||||
- **`platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml`** — four states pinned:
|
||||
fires for an unscraped namespace, never for the scraped one, honours `for: 1h` (pending at 59m),
|
||||
and **stops** once a scrape appears under a *different* job name. Series shapes copied off live
|
||||
prod, not invented.
|
||||
- **`runbooks/oauth2-proxy-auth-failures.md`** — how to tell "the gate refused me" from "the group
|
||||
check refused me" from "the app refused me" from "the XHR silently got HTML", with the
|
||||
2026-09-30 jackett trace as the worked example. Registered in `runbooks/kustomization.yaml`'s
|
||||
`configMapGenerator` and listed in `runbooks/README.md`.
|
||||
- **`runbooks/README.md`** — a new row in *"Alerts you should not trust the silence of"* for the
|
||||
per-app sidecars, plus the measured reason.
|
||||
- **`platform/oauth2-proxy/README.md`** — what this component can and cannot see, and the exact
|
||||
three-part change that would close the gap.
|
||||
|
||||
**Why the obvious alternative was rejected.** An alert on the *shared* proxy's
|
||||
`oauth2_proxy_requests_total{code="302"}` ratio is the one a reader expects to see here. It would
|
||||
need a new `SCRAPE_OWNERS` entry in `conformance/**` (a sensitive path needing human approval),
|
||||
and — decisively — **it would not have fired on this incident**: the shared instance is not in the
|
||||
data path for any sidecar-gated app, and its current pods export no `oauth2_proxy_requests_total`
|
||||
at all, having proxied nothing since they started (the counter exists in the TSDB only from an
|
||||
older ReplicaSet, `oauth2-proxy-78449b5657-*`, with `code` labels `302` and `401` and no `202`).
|
||||
Shipping it would have looked like coverage of the reported failure and been none.
|
||||
|
||||
## What I verified
|
||||
|
||||
Everything below was measured against live prod through the pinned container, read-only. No
|
||||
`apply`, no `edit`, no `patch`, no `flux reconcile`.
|
||||
|
||||
- **The full oauth2-proxy metric inventory.** `count by (__name__) ({__name__=~"oauth2_proxy.*"})`
|
||||
→ `oauth2_proxy_requests_in_flight => 2`. `count by (__name__) ({job="oauth2-proxy"})` returns
|
||||
go_*/process_*/promhttp_* plus that one family and nothing else. **Only the shared instance is
|
||||
scraped**: `up{job="oauth2-proxy"}` → 2 targets, `10.0.1.189:44180` and `10.0.2.47:44180`, both
|
||||
`1`, `namespace="oauth2-proxy"`.
|
||||
- **The per-app sidecars are unscraped, and unscrapable as configured.** Every pod carrying an
|
||||
`oauth2-proxy` container, with its declared ports: the two shared replicas have
|
||||
`[('http',4180),('metrics',44180)]`; crafty, jackett, pihole, qbittorrent, radarr and sonarr all
|
||||
have `[('http',4180)]` only. The live jackett sidecar's args contain no `--metrics-address`.
|
||||
`kubectl get servicemonitors,podmonitors -A` lists 16 ServiceMonitors and 5 PodMonitors; the
|
||||
only oauth2-proxy entry is `oauth2-proxy/oauth2-proxy`.
|
||||
- **No existing alert is inert *for this reason*, because no alert references these metrics at
|
||||
all.** `grep -rn oauth2_proxy --include=*.yaml` over the tree hits only cookie-name strings in
|
||||
`platform/crafty/`. The 39 live PrometheusRules include `platform-health-identity`, whose four
|
||||
rules are about Keycloak readiness and the client reconciler and are all keyed on `kube_*`
|
||||
(scraped, fine). **The inertness here is the absence of a rule, not a broken one** — which is
|
||||
why the runbook README's inert-alerts table gains a row phrased like the etcd one.
|
||||
- **Keycloak cannot express it either.** `count by (__name__) ({__name__=~"keycloak_.*"})` →
|
||||
empty, and empty over `[30d]`. The `keycloak-service` scrape yields JVM/Agroal/JGroups metrics
|
||||
only; there are no user-event or login counters, so "a burst of re-logins for one client" is not
|
||||
observable from the IdP side.
|
||||
- **`oauth2_proxy_requests_total` has existed historically but not on the current pods.**
|
||||
`count_over_time(oauth2_proxy_requests_total[30d])` returns three series, all from
|
||||
`oauth2-proxy-78449b5657-*` (a prior ReplicaSet): `code="302"` on both replicas and `code="401"`
|
||||
on one, ~634 samples each (~5h of a 30d window). **No `code="202"` ever.**
|
||||
- **The alert expression fires against live data.** Run verbatim on prod:
|
||||
`count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by
|
||||
(namespace) (up{container="oauth2-proxy"})` → `pihole, jackett, crafty, qbittorrent, sonarr,
|
||||
radarr`, value 1 each; `oauth2-proxy` correctly absent.
|
||||
- **The log signal exists and is collected.** Loki (`loki-0`, SingleBinary) and a 3-node Alloy
|
||||
DaemonSet are running, and Alloy labels pod logs with `namespace`/`container`. LogQL over 24h,
|
||||
`{container="oauth2-proxy"}`: crafty 12036 lines, jackett 11669, oauth2-proxy 34559, pihole
|
||||
11520, qbittorrent 11520, radarr 11763, sonarr 11520. Filtered to `|= "No valid authentication
|
||||
in request"`: **crafty 81, jackett 39, radarr 67**; to `|= "AuthSuccess"`: **crafty 1, jackett
|
||||
4, radarr 1**. Those ratios (81:1, 10:1, 67:1) are the incident's signature and are in the
|
||||
runbook as the thing to compare against.
|
||||
- **Why `kubectl logs` hid it.** The jackett sidecar's stdout is ~11.5k lines/day and almost all
|
||||
of it is `GET "/ping" ... 200 2` from the kubelet probe, twice every 15s. A `--tail=40` shows
|
||||
forty `/ping` lines and looks perfectly healthy. The runbook's commands all filter.
|
||||
- **Runbook registration, both halves.** `- oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md`
|
||||
is in `runbooks/kustomization.yaml`'s `configMapGenerator.files`, and the page is listed under
|
||||
*Identity* in `runbooks/README.md`. The page clears the L0 quality floor (the four operator
|
||||
sections, >1500 bytes, mentions `dev.sh` and `--context k3s-prod`), and the alert's
|
||||
`runbook_url` points at exactly that name.
|
||||
- **Gate:** run in the pinned toolbox from this worktree with the `ENGINE` shim. Exit code and
|
||||
per-stage outcomes in the next section — read them there, not here.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **The alert has never fired in Alertmanager.** Nothing is applied; the PrometheusRule exists
|
||||
only in git on this branch. What is proven is that its expression returns the expected vector
|
||||
against live prod data and that promtool agrees about its behaviour over synthetic series.
|
||||
**Merging is what deploys it.**
|
||||
- **`OAuth2ProxyGateUnobserved` would NOT have fired *during* the 2026-09-30 jackett incident in
|
||||
any useful sense** — it describes a standing condition that has been true since the sidecars
|
||||
were deployed, so it would have been firing before, during and after. That is deliberate and it
|
||||
is the honest claim: it is a **gap tracker**, not an incident detector. It would have meant that
|
||||
on 2026-09-30 an operator asking "why did nothing alert?" had the answer already on the board
|
||||
instead of discovering it during the investigation. **No alert on this platform could have
|
||||
detected the incident itself, and none can today.**
|
||||
- **The alert that the brief actually asked for does not exist, and cannot be written yet.** The
|
||||
sustained-redirect-ratio rule needs `oauth2_proxy_requests_total` from the *per-app* sidecars.
|
||||
**Enabling that requires editing files owned by the two sibling branches** — the sidecar args in
|
||||
`apps/wave-0/jackett.yaml` (Mira Vance) and `apps/wave-0/{qbittorrent,pihole}.yaml`,
|
||||
`apps/wave-1/{radarr,sonarr}.yaml` (Theo Bright) — so it was **not attempted here**. The change
|
||||
is three things per app, together: `--metrics-address=0.0.0.0:44180`, a `metrics` containerPort
|
||||
plus Service port, and a ServiceMonitor. Once any one app has them, the rule belongs beside
|
||||
`OAuth2ProxyGateUnobserved`, and that app's namespace drops out of it automatically.
|
||||
- **The Loki alerting path is inert and I did not fix it.** Loki *has* a ruler stanza and the
|
||||
chart's `loki-sc-rules` sidecar is running with `LABEL=loki_rule`, `FOLDER=/rules` — so a
|
||||
ConfigMap-shipped LogQL rule would be delivered. But `GET /loki/api/v1/rules` answers **"unable
|
||||
to read rule dir /var/loki/rules/fake: no such file or directory"** (the ruler reads a different
|
||||
directory from the one the sidecar writes), and the rendered config contains **no
|
||||
`alertmanager_url`** — `grep -i 'alertmanager\|rule_path'` over the `loki` ConfigMap returns
|
||||
nothing. So a LogQL alert on `|= "No valid authentication in request"` — which *would* have
|
||||
caught this incident, and is the single highest-value follow-up here — **would evaluate and
|
||||
notify nobody**. Wiring it means `rulerConfig` (alertmanager_url + a matching rules directory)
|
||||
in `platform/monitoring/loki.yaml`, which is not a file this brief gave me. **I did not ship an
|
||||
inert rule ConfigMap to look productive.**
|
||||
- **Nothing was reproduced in a browser.** The 302-to-200-with-HTML mechanism is read off the
|
||||
sidecar access log and the response sizes (`/UI/Dashboard 200 44379`), not observed in devtools.
|
||||
- **Loki retention is 72h** (`platform/monitoring/loki.yaml`), so the 24h ratios above are a
|
||||
snapshot; anything older than three days is already gone. This is stated in the runbook because
|
||||
it bounds what an operator can still investigate.
|
||||
- **The gate's L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), so no runtime
|
||||
requirement touched by this branch is certified. See the exit code below — **rc 3 is
|
||||
UNDETERMINED, not green.**
|
||||
|
||||
### Follow-ups, in order of value
|
||||
|
||||
1. **Wire the Loki ruler to Alertmanager** (`platform/monitoring/loki.yaml`). This is the only
|
||||
route to an alert that would have caught the actual 2026-09-30 failure, and the data is
|
||||
already being collected. The rule is one expression:
|
||||
`sum by (namespace) (rate({container="oauth2-proxy"} |= "No valid authentication in request" [15m]))
|
||||
> 5 * sum by (namespace) (rate({container="oauth2-proxy"} |= "AuthSuccess" [15m]))`.
|
||||
2. **Give one gate sidecar a metrics endpoint** and watch its namespace drop out of
|
||||
`OAuth2ProxyGateUnobserved`. That is the cheapest proof the alert is a tracker and not a nag,
|
||||
and it unlocks the ratio rule in Prometheus.
|
||||
3. **Consider whether `--set-xauthrequest` apps should get `--api-routes`** so API paths answer
|
||||
401 instead of 302. That is the fix that would make this failure *loud* rather than merely
|
||||
observable, and it belongs with whoever owns the sidecar args.
|
||||
|
||||
---
|
||||
|
||||
# Round 2 — 2026-10-01: the scrape got wired, and found a bug in this branch's own alert
|
||||
|
||||
Added to the same PR (#29) rather than a fourth concurrent branch. PR #28 merged in the meantime
|
||||
and `main` moved three PRs ahead; `origin/main` is merged in here, and the `runbooks/kustomization.yaml`
|
||||
conflict was resolved by **keeping both** runbook entries (mine and `oauth2-proxy-session`).
|
||||
|
||||
## What changed
|
||||
|
||||
**`platform/oauth2-proxy/podmonitor.yaml` (new).** PR #28 gave the five fleet sidecars
|
||||
`--metrics-address=0.0.0.0:44180` and a named `metrics` port. No Service exposes 44180 on any of
|
||||
them, so a ServiceMonitor would have nothing to select — a **PodMonitor** targets pod ports directly
|
||||
and needs no renderer change. Selects on the pod label `homelab.keithjackson.dev/auth-mode:
|
||||
forward-auth`, which is on exactly the five fleet gate pods and absent from crafty's: the property,
|
||||
not an app list, so a sixth forward-auth app needs no edit.
|
||||
|
||||
**`auth-visibility-alerts.yaml`: `up{container="oauth2-proxy"}` → `up{container="oauth2-proxy"} == 1`.
|
||||
This is a real bug fix, and it is the most important thing in this round.** A target that is
|
||||
discovered but failing still publishes `up`, with value **0**. The series exists, so
|
||||
`count by (namespace)` returned 1, `unless` removed the namespace, and `OAuth2ProxyGateUnobserved`
|
||||
would have **gone quiet while nothing was being scraped at all** — this branch's own disease,
|
||||
reproduced inside the alert written to cure it. Not hypothetical: the PodMonitor lands five targets
|
||||
whose scrape is dropped, so the bare form would have silently cleared on this very commit.
|
||||
|
||||
**Two new unit-test scenarios** pinning that: a discovered-but-failing target must still fire, and a
|
||||
target that works and then breaks must re-fire rather than latch cleared.
|
||||
|
||||
**Docs corrected rather than appended to.** The 2026-09-30 claim "the sidecars export no metrics at
|
||||
all" is now only half true and was rewritten in `runbooks/README.md`, `platform/oauth2-proxy/README.md`
|
||||
and the alert header, each showing the two-step history, because each step looked like the end of it.
|
||||
The runbook gains the policy-vs-no-target triage (step 1b) and the **radarr `/signalr` worked case**.
|
||||
|
||||
**Deliberately NOT shipped: the ratio alert.** `oauth2_proxy_requests_total` now exists on the
|
||||
endpoint, but not in the TSDB, because the scrape is blocked. Writing the ratio rule today would make
|
||||
it inert for precisely the reason it was refused on 2026-09-30. It becomes writable the day a scrape
|
||||
succeeds, not the day a listener exists.
|
||||
|
||||
## What I verified
|
||||
|
||||
- **The endpoint serves — by payload, not by flag.** Port-forwarded to the jackett pod's 44180:
|
||||
`http=200 bytes=10208`, exporting `oauth2_proxy_requests_in_flight`, `oauth2_proxy_requests_total`
|
||||
(`code="200" 3`, `code="302" 2`) and `oauth2_proxy_response_duration_seconds`. `--metrics-address`
|
||||
being in `args` and `/metrics` returning a body are different claims; both now hold.
|
||||
- **The scrape path is DROPPED, measured from the right place.** From grafana in ns `monitoring`
|
||||
(beside Prometheus, same policy position) to the jackett pod IP: `:44180` → `wget: download timed
|
||||
out`; `:4180` → **also** timed out; the shared proxy's `:44180` → 200 with `oauth2_proxy_*` present.
|
||||
Cause, read from the objects: ns jackett's `default-deny` is `podSelector: {}`,
|
||||
`policyTypes: [Ingress, Egress]`, with an `egress` rule and **no `ingress` key** (deny-all ingress);
|
||||
the only allowance is CNP `jackett-allow-gateway-ingress`, `fromEntities: [ingress]` to port 4180.
|
||||
The shared proxy is scrapable only because ns `oauth2-proxy` has **no NetworkPolicy at all**.
|
||||
- **No Service exposes 44180** on any of the five (enumerated).
|
||||
- **The pod label selector is exact:** `auth-mode=forward-auth` on all five fleet gate pods, absent
|
||||
from crafty's. Crafty's sidecar still has no `--metrics-address` (args read in full).
|
||||
- **Flux ordering forbids shipping the fix from here:** `oauth2-proxy dependsOn=['platform-config']`
|
||||
and `apps-wave-0 dependsOn=['platform-config']` — siblings, no ordering. A NetworkPolicy naming ns
|
||||
`jackett` from this tier would apply today and fail on a cold rebuild.
|
||||
- **The `== 1` guard is non-vacuous — proven by mutation.** Reverting it to the bare `up` makes the
|
||||
two new scenarios FAIL (`got: nil` at 1h5m and 2h5m) and leaves every other assertion passing;
|
||||
restoring it returns the suite to SUCCESS under pinned promtool v3.14.0.
|
||||
- **Gate:** see the next section for the exit code and skips.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **The PodMonitor has never been applied.** That it discovers the five pods is reasoned from the
|
||||
measured pod labels and port name, not observed — no target list has been read with it in place,
|
||||
because applying it is an operator action. **Expect `up == 0` on all five when it lands**, which is
|
||||
the documented state, not a regression.
|
||||
- **I did not fix the NetworkPolicy, and this PR therefore does not make the gates observable.** It
|
||||
makes them *discovered*, and makes the remaining gap loud instead of silent. The one narrow rule
|
||||
per app (ns monitoring → `auth-mode=forward-auth` → TCP 44180, per
|
||||
`platform/longhorn/networkpolicy-metrics.yaml`) has to be rendered with each app's own policy, in
|
||||
the catalog/renderer that is in flight on another branch. **It needs an owner.**
|
||||
- **Whether Prometheus will actually select a PodMonitor in ns `oauth2-proxy` targeting other
|
||||
namespaces.** `namespaceSelector.any: true` plus `podMonitorSelectorNilUsesHelmValues: false` and
|
||||
the fact that cross-namespace ServiceMonitors/PrometheusRules already work here say yes; it is not
|
||||
measured with this object.
|
||||
- **No session-lifecycle claim can be made from the current window.** `AuthSuccess` is **0** on both
|
||||
jackett and radarr since the PR #28 restart, so nobody has signed in and `--cookie-refresh=2m30s`
|
||||
has never executed a refresh. The cookie fix is unproven in the one dimension that matters most.
|
||||
- **The radarr `/signalr` finding is the coordinator's measurement, not mine** — I reproduced neither
|
||||
the 206/209 sample nor the jackett 0-in-13,612. They are recorded as reported.
|
||||
- **L2/L3 conformance still did not run** (`CLUSTER_AVAILABLE` unset). Nothing is certified.
|
||||
|
||||
### Round 2 addendum — CI is RED, by design, and the local gate cannot see why
|
||||
|
||||
After pushing round 2, `gate / gate (pull_request)` reported **failure after 1m42s** while the local
|
||||
toolbox run was rc 3 with nothing failed. Measured cause, singular:
|
||||
|
||||
sensitive paths touched:
|
||||
conformance/tests/l0-static/servicemonitor-ports-resolve/check.sh (matched conformance/**)
|
||||
approving reviews: (no reviews at all) author=gitea-admin
|
||||
|
||||
`ci/approval-policy.sh` is fail-closed and exits 1 when a PR touches a `sensitive_paths` glob with no
|
||||
`APPROVED` review from a non-author — "a change to the gate cannot be gated by the gate". The single
|
||||
`KNOWN_EXTERNAL_MONITORS` entry is what trips it. **Nothing in L0/L1 failed.**
|
||||
|
||||
**The local gate structurally cannot catch this, and said `approval-policy: green (N/A)`.** The stage
|
||||
short-circuits to N/A when `GITHUB_EVENT_NAME` is unset, which it always is off-CI. So for any branch
|
||||
touching a sensitive path, a clean local gate is a FALSE REASSURANCE — the PR run is the only honest
|
||||
signal. That is a general trap, not specific to this change, and it is worth knowing before someone
|
||||
concludes a red PR means broken code.
|
||||
|
||||
To clear: a non-author approving review, then **re-run the gate** — approval does not re-trigger the
|
||||
status check, so the required context stays red until the job runs again. `required_approvals=1`
|
||||
already applied to every PR here, so the human was needed regardless; the conformance entry only
|
||||
turns the tick red while that is pending.
|
||||
|
||||
The alternative is to drop the register entry, which means dropping `podmonitor.yaml` with it (the
|
||||
check fails without the declaration). I did not take it: the entry is the "declare the limit"
|
||||
mechanism the check explicitly asks for, it is shrink-only so it cannot rot quietly, and
|
||||
`flux-podmonitor.yaml` is the same case already in the register. But it is a real trade and the
|
||||
reviewer should make it knowingly.
|
||||
@@ -0,0 +1,174 @@
|
||||
---
|
||||
date: 2026-09-30
|
||||
branch: feat/plex-quicksync
|
||||
title: Plex opts into `hardware.gpu: intel-quicksync` — the device mount is GitOps, the Plex-side toggle is a documented manual step, and hardware transcoding has NOT been observed working
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Three things, and the third is the point of the branch.
|
||||
|
||||
1. **`catalog/plex.yaml`** — added the four lines the whole feature reduces to:
|
||||
|
||||
```yaml
|
||||
hardware:
|
||||
gpu: intel-quicksync
|
||||
```
|
||||
|
||||
placed between `env` and `exposure` to match the schema's own property order, with ~48 lines of
|
||||
comment in the file's established style. The comment records: the field is **intent, not
|
||||
mechanism** (a later move to the Intel device plugin is a `render.py` change and this block does
|
||||
not move); that `/dev/dri/renderD128` is present on **all three** prod nodes with `i915` bound;
|
||||
**why a `supplementalGroups` entry would be inert** (the image's cont-init hook does the group
|
||||
work inside the container and `s6-setuidgid` replaces the process group list from `/etc/group`);
|
||||
why `DirectoryOrCreate` is the load-bearing safe failure mode (`apps-wave-0` is `wait: true` with
|
||||
`apps-wave-1` depending on it, so a failed mount would stall the whole app tier for a feature
|
||||
that is not load-bearing); and that the Plex-side toggle is a **separate manual operator step**,
|
||||
pointing at the new runbook.
|
||||
|
||||
2. **`apps/wave-0/plex.yaml`** — re-rendered. Six added lines, nothing else, in the whole `apps/`
|
||||
tree:
|
||||
|
||||
```
|
||||
- name: dri
|
||||
mountPath: /dev/dri
|
||||
- name: dri
|
||||
hostPath:
|
||||
path: /dev/dri
|
||||
type: DirectoryOrCreate
|
||||
```
|
||||
|
||||
3. **`runbooks/plex-quicksync.md`** (new), registered in `runbooks/kustomization.yaml`'s
|
||||
`configMapGenerator` as `- plex-quicksync=plex-quicksync.md` and indexed in `runbooks/README.md`
|
||||
under "Operator procedures". **The file alone would have been a 404** — the pod serves a
|
||||
ConfigMap, not the git tree, and `conformance/tests/l0-static/alert-runbooks-resolvable` exists
|
||||
precisely because "the file is in git" and "the URL resolves" are different claims. The page
|
||||
covers: the GitOps/manual boundary as a table; the toggle (Settings → Transcoder → "Use hardware
|
||||
acceleration when available", preference key `HardwareAcceleratedCodecs`); the real neighbouring
|
||||
keys and the widely-cited one that **does not exist**; that the only proof is `Transcode (hw)` on
|
||||
the Plex dashboard **and why no alert is possible**; the device-cgroup failure to expect first
|
||||
with a concrete open() diagnostic and the Intel-device-plugin fallback; and the floating-`:latest`
|
||||
caveat with a digest check.
|
||||
|
||||
**Stacked on PR #27 (`feat/sd-hardware-capability`), which is open and unmerged.** This branch was
|
||||
cut from it rather than from `main`, so its one commit (`d9baa2a`, the schema + renderer + contract)
|
||||
is in this PR's range. That is deliberate: merging #27 is not an agent's action (CLAUDE.md §1), and
|
||||
duplicating its changes here would have been worse than showing them. **This must merge after #27.**
|
||||
|
||||
Not done, deliberately: no `--emit-waves` (it writes into `clusters/base/`, a sensitive path that
|
||||
would force a human reviewer, and it is a no-op unless the *set* of files under `apps/` changes — it
|
||||
does not here). No `podSecurityContext`. No `nodeSelector`. No alert rule. No `kubectl apply`.
|
||||
|
||||
### Two follow-ups left unclaimed, with the reasoning
|
||||
|
||||
- **Transcode scratch is on the wrong medium, and it is invisible.** The image sets
|
||||
`TranscoderTempDirectory=/transcode`, which is **not a volume** — it is the container's writable
|
||||
overlay, i.e. the node's `/data`. That is NVMe on prod-2 but **SATA on prod-1 and prod-3**, so
|
||||
transcode scratch performance changes silently on reschedule, with nothing reporting it. Both
|
||||
candidate fixes need a **schema** change and so neither belongs in this PR: pin Plex to prod-2
|
||||
(needs a `nodeSelector` field the schema does not have), or give `/transcode` a `medium: Memory`
|
||||
emptyDir and transcode in RAM (~31 GiB per node; needs an emptyDir field the schema does not
|
||||
have). **The RAM option is the better one** — it removes the node asymmetry outright instead of
|
||||
working around it — which is exactly why it deserves its own PR and its own review rather than
|
||||
being smuggled in behind a four-line catalog change. Documented in the runbook's "Known gaps".
|
||||
- **GuC/HuC firmware is not loaded** (`i915.enable_guc=0`). The cost is low-power H.264 **encode**
|
||||
quality only; decode and this feature are unaffected. Changing it needs a kernel parameter plus a
|
||||
**serialized rolling reboot** — an operator action under CLAUDE.md §5, on a cluster with a
|
||||
documented history of that going badly (a prod node hung on a warm reboot on 2026-09-03). Kept
|
||||
well clear of this PR.
|
||||
|
||||
### Rebase note (2026-10-01, Ruth Okonkwo) — resolution only, no redesign
|
||||
|
||||
Rebased onto `main` @ `08deb20` (was cut from `0fbf544`), restacked on the rebased
|
||||
`feat/sd-hardware-capability` (PR #27) so the stack stays coherent — this branch still carries
|
||||
exactly PR #27's commit plus its own, with no duplicate of anything already in `main`. The branch
|
||||
ref was force-updated with `--force-with-lease`, so PR #32 is the same PR with a new head.
|
||||
|
||||
**The `catalog/render.py` conflict disappeared** once PR #27 was rebased; it was never this
|
||||
branch's own conflict. **One conflict remained, in `runbooks/README.md`, purely textual.** PR #30
|
||||
(`fix/jackett-oauth2-proxy-session`, merged as `08deb20`) appended an `oauth2-proxy-session` index
|
||||
entry under *Operator procedures* at the same point this branch appended its `plex-quicksync` entry.
|
||||
**Both entries were kept, in full.** Ordering follows this file's existing convention, which is
|
||||
grouped/narrative rather than alphabetical (the *Nodes* section runs down/io/memory/disk/cpu):
|
||||
`plex-quicksync` sits next to `plex-claim` as the other Plex page, leaving `oauth2-proxy-session`
|
||||
adjacent to `crafty-admin-credential`, which cross-references the same sidecar gate. The change to
|
||||
`runbooks/README.md` is an 8-line pure insertion, so PR #30's entry is untouched.
|
||||
`runbooks/kustomization.yaml` auto-merged and **both** `configMapGenerator` entries are present.
|
||||
|
||||
Re-verified after the resolution, on the rebased stack: `--selftest` PASS (rc 0); a full re-render
|
||||
reproduces every committed golden byte-identically; and the **only** `apps/` diff against the new
|
||||
`main` is `apps/wave-0/plex.yaml`, confined to the expected six lines (the `dri` mount and its
|
||||
`DirectoryOrCreate` hostPath volume). No other service's golden moved — PR #28 added ~176 lines of
|
||||
oauth2-proxy sidecar rendering, and this confirms by measurement rather than assumption that Plex
|
||||
(`auth: native`, no sidecar) is unaffected by it.
|
||||
|
||||
## What I verified
|
||||
|
||||
- **The render is exactly the intended change, and nothing else moved.** After
|
||||
`python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation` (exit 0,
|
||||
silent): `git diff --stat apps/` = `apps/wave-0/plex.yaml | 6 ++++++`, and
|
||||
`git status --porcelain apps/` = a single ` M apps/wave-0/plex.yaml`. Read line by line, the diff
|
||||
is the `dri` volumeMount and the `dri` hostPath volume, and nothing more. **Specifically checked:**
|
||||
the same invocation also rewrites `apps/wave-1/keycloak.yaml` and `apps/wave-0/vm-demo.yaml`, which
|
||||
are git-tracked but excluded from the golden comparison by `RENDER_GOLDEN_NOT_ONBOARDED` in
|
||||
`ci/render-golden.sh` — a spurious diff there would have landed **ungated**. Neither file shows a
|
||||
diff; whole-repo `git status --porcelain` lists only `apps/wave-0/plex.yaml` and `catalog/plex.yaml`
|
||||
plus my new files.
|
||||
- **The rendered output matches what the contract promises** — `schemas/renderer-contract.md`
|
||||
`### hardware` says exactly two things (a `/dev/dri` hostPath with `type: DirectoryOrCreate` and a
|
||||
read-write mount on the app container) and exactly two things appeared. No `securityContext`, no
|
||||
`nodeSelector`, no resources in the diff.
|
||||
- **The runbook is projected into the server, not merely committed.** `runbooks/kustomization.yaml`
|
||||
carries `- plex-quicksync=plex-quicksync.md`, whose shape matches the generator-entry regex the
|
||||
L0 check parses; the extensionless key is what makes `/runbooks/plex-quicksync` a plain static
|
||||
lookup. Also indexed in `runbooks/README.md`.
|
||||
- **The runbook clears the L0 quality floor** that `alert-runbooks-resolvable` applies to
|
||||
alert-linked pages, even though nothing links to this one (it is advisory for an unreferenced
|
||||
page): all four required sections (`## What this means here`, `## First commands`,
|
||||
`## Likely causes, most likely first`, `## When to escalate or stop`), well over the 1500-byte
|
||||
floor, `dev.sh` used throughout, and `--context k3s-prod` on every command.
|
||||
- **The gate.** See the run recorded in the PR description for the per-stage table and the real rc.
|
||||
- **No secret was printed.** Every `Preferences.xml` read in the runbook greps one named key, and the
|
||||
page states explicitly why: the file holds `PlexOnlineToken`. Nothing was `sops -d`'d.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
**Hardware transcoding has NOT been observed working.** Nothing in this branch is evidence that it
|
||||
does. A green gate here proves the manifest renders and the golden matches — it proves nothing about
|
||||
runtime behaviour, and `vars.CLUSTER_AVAILABLE` is unset so L2/L3 conformance did not run at all.
|
||||
|
||||
- **The device-cgroup question is OPEN, and it is the largest remaining technical risk.** Whether
|
||||
containerd on this k3s build permits a **non-privileged** pod to `open()` a bind-mounted character
|
||||
device is **UNVERIFIED**. It can only be settled by deploying and trying. The container inits as
|
||||
uid 0, which settles *file* permissions, but uid 0 does **not** bypass the cgroup device gate — and
|
||||
the two failures look identical from the outside. If it is refused, the fallback is the Intel
|
||||
Device Plugin (`gpu.intel.com/i915`), which is a `catalog/render.py` change plus a re-render with
|
||||
**no ServiceDefinition edit**; `privileged: true` is not available (Kyverno
|
||||
`restrict-host-namespaces` is Enforce). The runbook has the diagnostic sequence.
|
||||
- **The Plex-side toggle was still unset at the time of writing.** `HardwareAcceleratedCodecs` was
|
||||
**absent** from `Preferences.xml` — the toggle has never been enabled on this server. Until an
|
||||
operator ticks it in the UI, this branch changes the pod spec and **nothing about transcoding**.
|
||||
That is the whole reason the runbook exists.
|
||||
- **No metric, so no alert, so no automated detection — permanently.** `catalog/plex.yaml` declares
|
||||
no `metricsEndpoint`, so Plex renders no ServiceMonitor; there is no transcode-mode signal for a
|
||||
rule to evaluate. A silent degradation to software transcoding (a node losing its iGPU, the
|
||||
`DirectoryOrCreate` path) will **not** be reported by anything. I deliberately did not add an
|
||||
alert: with no metric behind it, it could never fire, and this repo has been burned at scale by
|
||||
exactly that. Closing the gap honestly means an exporter first.
|
||||
- **I did not re-derive this session's node or in-container measurements.** The three-node
|
||||
`renderD128`/`i915`/GID-992 facts, the cont-init hook behaviour, the `s6-setuidgid` group
|
||||
replacement, the `PLEX_PREFERENCE*` absence, the six `setPref` keys and the
|
||||
`HardwareAcceleratedCodecs` string were established earlier in this session and are cited, not
|
||||
re-measured, by me. Per CLAUDE.md §2b, treat this file as a record of belief and re-measure against
|
||||
the cluster before relying on any of it.
|
||||
- **The image is a floating tag.** Everything known about the hook came from digest
|
||||
`sha256:e0ab27395614…` (`plexmediaserver 1.43.4.10903`), while the Deployment runs
|
||||
`plexinc/pms-docker:latest` with `imagePullPolicy: Always`. A silent pull can invalidate the
|
||||
"no `supplementalGroups` needed" conclusion **with no commit in this repo**. Not guarded against —
|
||||
the runbook tells the reader to re-check the hook after an image bump, which is documentation, not
|
||||
enforcement.
|
||||
- **I did not verify the live Gitea gate state** (branch protection, required contexts, runner
|
||||
online) for this PR. Per §2b that must be queried, not read out of `CLAUDE.md`'s table; I did not
|
||||
query it.
|
||||
- **`/runbooks/plex-quicksync` was not fetched.** The L0 check asserts every static precondition for
|
||||
an HTTP 200; it does not make the request, and I did not either.
|
||||
@@ -88,3 +88,72 @@ This proves the R-IDP-02 property (unauthenticated cannot reach the upstream) on
|
||||
3. Log in as a **non-platform-admins** user -> expect **403** (authenticated but not authorized).
|
||||
4. Confirm the session cookie is `.home.keithjackson.dev`-scoped (single sign-on across the other
|
||||
protected UIs).
|
||||
|
||||
## Observability: what this component can and cannot see (measured 2026-09-30)
|
||||
|
||||
`servicemonitor.yaml` scrapes **this namespace's** deployment and nothing else. On prod that is
|
||||
two targets, `job="oauth2-proxy"`, both `up=1`. The complete set of oauth2-proxy-specific series
|
||||
in the TSDB is:
|
||||
|
||||
count by (__name__) ({__name__=~"oauth2_proxy.*"})
|
||||
oauth2_proxy_requests_in_flight => 2
|
||||
|
||||
**The per-app gate sidecars are not scraped, and there is nothing to scrape.** Six apps run an
|
||||
`oauth2-proxy` container of their own — crafty, jackett, pihole, qbittorrent, radarr, sonarr —
|
||||
each with `--http-address=0.0.0.0:4180` and **no `--metrics-address`**, so oauth2-proxy's metrics
|
||||
listener never starts. They declare only a `http:4180` container port, no `metrics` port exists on
|
||||
their Services, and `kubectl get servicemonitors,podmonitors -A` selects none of them. Any alert
|
||||
written against `oauth2_proxy_requests_total{namespace="jackett"}` would be structurally inert.
|
||||
|
||||
This matters because the gate's characteristic failure is invisible by construction: an
|
||||
unauthenticated **XHR** gets a `302` to Keycloak rather than a `401`, the browser follows it
|
||||
against a live SSO session with no prompt, and the call resolves **200 carrying an HTML page**
|
||||
where the app expected JSON. Nothing in the request path reports an error. On 2026-09-30 jackett
|
||||
spent a burst of requests in exactly that state, re-established a session, and lost it 26 seconds
|
||||
into a 5-minute session, while `jackett-down` sat at 1 replica available the whole time.
|
||||
|
||||
`auth-visibility-alerts.yaml` therefore alerts on **the gap**, not on the incident:
|
||||
`OAuth2ProxyGateUnobserved` compares `kube_pod_container_info{container="oauth2-proxy"}` against
|
||||
`up{container="oauth2-proxy"}` — two already-scraped families — and names every namespace running
|
||||
a gate that nothing observes. It fires for six namespaces today and clears per-namespace as each
|
||||
one gains a scrape. `runbooks/oauth2-proxy-auth-failures.md` is the by-hand substitute meanwhile,
|
||||
and it reads the sidecar logs out of Loki (Alloy collects them; 72h retention).
|
||||
|
||||
**To close the gap for an app**, three things are needed together, on that app's own manifest:
|
||||
`--metrics-address=0.0.0.0:44180` on the sidecar, a `metrics` containerPort plus the matching
|
||||
Service port, and a ServiceMonitor selecting it. Only then does the auth-churn rule — sustained
|
||||
`code="302"` as a share of `oauth2_proxy_requests_total`, held across a window longer than the
|
||||
session lifetime — become writable rather than decorative.
|
||||
|
||||
### Update 2026-10-01: the listener exists; the scrape path does not
|
||||
|
||||
PR #28 added `--metrics-address=0.0.0.0:44180` and a named `metrics` container port to the five
|
||||
fleet sidecars (jackett, radarr, sonarr, qbittorrent, pihole). `podmonitor.yaml` in this directory
|
||||
discovers them by the pod label `homelab.keithjackson.dev/auth-mode: forward-auth` — a **PodMonitor**
|
||||
rather than a ServiceMonitor because no Service exposes 44180 and adding one would mean changing the
|
||||
renderer.
|
||||
|
||||
The endpoint is real: port-forwarded straight to the jackett pod, `/metrics` returns **http=200,
|
||||
10208 bytes**, exporting `oauth2_proxy_requests_in_flight`, `oauth2_proxy_requests_total` (with
|
||||
`code` labels) and `oauth2_proxy_response_duration_seconds`. That is verified by payload, not by the
|
||||
flag appearing in `args` — a distinction worth keeping, because the two claims came apart here.
|
||||
|
||||
**The scrape is nevertheless dropped.** From a pod in ns `monitoring`, both `:44180` and `:4180` on
|
||||
the jackett pod time out; the shared proxy's `:44180` answers. The only difference is that ns
|
||||
`oauth2-proxy` has **no NetworkPolicy at all**, while each app namespace has a `default-deny` whose
|
||||
`policyTypes` include `Ingress` with no `ingress` rules (deny-all) plus a single CNP admitting
|
||||
`fromEntities: [ingress]` to port 4180 — the Gateway's Envoy and nothing else. Prometheus matches no
|
||||
clause.
|
||||
|
||||
So the targets read `up == 0` and the TSDB is still empty for those namespaces. This is the same bug
|
||||
and the same fingerprint as `platform/longhorn/networkpolicy-metrics.yaml` documents for Longhorn,
|
||||
and the same narrow fix applies: ns `monitoring` → the gate pod label → TCP 44180 only. It cannot
|
||||
ship from this directory, because `oauth2-proxy` and `apps-wave-0` are **siblings** in the Flux graph
|
||||
(both `dependsOn: [platform-config]`), so a NetworkPolicy naming ns `jackett` would apply on today's
|
||||
cluster and fail on a cold rebuild. It belongs with each app's own rendered policy.
|
||||
|
||||
Consequence for `auth-visibility-alerts.yaml`: its `unless` clause requires `up == 1`, not merely a
|
||||
present `up` series. A discovered-but-failing target publishes `up` with value 0, so the bare form
|
||||
would have cleared `OAuth2ProxyGateUnobserved` the moment the PodMonitor landed — reporting the gates
|
||||
as observed while nothing was being scraped. The unit test pins that scenario; deleting the `== 1`
|
||||
fails it and nothing else.
|
||||
@@ -0,0 +1,177 @@
|
||||
---
|
||||
# promtool unit tests for OAuth2ProxyGateUnobserved (platform/oauth2-proxy/auth-visibility-alerts.yaml).
|
||||
#
|
||||
# WHY. The rule is an `unless` between two aggregations over DIFFERENT metric families, and the
|
||||
# failure modes of that shape are both silent: a label that does not survive `count by` makes the
|
||||
# right-hand side match nothing (the alert fires for the scraped namespace too, i.e. permanently),
|
||||
# and a right-hand side that over-matches makes it fire for nobody (i.e. inert). Neither is
|
||||
# visible by reading the expression, and neither is visible on a live cluster where the answer
|
||||
# happens to look plausible. So all four states are pinned: fires for an unscraped namespace,
|
||||
# stays quiet for a scraped one, honours `for: 1h`, and STOPS once a scrape appears.
|
||||
#
|
||||
# HOW TO RUN. The one repo-wide harness, by the companion convention:
|
||||
# ./infra/devcontainer/dev.sh run bash -c 'platform/monitoring/tests/run.sh platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml'
|
||||
# promtool is pinned in infra/devcontainer/versions.env; never run it off the host PATH.
|
||||
#
|
||||
# THE SERIES BELOW ARE REAL SHAPES read off live prod Prometheus on 2026-09-30 before being
|
||||
# copied here — kube_pod_container_info as kube-state-metrics emits it (job="kube-state-metrics",
|
||||
# its own instance/pod labels, image/image_id/container_id) and `up` as the oauth2-proxy
|
||||
# ServiceMonitor emits it (endpoint="metrics", port 44180). The point of using the real shapes is
|
||||
# that `count by (namespace)` has to discard all of them; a test with tidy two-label series would
|
||||
# pass while the real expression collapsed differently.
|
||||
#
|
||||
# ASSERTIONS USE `ALERTS` RATHER THAN exp_alerts: the description annotation is long prose and
|
||||
# comparing it verbatim would couple this file to every wording change while proving nothing the
|
||||
# ALERTS series does not already prove.
|
||||
rule_files:
|
||||
- auth-visibility-alerts.rules.yaml
|
||||
evaluation_interval: 1m
|
||||
|
||||
tests:
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# 1. THE STATE PROD IS ACTUALLY IN (2026-09-30): jackett runs a gate sidecar with no scrape;
|
||||
# the shared oauth2-proxy namespace runs two that ARE scraped. Asserts the alert fires for
|
||||
# jackett and NOT for oauth2-proxy, and that `for: 1h` is honoured (pending at 59m).
|
||||
- interval: 1m
|
||||
name: an unscraped gate namespace fires after 1h; the scraped one never does
|
||||
input_series:
|
||||
- series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
|
||||
values: '1x130'
|
||||
# The shared instance: kube-state-metrics sees both replicas...
|
||||
- series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://aa11bb22",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000001"}'
|
||||
values: '1x130'
|
||||
- series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://cc33dd44",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000002"}'
|
||||
values: '1x130'
|
||||
# ...and both replicas are genuinely scraped, so `up` carries container="oauth2-proxy" there.
|
||||
- series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.189:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="oauth2-proxy"}'
|
||||
values: '1x130'
|
||||
- series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.2.47:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="oauth2-proxy"}'
|
||||
values: '1x130'
|
||||
promql_expr_test:
|
||||
# Not yet firing at 59m — `for: 1h` has not elapsed. Guards against someone "simplifying"
|
||||
# the hold away and turning every rollout into a page.
|
||||
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
|
||||
eval_time: 59m
|
||||
exp_samples: []
|
||||
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett"}'
|
||||
eval_time: 59m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett",severity="warning"}'
|
||||
value: 1
|
||||
# Firing for jackett past the hold...
|
||||
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
|
||||
eval_time: 65m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
|
||||
value: 1
|
||||
# ...and never for the namespace that IS scraped. This is the assertion that catches an
|
||||
# `unless` whose right-hand side stopped matching: without it, a broken rule that fires for
|
||||
# every gate namespace would pass the test above.
|
||||
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="oauth2-proxy"}'
|
||||
eval_time: 65m
|
||||
exp_samples: []
|
||||
# The count must be 1 per namespace-with-a-gate, not 1 overall: a `count by (namespace)`
|
||||
# that lost its `by` would collapse to a single unlabelled series.
|
||||
- expr: 'count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by (namespace) (up{container="oauth2-proxy"})'
|
||||
eval_time: 65m
|
||||
exp_samples:
|
||||
- labels: '{namespace="jackett"}'
|
||||
value: 1
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# 2. THE FIX LANDING. jackett's sidecar gains --metrics-address and a ServiceMonitor 70m in, so
|
||||
# `up` starts carrying container="oauth2-proxy" for namespace="jackett" under a DIFFERENT job
|
||||
# name than the shared proxy's. The alert must stop. This is what makes the rule a gap
|
||||
# tracker rather than a permanent nag, and it pins that the match is on `container`, not on
|
||||
# a hardcoded job="oauth2-proxy" (which would keep firing forever after a correct fix).
|
||||
- interval: 1m
|
||||
name: the alert clears once the sidecar is actually scraped
|
||||
input_series:
|
||||
- series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
|
||||
values: '1x130'
|
||||
# Absent for the first 70 minutes, then present. `_x` is "no sample"; the scrape did not
|
||||
# exist yet.
|
||||
- series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.213:44180",job="jackett-auth",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="jackett-auth-metrics"}'
|
||||
values: '_x70 1x60'
|
||||
promql_expr_test:
|
||||
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett"}'
|
||||
eval_time: 65m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
|
||||
value: 1
|
||||
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="jackett"}'
|
||||
eval_time: 80m
|
||||
exp_samples: []
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# 3. THE REGRESSION GUARD FOR `== 1` — the scenario that exposed a real bug on 2026-10-01.
|
||||
#
|
||||
# A target is DISCOVERED but every scrape FAILS: `up` exists with value 0. This is not
|
||||
# hypothetical — it is the exact state podmonitor.yaml creates today, because the app
|
||||
# namespaces' default-deny drops the scrape (measured; see podmonitor.yaml).
|
||||
#
|
||||
# With the original `unless count by (namespace) (up{container="oauth2-proxy"})`, the `up`
|
||||
# SERIES EXISTS, so `count by (namespace)` returns 1, `unless` removes the namespace, and the
|
||||
# alert GOES QUIET while nothing is being scraped at all — a green signal standing in for an
|
||||
# absent one, which is the precise disease this whole rule was written to cure. Deleting the
|
||||
# `== 1` makes THIS test fail and every other test in this file still pass, which is the
|
||||
# property that makes it worth having.
|
||||
- interval: 1m
|
||||
name: a discovered-but-FAILING target does not count as observed (up==0 must still fire)
|
||||
input_series:
|
||||
- series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-6877fcd9f5-ht9d9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
|
||||
values: '1x130'
|
||||
# The PodMonitor target exists and never succeeds: NetworkPolicy drops the packets, so the
|
||||
# scrape times out and `up` is published as 0 rather than going absent.
|
||||
- series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.0.189:44180",job="oauth2-proxy-sidecars",namespace="jackett",pod="jackett-6877fcd9f5-ht9d9"}'
|
||||
values: '0x130'
|
||||
promql_expr_test:
|
||||
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett"}'
|
||||
eval_time: 65m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
|
||||
value: 1
|
||||
# And the inner clause itself yields NOTHING while the scrape is failing — which is the
|
||||
# mechanism the alert depends on, asserted directly so a future reader can see why the
|
||||
# `== 1` matters without re-deriving it. (Scenario 4 below covers the other direction:
|
||||
# a succeeding scrape really does clear the alert, so `== 1` has not made the rule
|
||||
# impossible to satisfy.)
|
||||
- expr: 'count by (namespace) (up{container="oauth2-proxy"} == 1)'
|
||||
eval_time: 65m
|
||||
exp_samples: []
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# 4. A FLAPPING target: up goes 1 -> 0 and stays there. The alert must come BACK, not latch
|
||||
# cleared. Covers the ordering bug where a once-observed namespace is permanently excluded.
|
||||
- interval: 1m
|
||||
name: a target that starts working and then fails re-fires
|
||||
input_series:
|
||||
- series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="radarr",pod="radarr-7d55cbffb9-42hmh",service="kube-prometheus-stack-kube-state-metrics",uid="3333cccc-0000-0000-0000-000000000001"}'
|
||||
values: '1x250'
|
||||
# Healthy for the first 60m, then every scrape fails for the rest.
|
||||
- series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.77:44180",job="oauth2-proxy-sidecars",namespace="radarr",pod="radarr-7d55cbffb9-42hmh"}'
|
||||
values: '1x60 0x180'
|
||||
promql_expr_test:
|
||||
# Quiet while it was working.
|
||||
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="radarr"}'
|
||||
eval_time: 55m
|
||||
exp_samples: []
|
||||
# Firing again an hour after it broke.
|
||||
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="radarr"}'
|
||||
eval_time: 125m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="radarr",severity="warning"}'
|
||||
value: 1
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# 5. NO GATE, NO ALERT. A namespace with no oauth2-proxy container must never appear, or the
|
||||
# rule would page for every app on the platform the day someone widened the matcher.
|
||||
- interval: 1m
|
||||
name: namespaces without a gate sidecar are never reported
|
||||
input_series:
|
||||
- series: 'kube_pod_container_info{container="plex",container_id="containerd://feed0001",endpoint="http",image="ghcr.io/example/plex:1",image_id="ghcr.io/example/plex@sha256:feed0001",image_spec="ghcr.io/example/plex:1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="plex",pod="plex-0",service="kube-prometheus-stack-kube-state-metrics",uid="2222bbbb-0000-0000-0000-000000000001"}'
|
||||
values: '1x130'
|
||||
promql_expr_test:
|
||||
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved"}'
|
||||
eval_time: 65m
|
||||
exp_samples: []
|
||||
@@ -0,0 +1,137 @@
|
||||
---
|
||||
# oauth2-proxy AUTH-VISIBILITY alerting (R-OBS-04).
|
||||
#
|
||||
# ============================================================================================
|
||||
# WHY THIS FILE EXISTS — THE 2026-09-30 JACKETT FAILURE THAT NOTHING OBSERVED
|
||||
# ============================================================================================
|
||||
# Jackett's UI stopped working. Reloading never fixed it and the user was never re-prompted to
|
||||
# log in. The sidecar's own access log had the whole story:
|
||||
#
|
||||
# [oauthproxy.go:1026] No valid authentication in request. Initiating login.
|
||||
# GET "/api/v2.0/indexers/yts/config?_=..." HTTP/1.1 ... 302 400
|
||||
# GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
|
||||
# ... ten of these at 15:30:53, then a fresh [AuthSuccess] at 15:31:06,
|
||||
# then "No valid authentication in request" again 26 SECONDS LATER, inside the same session.
|
||||
#
|
||||
# NOTHING ANYWHERE ELSE RECORDED IT. oauth2-proxy answers an unauthenticated XHR with a 302 to
|
||||
# Keycloak rather than a 401; the browser follows it silently, the live SSO session satisfies it
|
||||
# without a prompt, and the XHR resolves 200 carrying a 44 KB HTML dashboard where the app's
|
||||
# JavaScript wanted JSON. From the browser, from the user, and from any uptime check, the service
|
||||
# is healthy. `jackett-down` (apps/wave-0/jackett.yaml) watches
|
||||
# kube_deployment_status_replicas_available, which was 1 throughout and would have been 1 if the
|
||||
# gate had rejected every request ever made.
|
||||
#
|
||||
# ============================================================================================
|
||||
# WHY THIS FILE DOES NOT CONTAIN THE OBVIOUS ALERT — MEASURED, prod, 2026-09-30
|
||||
# ============================================================================================
|
||||
# The alert you would want is "a gated app is serving a sustained rate of auth redirects to a
|
||||
# user who is already authenticated". It is NOT WRITABLE against this cluster's TSDB, and the
|
||||
# reason is this file's real subject:
|
||||
#
|
||||
# THE PER-APP GATE SIDECARS EXPORT NO METRICS AT ALL.
|
||||
#
|
||||
# $ promtool query instant :9090 'count by (__name__) ({__name__=~"oauth2_proxy.*"})'
|
||||
# oauth2_proxy_requests_in_flight => 2 <- the SHARED proxy's two replicas. That is ALL.
|
||||
#
|
||||
# PARTLY SUPERSEDED 2026-10-01, and the correction matters more than the original. PR #28 gave
|
||||
# the five FLEET sidecars `--metrics-address=0.0.0.0:44180` and a named `metrics` port, and the
|
||||
# endpoint genuinely serves (verified by payload: http=200, 10208 bytes, all three
|
||||
# `oauth2_proxy_*` families present — not merely by the flag being in `args`). podmonitor.yaml
|
||||
# in this directory now discovers them. But the scrape is still DROPPED by the app namespaces'
|
||||
# default-deny ingress, so the TSDB remains empty for those namespaces; and crafty still has no
|
||||
# metrics listener at all. The rule below is unchanged in purpose and still has work to do.
|
||||
#
|
||||
# Every sidecar (crafty, jackett, pihole, qbittorrent, radarr, sonarr) ran with
|
||||
# `--http-address=0.0.0.0:4180` and NO `--metrics-address`, so oauth2-proxy's metrics listener
|
||||
# was never started; the container declared only port `http:4180`, there was no `metrics` port on
|
||||
# any of their Services, and there was no ServiceMonitor or PodMonitor for any of them anywhere
|
||||
# in the cluster (measured: `kubectl get servicemonitors,podmonitors -A`). Six gates, zero
|
||||
# telemetry. An alert written against `oauth2_proxy_requests_total{namespace="jackett"}` would
|
||||
# have been PERMANENTLY INERT — present, evaluating, silent, and indistinguishable from health.
|
||||
# This repo has shipped that mistake before (VeleroNoRecentBackup, the Longhorn rules), which is
|
||||
# why `conformance/tests/l0-static/alert-metrics-have-scrape-targets` exists, and why the
|
||||
# temptation was refused here rather than ratcheted into a register.
|
||||
#
|
||||
# The SHARED proxy IS scraped (job="oauth2-proxy", 2 targets, up=1) but it is not in the data
|
||||
# path for any sidecar-gated app — each app gates itself — and its current pods export no
|
||||
# `oauth2_proxy_requests_total` at all, having proxied nothing since they started. An alert on
|
||||
# the shared instance would not have fired on this incident either. Saying so is the point.
|
||||
#
|
||||
# So this file alerts on the CONDITION THAT MADE THE INCIDENT INVISIBLE, not on the incident.
|
||||
# That condition is measurable today, from two already-scraped metric families, and it is the
|
||||
# precondition for ever being able to alert on the incident itself. It self-clears the moment a
|
||||
# sidecar gains a metrics endpoint and a scrape — at which point the auth-churn rule becomes
|
||||
# writable and belongs here beside it. See runbooks/oauth2-proxy-auth-failures.md, which is the
|
||||
# by-hand equivalent an operator uses in the meantime.
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: oauth2-proxy-auth-visibility
|
||||
namespace: oauth2-proxy
|
||||
labels:
|
||||
release: kube-prometheus-stack # kube-prometheus-stack's ruleSelector
|
||||
app.kubernetes.io/name: oauth2-proxy
|
||||
spec:
|
||||
groups:
|
||||
- name: oauth2-proxy.auth-visibility
|
||||
rules:
|
||||
- alert: OAuth2ProxyGateUnobserved
|
||||
# A namespace runs an oauth2-proxy gate container, and Prometheus holds no scrape
|
||||
# target for that container in that namespace.
|
||||
#
|
||||
# BOTH SIDES ARE ALREADY-SCRAPED FAMILIES, deliberately:
|
||||
# * kube_pod_container_info — kube-state-metrics, the declared owner in
|
||||
# alert-metrics-have-scrape-targets. It sees the container whether or not the
|
||||
# container exports anything, which is exactly the property needed here.
|
||||
# * up — UNIVERSAL. `up` carries the `container` label for any
|
||||
# ServiceMonitor/PodMonitor target, so this works for a future per-app scrape
|
||||
# without knowing what job name it will pick.
|
||||
# Nothing in this expression touches `oauth2_proxy_*`, so the rule cannot itself be
|
||||
# inert for the reason it is reporting.
|
||||
#
|
||||
# MEASURED ON PROD 2026-09-30 — this expression returned six namespaces:
|
||||
# {namespace="crafty"} {namespace="jackett"} {namespace="pihole"}
|
||||
# {namespace="qbittorrent"} {namespace="radarr"} {namespace="sonarr"}
|
||||
# and excluded namespace="oauth2-proxy", whose two replicas ARE scraped. It fires on
|
||||
# the state the cluster is in; it is not a rule waiting for a hypothetical.
|
||||
#
|
||||
# `== 1` IS LOAD-BEARING, AND IT IS A BUG FIX (2026-10-01). This clause was originally
|
||||
# a bare `up{container="oauth2-proxy"}`, which was WRONG in the one way that matters
|
||||
# here: a target that is DISCOVERED BUT FAILING still publishes `up`, with value 0. So
|
||||
# the series exists, `count by (namespace)` returns 1, `unless` removes the namespace,
|
||||
# and the alert GOES QUIET while nothing whatsoever is being scraped. That is this
|
||||
# branch's own disease — a green signal standing in for an absent one — reproduced
|
||||
# inside the alert written to cure it. It was not hypothetical: podmonitor.yaml lands
|
||||
# five targets whose scrape is DROPPED by NetworkPolicy (measured; see that file), so
|
||||
# the bare form would have silently cleared on the very next commit. `== 1` means
|
||||
# "observed" requires a scrape that actually succeeds.
|
||||
#
|
||||
# A failing target is therefore reported by BOTH this rule and kube-prometheus-stack's
|
||||
# generic TargetDown. That duplication is deliberate: TargetDown says "a target is
|
||||
# down", this says "an authentication gate is unobserved", and during an incident those
|
||||
# are different sentences. The overlap is the cheap half of the trade.
|
||||
expr: |
|
||||
count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
|
||||
unless
|
||||
count by (namespace) (up{container="oauth2-proxy"} == 1)
|
||||
# 1h, not 5m: a rollout briefly has a running sidecar whose target has not been
|
||||
# discovered yet, and this condition is a standing architectural gap, never an
|
||||
# emergency. Long enough that no deploy can produce it.
|
||||
for: 1h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "oauth2-proxy gate in {{ $labels.namespace }} exports no metrics"
|
||||
description: >-
|
||||
The {{ $labels.namespace }} namespace runs an oauth2-proxy gate container and
|
||||
Prometheus has NO scrape target for it, so every authentication decision that gate
|
||||
makes — every redirect to Keycloak, every 403 on a group check, every session that
|
||||
silently fails to hold — is invisible to metrics and to alerting. This is not a
|
||||
theoretical gap: on 2026-09-30 the jackett gate answered a burst of the app's own
|
||||
XHR calls with 302s to Keycloak, re-established a session, and lost it again 26
|
||||
seconds later, and the only record anywhere was a line in the sidecar's stdout.
|
||||
No alert could have fired, because there is nothing to alert on. Until a scrape
|
||||
exists the runbook's LogQL queries are the substitute; the fix is
|
||||
`--metrics-address=0.0.0.0:44180` on the sidecar plus a `metrics` port and a
|
||||
ServiceMonitor, after which the auth-churn rule becomes writable.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/oauth2-proxy-auth-failures"
|
||||
@@ -14,3 +14,12 @@ resources:
|
||||
- oauth2-proxy-credentials.sops.yaml
|
||||
- httproute.yaml # forward-auth sign-in/callback ingress (was referenced in-place by platform-config)
|
||||
- servicemonitor.yaml # Prometheus scrape (was referenced in-place by platform-config)
|
||||
# Alerting on whether the gates are OBSERVABLE at all. Travels with the servicemonitor above
|
||||
# for the same reason it does: monitoring.coreos.com CRDs are registered by the `platform`
|
||||
# tier, which is Ready before this tier's platform-config dependency completes.
|
||||
- auth-visibility-alerts.yaml
|
||||
# Discovers the PER-APP gate sidecars' :44180 metrics endpoints by pod label (PR #28 added the
|
||||
# listener; no Service exposes it, so this is a PodMonitor, not a ServiceMonitor). The scrape is
|
||||
# still denied by the app namespaces' default-deny ingress — read podmonitor.yaml's header
|
||||
# before concluding the targets should be up.
|
||||
- podmonitor.yaml
|
||||
@@ -0,0 +1,109 @@
|
||||
---
|
||||
# PodMonitor for the PER-APP oauth2-proxy gate sidecars (R-OBS-04).
|
||||
#
|
||||
# ============================================================================================
|
||||
# WHY A PodMonitor AND NOT A ServiceMonitor
|
||||
# ============================================================================================
|
||||
# PR #28 gave all five fleet sidecars `--metrics-address=0.0.0.0:44180` and a named `metrics`
|
||||
# container port. It did NOT add a Service port, and measured on prod 2026-10-01 none of the five
|
||||
# Services exposes 44180:
|
||||
#
|
||||
# jackett jackett: [('port-9117', 9117), ('authproxy', 4180)]
|
||||
# radarr radarr: [('port-7878', 7878), ('authproxy', 4180)]
|
||||
# sonarr sonarr: [('port-8989', 8989), ('authproxy', 4180)]
|
||||
# qbittorrent qbittorrent: [('port-8080', 8080), ('authproxy', 4180)]
|
||||
# pihole pihole: [('port-80', 80), ('authproxy', 4180)]
|
||||
#
|
||||
# A ServiceMonitor selects Service PORTS, so it would have nothing to select. A PodMonitor targets
|
||||
# pod ports directly and bypasses Services entirely — so this needs no change to the Services, and
|
||||
# therefore no change to `catalog/render.py` or the `apps/wave-*` renders and goldens.
|
||||
#
|
||||
# SELECTOR: the `homelab.keithjackson.dev/auth-mode: forward-auth` POD LABEL, not a namespace list.
|
||||
# Measured 2026-10-01: that label is present on exactly the five fleet gate pods (jackett, radarr,
|
||||
# sonarr, qbittorrent, pihole) and absent from crafty's, which is a separately-managed deployment
|
||||
# (platform/crafty) that still has no `--metrics-address` at all. So the selector describes the
|
||||
# PROPERTY being scraped rather than enumerating today's apps, and a sixth forward-auth app is
|
||||
# picked up with no edit here. `namespaceSelector.any: true` because those five namespaces are
|
||||
# created by the apps-wave-0/1 tiers, which this tier does not and must not depend on.
|
||||
#
|
||||
# ============================================================================================
|
||||
# READ THIS BEFORE BELIEVING THE TARGETS ARE UP: THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY
|
||||
# ============================================================================================
|
||||
# This PodMonitor is NECESSARY AND NOT SUFFICIENT. The endpoint serves, and the path from ns
|
||||
# monitoring is blocked. Both halves measured on prod 2026-10-01, read-only:
|
||||
#
|
||||
# THE ENDPOINT IS HEALTHY — port-forward straight to the jackett pod's 44180:
|
||||
# GET /metrics -> http=200 bytes=10208
|
||||
# # TYPE oauth2_proxy_requests_in_flight gauge
|
||||
# # TYPE oauth2_proxy_requests_total counter
|
||||
# # TYPE oauth2_proxy_response_duration_seconds histogram
|
||||
# oauth2_proxy_requests_total{code="200"} 3
|
||||
# oauth2_proxy_requests_total{code="302"} 2
|
||||
#
|
||||
# THE SCRAPE PATH IS DROPPED — from a pod in ns monitoring (grafana, which sits beside
|
||||
# Prometheus and so occupies the same policy position) to the jackett pod IP:
|
||||
# http://10.0.0.189:44180/metrics -> wget: download timed out
|
||||
# http://10.0.0.189:4180/ping -> wget: download timed out (even the "allowed" port)
|
||||
# http://<shared-proxy>:44180/metrics -> 200, oauth2_proxy_* present (control: ns oauth2-proxy
|
||||
# has NO NetworkPolicy, which is the only reason the
|
||||
# shared instance has ever been scrapable)
|
||||
#
|
||||
# WHY: ns jackett's `default-deny` NetworkPolicy is `podSelector: {}` with
|
||||
# `policyTypes: [Ingress, Egress]` and an `egress` rule ONLY — no `ingress` key, which in
|
||||
# NetworkPolicy semantics is deny-all-ingress. The only ingress allowance is the CNP
|
||||
# `jackett-allow-gateway-ingress`: `fromEntities: [ingress]` to port 4180 — the Gateway's Envoy
|
||||
# and nothing else. Prometheus matches no clause and its packets are dropped, not refused.
|
||||
#
|
||||
# SO THE TARGETS BELOW WILL READ `up == 0` UNTIL AN INGRESS ALLOWANCE LANDS. That is a known,
|
||||
# measured, documented state — NOT a regression introduced here, and NOT something to silence.
|
||||
# It is deliberately left visible: `OAuth2ProxyGateUnobserved` in auth-visibility-alerts.yaml
|
||||
# requires `up == 1`, precisely so a discovered-but-failing target does not read as "observed"
|
||||
# (see that file's header — this exact trap is why the `== 1` is there).
|
||||
#
|
||||
# THIS IS THE SAME BUG LONGHORN HAD, and the platform has a documented fix idiom for it:
|
||||
# platform/longhorn/networkpolicy-metrics.yaml, whose header records the identical fingerprint
|
||||
# measured on 2026-09-13 — ServiceMonitor in place, 3 targets DISCOVERED, `up` 0/0/0, TSDB empty,
|
||||
# `lastError: context deadline exceeded` (a drop, not a refusal). The fix there is a narrow
|
||||
# additive `networking.k8s.io/v1` NetworkPolicy: ns monitoring -> the pod label, that one port.
|
||||
#
|
||||
# WHY THAT POLICY IS NOT IN THIS DIRECTORY. It would have to be five objects, one per app
|
||||
# namespace, and those namespaces are created by apps-wave-0/1. Measured Flux graph, 2026-10-01:
|
||||
# oauth2-proxy dependsOn=['platform-config'] <- this tier
|
||||
# apps-wave-0 dependsOn=['platform-config'] <- SIBLING, no ordering between them
|
||||
# apps-wave-1 dependsOn=['apps-wave-0']
|
||||
# They are siblings, so on a cold bootstrap there is no guarantee ns jackett exists when this tier
|
||||
# applies, and a NetworkPolicy naming a missing namespace fails to apply. It would work on today's
|
||||
# running cluster and break on a rebuild — the "works now, broken on restore" shape this repo
|
||||
# keeps paying for. Longhorn's precedent is safe only because longhorn-system is created by the
|
||||
# same tier that ships its policy. The allowance therefore belongs WITH THE APPS, rendered next to
|
||||
# the rest of each app's network policy — i.e. in the catalog/renderer, which is in flight on
|
||||
# another branch. Tracked in the handoff; it is one narrow rule per app, not a redesign.
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PodMonitor
|
||||
metadata:
|
||||
name: oauth2-proxy-sidecars
|
||||
namespace: oauth2-proxy
|
||||
labels:
|
||||
release: kube-prometheus-stack # kube-prometheus-stack's podMonitorSelector
|
||||
app.kubernetes.io/name: oauth2-proxy
|
||||
homelab.keithjackson.dev/purpose: metrics-scrape
|
||||
spec:
|
||||
namespaceSelector:
|
||||
# The gate sidecars live in the app namespaces, not this one. `any` rather than a fixed list so
|
||||
# a new forward-auth app needs no edit here — the pod label below is what scopes this.
|
||||
any: true
|
||||
selector:
|
||||
matchLabels:
|
||||
# The PROPERTY, not the app list. Present on exactly the five fleet gate pods (measured
|
||||
# 2026-10-01); absent from crafty, which has no metrics listener yet.
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
podMetricsEndpoints:
|
||||
# By PORT NAME (`metrics`, 44180), never by number: the name is what PR #28 declared on the
|
||||
# container, and a numeric port here would silently stop matching if the port were renumbered.
|
||||
- port: metrics
|
||||
path: /metrics
|
||||
interval: 30s
|
||||
# 10s, matching the drop signature above: a DENIED path does not refuse, it hangs until the
|
||||
# scrape times out. Leaving the default would make each failing target hold a scrape slot for
|
||||
# the full interval.
|
||||
scrapeTimeout: 10s
|
||||
@@ -51,6 +51,36 @@ rule groups still evaluate against nothing and **cannot fire**:
|
||||
| Flux (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` |
|
||||
| Hubble (`hubble_drop_total`) | `HubbleDeniedFlowSpike` |
|
||||
| etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all |
|
||||
| per-app oauth2-proxy sidecars (`oauth2_proxy_*`) — updated 2026-10-01 | no authentication alerting exists at all for crafty, jackett, pihole, qbittorrent, radarr, sonarr |
|
||||
|
||||
The oauth2-proxy row is the newest, and it has **moved twice in two days** — worth reading in order,
|
||||
because each step looked like the end of it.
|
||||
|
||||
**2026-09-30 — nothing to scrape.** `count by (__name__) ({__name__=~"oauth2_proxy.*"})` returned
|
||||
exactly one family, `oauth2_proxy_requests_in_flight`, from the two replicas of the **shared**
|
||||
forward-auth instance. All six per-app gate sidecars ran without `--metrics-address`, so their metrics
|
||||
listener never started, and no ServiceMonitor or PodMonitor selected them.
|
||||
|
||||
**2026-10-01 — the listener exists, and the scrape is still blocked.** PR #28 added
|
||||
`--metrics-address=0.0.0.0:44180` and a named `metrics` port to the five fleet sidecars, and the
|
||||
endpoint genuinely serves (port-forwarded: `http=200`, 10208 bytes, all three `oauth2_proxy_*`
|
||||
families, `oauth2_proxy_requests_total{code="302"}` present). `platform/oauth2-proxy/podmonitor.yaml`
|
||||
now discovers them by pod label. **But the app namespaces' `default-deny` NetworkPolicy drops the
|
||||
scrape**: from a pod in ns `monitoring`, both `:44180` and `:4180` on the jackett pod time out, while
|
||||
the shared proxy's `:44180` answers — the only difference being that ns `oauth2-proxy` has no
|
||||
NetworkPolicy at all. So the targets are discovered and `up == 0`, and the TSDB is still empty for
|
||||
those namespaces. **This is the same bug Longhorn had**, with the same fingerprint; the fix idiom is
|
||||
`platform/longhorn/networkpolicy-metrics.yaml` (ns monitoring → one pod label → one port), and for
|
||||
these apps it has to be rendered alongside each app's own policy rather than shipped from the
|
||||
oauth2-proxy tier, which is a *sibling* of `apps-wave-0` in the Flux graph and so cannot rely on those
|
||||
namespaces existing yet.
|
||||
|
||||
`OAuth2ProxyGateUnobserved` (`platform/oauth2-proxy/auth-visibility-alerts.yaml`) alerts on the gap
|
||||
itself from `kube_pod_container_info` and `up` — two families that *are* scraped. Note its `up == 1`:
|
||||
a discovered-but-failing target publishes `up` with value **0**, so the earlier bare form would have
|
||||
gone quiet the moment the PodMonitor landed, while nothing was being scraped. That near-miss is the
|
||||
whole reason to distrust a cleared alert here. Crafty remains outside all of this — no metrics
|
||||
listener at all. See `oauth2-proxy-auth-failures`.
|
||||
|
||||
Each affected runbook says so on its own page and gives the by-hand equivalent. See `targets-down`.
|
||||
|
||||
@@ -108,6 +138,14 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
|
||||
- **`keycloak-db`** — Keycloak's CloudNativePG database is unavailable.
|
||||
- **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
|
||||
every green signal missed.
|
||||
- **`oauth2-proxy-auth-failures`** — a gated app "is up but does not work". Linked from
|
||||
`OAuth2ProxyGateUnobserved`. The gate answers an unauthenticated **XHR** with a 302 to Keycloak
|
||||
rather than a 401, the browser follows it silently against a live SSO session, and the call
|
||||
resolves **200 carrying HTML** where the app's JavaScript wanted JSON — so the user is never
|
||||
re-prompted and every uptime check stays green. Tells the gate's refusal from the group check
|
||||
from the app's own login, from the sidecar log and from Loki, with the 2026-09-30 jackett trace
|
||||
as the worked example. **Read the caveat**: the per-app gate sidecars export no metrics at all,
|
||||
so these logs (72h of them) are the only record this class of failure leaves.
|
||||
|
||||
### Operator procedures (no alert links here)
|
||||
- **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only
|
||||
@@ -115,6 +153,14 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
|
||||
plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
|
||||
token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
|
||||
Includes why a changed Secret does not restart anything, and why an expired token is harmless.
|
||||
- **`plex-quicksync`** — hardware transcoding on the host iGPU. GitOps puts `/dev/dri` in the pod
|
||||
(`hardware.gpu: intel-quicksync`); **a human still has to tick Settings → Transcoder → "Use hardware
|
||||
acceleration when available"**, and no environment variable can do it (the image supports no
|
||||
`PLEX_PREFERENCE*`, verified). The page names the real preference key (`HardwareAcceleratedCodecs`,
|
||||
read out of the binary), says which widely-cited key **does not exist**, and is explicit that the
|
||||
**only** proof is `Transcode (hw)` on the Plex dashboard — Plex renders no ServiceMonitor, so no
|
||||
alert could ever fire. Also: the device-cgroup failure to expect first and its device-plugin
|
||||
fallback, and why a `:latest` pull can silently break it.
|
||||
- **`oauth2-proxy-session`** — a gated UI loads but its saves quietly do nothing, and you are never
|
||||
re-prompted to log in. Tells the three stacked auth layers apart from the sidecar's own access log
|
||||
(the four lines that matter, including the upstream `-` that means the proxy answered a request the
|
||||
|
||||
@@ -85,16 +85,29 @@ configMapGenerator:
|
||||
- node-io=node-io.md
|
||||
- node-memory=node-memory.md
|
||||
- node-memory-swap=node-memory-swap.md
|
||||
# Linked from OAuth2ProxyGateUnobserved, and the diagnostic procedure for any gated app
|
||||
# that "is up but does not work" — the gate answers an unauthenticated XHR with a 302,
|
||||
# not a 401, so the failure arrives as an HTTP 200 carrying the wrong body.
|
||||
- oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md
|
||||
# Operator procedure, not alert-linked. Reading the per-app oauth2-proxy sidecar log to tell
|
||||
# Keycloak, the sidecar and the app's own login apart. Cookie collision between two sidecars
|
||||
# presents as random session loss in ONE app's log and is only legible with both side by side,
|
||||
# so the page has to be reachable at 3am without going looking for it in git.
|
||||
# SIBLING PAGE to oauth2-proxy-auth-failures above, and they are deliberately both here: this
|
||||
# one is the SESSION-LIFECYCLE procedure (why a session died), that one is the DIAGNOSTIC
|
||||
# TRIAGE (which of the three layers refused, and how to tell a 302-to-200 from a real success).
|
||||
- oauth2-proxy-session=oauth2-proxy-session.md
|
||||
# Operator procedure, not an alert page (no alert links to it — the
|
||||
# alert-runbooks-resolvable check treats that as advisory). It is served
|
||||
# because the plex.tv claim window is short and is not a thing to go looking
|
||||
# for in git: /runbooks/plex-claim has to resolve the moment it is needed.
|
||||
- plex-claim=plex-claim.md
|
||||
# Operator procedure, not an alert page, and NO alert is possible for it: plex renders no
|
||||
# metricsEndpoint, so there is no ServiceMonitor, no transcode-mode metric and nothing a rule
|
||||
# could evaluate. Served because the feature it completes is INERT until a human ticks a box in
|
||||
# the Plex UI — the one step GitOps cannot reach — and a page nobody can find is how a shipped
|
||||
# feature stays switched off for months.
|
||||
- plex-quicksync=plex-quicksync.md
|
||||
- pvc-capacity=pvc-capacity.md
|
||||
- targets-down=targets-down.md
|
||||
- update-rollback=update-rollback.md
|
||||
|
||||
@@ -0,0 +1,244 @@
|
||||
# oauth2-proxy auth failures — telling the three layers apart
|
||||
|
||||
Linked from `OAuth2ProxyGateUnobserved`, and the page to open whenever a gated app "is up but
|
||||
does not work": a UI that loads and then shows nothing, a page that never finishes, a list that
|
||||
stays empty, a save that silently does not save. The alert is about the *visibility* gap; this
|
||||
page is the diagnostic procedure, and it works whether or not the alert ever fired.
|
||||
|
||||
## What this means here
|
||||
|
||||
Every app behind Keycloak on this platform has **three** places a request can be refused, and
|
||||
from a browser they look identical — because two of the three answer with **HTTP 200**.
|
||||
|
||||
| Layer | What refuses | What the browser sees |
|
||||
|---|---|---|
|
||||
| 1. The gate (oauth2-proxy sidecar) | no valid session cookie | **302 to Keycloak**, then usually **200** |
|
||||
| 2. The gate's group check | session valid, wrong group | 403, or a redirect loop |
|
||||
| 3. The app itself | the app's own login / API key | the app's own 401 or login page |
|
||||
|
||||
**Layer 1 is the one that hides.** oauth2-proxy answers an unauthenticated request with a
|
||||
redirect, not a 401 — correct for a browser navigation, disastrous for an **XHR**. The app's
|
||||
JavaScript asks for JSON, the gate answers `302`, the browser follows it silently, the live
|
||||
Keycloak SSO session satisfies it with no prompt, and the XHR resolves **200 with an HTML
|
||||
dashboard in the body**. `fetch()` sees a success. The user sees an empty page. Reloading does
|
||||
nothing, and nobody is ever asked to log in, because from the browser's point of view nothing
|
||||
failed.
|
||||
|
||||
The worked example, jackett on 2026-09-30, straight from the sidecar log:
|
||||
|
||||
[oauthproxy.go:1026] No valid authentication in request. Initiating login.
|
||||
GET "/api/v2.0/indexers/yts/config?_=..." HTTP/1.1 ... 302 400
|
||||
GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
|
||||
... ten of these at 15:30:53
|
||||
[AuthSuccess] Authenticated via OAuth2: Session{email:... created:2026-09-30 15:31:06
|
||||
expires:2026-09-30 15:36:06 refresh_token:true groups:[platform-admins]}
|
||||
GET "/UI/Dashboard" 200 44379
|
||||
GET "/api/v2.0/indexers?_=..." 200 1501229
|
||||
... and 26 seconds later, inside that same 5-minute session:
|
||||
[oauthproxy.go:1026] No valid authentication in request. Initiating login.
|
||||
|
||||
A session created at 15:31:06, valid until 15:36:06, not honoured at 15:31:32. That is the
|
||||
signature of the session being **re-established over and over instead of held** — a cookie that
|
||||
another proxy is overwriting, or a cookie that never comes back at all — not of Keycloak being
|
||||
down and not of the app being broken.
|
||||
|
||||
**Read this before you trust any silence (updated 2026-10-01).** The five fleet gate sidecars now
|
||||
*do* run a metrics listener on `:44180`, and `platform/oauth2-proxy/podmonitor.yaml` discovers
|
||||
them — but **the scrape is dropped by the app namespaces' `default-deny` NetworkPolicy**, so those
|
||||
targets read `up == 0` and the TSDB holds nothing for them. Crafty has no listener at all. Only
|
||||
the shared `oauth2-proxy` namespace instance is genuinely scraped, and it is not in the data path
|
||||
for any of these apps. So there is still nothing to graph and nothing that could have alerted.
|
||||
**The logs below remain the only record that this class of failure happened.** Check which state
|
||||
you are in before you trust a quiet dashboard — "First commands" step 1 tells you in one query.
|
||||
|
||||
## First commands
|
||||
|
||||
`kubectl` does not exist on the workstation; everything runs in the pinned container, and always
|
||||
with an explicit context.
|
||||
|
||||
**1. Which namespaces have an unobserved gate** (this is what the alert measures):
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
|
||||
-c prometheus -- /bin/promtool query instant http://localhost:9090 \
|
||||
"count by (namespace) (kube_pod_container_info{container=\"oauth2-proxy\"}) unless count by (namespace) (up{container=\"oauth2-proxy\"} == 1)"'
|
||||
```
|
||||
|
||||
**1b. If a namespace is listed, is it "no target" or "target failing"?** The two need different
|
||||
fixes and the alert deliberately does not distinguish them:
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
|
||||
-c prometheus -- /bin/promtool query instant http://localhost:9090 \
|
||||
"up{container=\"oauth2-proxy\"}"'
|
||||
```
|
||||
|
||||
No series for the namespace → nothing discovers it (no PodMonitor match; check the pod carries
|
||||
`homelab.keithjackson.dev/auth-mode=forward-auth` and a named `metrics` port). A series with
|
||||
value **0** → discovered but every scrape fails; get the reason from the target's `lastError`:
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
|
||||
curl -s localhost:19090/api/v1/targets | python3 -c "
|
||||
import json,sys
|
||||
for t in json.load(sys.stdin)[\"data\"][\"activeTargets\"]:
|
||||
if t[\"labels\"].get(\"container\")==\"oauth2-proxy\":
|
||||
print(t[\"labels\"].get(\"namespace\"), t[\"health\"], t.get(\"lastError\",\"\")[:90])"'
|
||||
```
|
||||
|
||||
`context deadline exceeded` is a **drop**, not a refusal — that is NetworkPolicy, not a dead
|
||||
endpoint. Confirm the endpoint itself is fine by port-forwarding straight to the pod, which
|
||||
bypasses policy entirely:
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n jackett port-forward \
|
||||
$(kubectl --context k3s-prod -n jackett get pod -l app.kubernetes.io/name=jackett -o name | head -1) \
|
||||
14418:44180 & sleep 5; curl -s -o /dev/null -w "http=%{http_code} bytes=%{size_download}\n" \
|
||||
http://127.0.0.1:14418/metrics'
|
||||
```
|
||||
|
||||
`http=200` with a few KB means the listener is healthy and the path is what is broken. As of
|
||||
2026-10-01 that is the state of all five fleet apps, and the fix is an ingress allowance from ns
|
||||
`monitoring` to port 44180 — see `platform/longhorn/networkpolicy-metrics.yaml` for the idiom.
|
||||
|
||||
**2. The sidecar log — but you must grep it.** The gate's stdout is ~11,500 lines a day per app
|
||||
and almost all of it is the kubelet health probe (`GET "/ping" ... 200`). A plain
|
||||
`kubectl logs --tail=100` shows you nothing but `/ping` and looks perfectly healthy. Filter:
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n jackett logs deploy/jackett -c oauth2-proxy --since=2h \
|
||||
| grep -v "/ping" | grep -E "No valid authentication|AuthSuccess|403|401|callback"'
|
||||
```
|
||||
|
||||
Substitute the namespace: `crafty`, `jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`.
|
||||
|
||||
**3. The same question across every gate at once, from Loki.** Alloy collects all of these, so
|
||||
you can ask about the whole fleet and about a window that has already scrolled out of
|
||||
`kubectl logs`. Port-forward Loki and query it:
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n monitoring port-forward svc/loki 13100:3100 & sleep 5; \
|
||||
curl -s -G --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"No valid authentication in request\" [24h]))" \
|
||||
http://127.0.0.1:13100/loki/api/v1/query | python3 -m json.tool | grep -E "namespace|value" '
|
||||
```
|
||||
|
||||
and the denominator, which is what makes the number mean something:
|
||||
|
||||
```
|
||||
... --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"AuthSuccess\" [24h]))" ...
|
||||
```
|
||||
|
||||
**How to read the ratio.** A healthy sign-in is roughly **one or two** `No valid authentication`
|
||||
lines per `AuthSuccess` — the first request of a session is unauthenticated by definition. A
|
||||
ratio in the tens means sessions are not surviving. Measured on prod 2026-09-30 (24h):
|
||||
|
||||
| namespace | `No valid authentication` | `AuthSuccess` | ratio |
|
||||
|---|---|---|---|
|
||||
| crafty | 81 | 1 | 81:1 |
|
||||
| radarr | 67 | 1 | 67:1 |
|
||||
| jackett | 39 | 4 | ~10:1 |
|
||||
| pihole, qbittorrent, sonarr | 0 | 0 | (unused in the window) |
|
||||
|
||||
All three of those are the failure, not the baseline.
|
||||
|
||||
**The cleanest worked case of all — radarr, measured 2026-10-01.** After the cookie fix landed,
|
||||
jackett dropped to **0** `No valid authentication` lines in 13,612 (from 39), and `removing
|
||||
session` / `cookie signature not valid` went to **0 on both apps** (radarr had 8 and 8 before).
|
||||
The eviction fingerprint is gone. But radarr still logged **206** rejections — and sampling 209 of
|
||||
them, **every single one was the same request**:
|
||||
|
||||
GET "/signalr/messages/negotiate" ... 302 <- browser UA, over and over
|
||||
|
||||
That is one left-open browser tab. Its cookie is gone, its SignalR socket keeps trying to
|
||||
reconnect, and every attempt is silently 302'd to Keycloak. Nobody is at the keyboard, so nobody
|
||||
is ever re-prompted; the tab just sits there looking like a working page that has stopped
|
||||
updating. **This is the entire failure class in one line**: a non-navigation request answered with
|
||||
a redirect, forever, with no error surfaced anywhere.
|
||||
|
||||
So when you read a rejection count, **always group by request path before concluding anything**:
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n radarr logs deploy/radarr -c oauth2-proxy --since=24h \
|
||||
| grep -v "/ping" | grep "No valid authentication" -A 1 \
|
||||
| grep -oE "\"[^\"]+\" HTTP" | sort | uniq -c | sort -rn | head'
|
||||
```
|
||||
|
||||
One path dominating means an orphaned client, not a broken gate — annoying but harmless, and it
|
||||
clears when the tab is closed. Many different paths from one session means the session itself is
|
||||
not surviving, which is the real defect. A per-app `--api-route` is the structural fix for the
|
||||
first shape: it makes those paths answer **401** instead of 302, so the client fails loudly
|
||||
instead of looping. Note that a count of **0 `AuthSuccess`** alongside a high rejection count
|
||||
means nobody has signed in since the last restart — so `--cookie-refresh` has never run, and no
|
||||
session-lifecycle conclusion can be drawn from that window at all.
|
||||
|
||||
**4. Did the gate admit the user, or did the app refuse them?** Look for the `[AuthSuccess]` line
|
||||
and what follows it. If `[AuthSuccess]` appears and the very next thing is the *app's own* login
|
||||
page or the app's 401, the gate did its job and layer 3 is your problem — that is the Crafty
|
||||
shape, and `runbooks/crafty-admin-credential.md` covers it.
|
||||
|
||||
**5. Cookie names, when sessions do not survive.** Two oauth2-proxy instances emitting the
|
||||
**same** cookie name on the **same** domain hold different cookie-secrets, so each overwrites the
|
||||
other's cookie and the loser logs exactly the `No valid authentication` line above. List what
|
||||
each gate is configured to emit (arguments only — never print a Secret's data):
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'for ns in crafty jackett pihole qbittorrent radarr sonarr; do printf "%-12s " "$ns"; \
|
||||
kubectl --context k3s-prod -n $ns get deploy $ns \
|
||||
-o jsonpath="{range .spec.template.spec.containers[?(@.name==\"oauth2-proxy\")]}{.args}{end}" \
|
||||
| tr "," "\n" | grep -E "cookie-name|cookie-domain" | tr "\n" " "; echo; done'
|
||||
```
|
||||
|
||||
A gate with no `--cookie-name` is using oauth2-proxy's default `_oauth2_proxy`. If two of them
|
||||
share that default **and** a zone-wide `--cookie-domain=.home.keithjackson.dev`, they are
|
||||
evicting each other, and the symptom is intermittent and depends on which app you visited last.
|
||||
|
||||
## Likely causes, most likely first
|
||||
|
||||
1. **Cookie collision between two gates on one domain.** Default cookie names plus a zone-wide
|
||||
cookie domain. Signature: high `No valid authentication`-to-`AuthSuccess` ratio, sessions
|
||||
dying seconds after being created, symptom depends on browsing order, and *nothing* wrong at
|
||||
Keycloak. Fix is a unique `--cookie-name` and a host-scoped `--cookie-domain` per gate — see
|
||||
`decisions/2026-09-29-crafty-oauth2-proxy-signin.md`, which found and fixed exactly this.
|
||||
2. **The cookie is too large to come back.** With `--pass-access-token` the session carries the
|
||||
access token; a fat Keycloak token pushes the cookie past 4 KB and it gets split into
|
||||
`_oauth2_proxy_0`, `_oauth2_proxy_1`… If a chunk is dropped the session is unreadable and the
|
||||
gate logs the same line. Look for numbered cookie names in the log, and for a session that
|
||||
never survives a single navigation.
|
||||
3. **The XHR path is not the navigation path.** The app's API is under a prefix the gate handles
|
||||
differently, or the app issues cross-origin XHRs the cookie does not accompany. Signature:
|
||||
`/UI/...` returns 200 and `/api/...` returns 302 in the *same second*, which is precisely the
|
||||
jackett trace above.
|
||||
4. **Session lifetime shorter than the app's polling interval,** with no refresh. The jackett
|
||||
session above was a 5-minute session. If `refresh_token:true` is present and sessions still
|
||||
die early, the refresh is failing, not missing.
|
||||
5. **Group check, not authentication.** `--allowed-group` is a floor: a valid Keycloak session in
|
||||
the wrong group is refused *after* a successful login, which reads as a loop. Look for 403 and
|
||||
for the `groups:[...]` list on the `[AuthSuccess]` line.
|
||||
6. **The app, not the gate.** `[AuthSuccess]` followed by the app's own login page or 401.
|
||||
|
||||
## When to escalate or stop
|
||||
|
||||
- **Stop and hand over if the fix is a manifest change.** Everything above is read-only
|
||||
diagnosis. Changing a `--cookie-name`, adding `--metrics-address`, or editing an alert goes
|
||||
through a branch and a PR; `kubectl edit` on a gate is drift that the next Flux reconcile
|
||||
reverts, and in the meantime you have an unreproducible cluster.
|
||||
- **Stop if you find yourself about to print a Secret.** The cookie-secret and client-secret live
|
||||
in `oauth2-proxy-credentials` and the per-app equivalents. Compare *key names* and whether two
|
||||
Secrets differ; never dump `.data` or `.stringData`. A password has been leaked into a
|
||||
transcript here exactly this way.
|
||||
- **Escalate to the platform owner if Keycloak itself is the suspect** — check `KeycloakDown` and
|
||||
`KeycloakDatabaseDown` first (`runbooks/keycloak-down.md`, `runbooks/keycloak-db.md`). In the
|
||||
2026-09-30 incident Keycloak was healthy throughout and blaming it would have cost days.
|
||||
- **The honest limit of this page.** With no metrics from the sidecars there is no history beyond
|
||||
Loki's **72h** retention (`platform/monitoring/loki.yaml`, `retention_period: 72h`). If the
|
||||
report is older than three days, the evidence is gone and the only remaining move is to
|
||||
reproduce it. That is the cost of the gap `OAuth2ProxyGateUnobserved` is tracking, and it is
|
||||
the reason to close it rather than silence it.
|
||||
@@ -0,0 +1,283 @@
|
||||
# plex-quicksync — making Plex actually use Intel Quick Sync, and proving it did
|
||||
|
||||
**Alert:** none, and none is possible — see "How to verify" below. This is an **operator procedure**,
|
||||
not an incident page. It is indexed here because the feature it completes is **inert until a human
|
||||
clicks one button**, and a switched-off feature that looks shipped is exactly what this page exists
|
||||
to prevent.
|
||||
**Files:** `catalog/plex.yaml` (`hardware.gpu: intel-quicksync`), `apps/wave-0/plex.yaml` (the
|
||||
rendered golden — the `dri` hostPath volume and the `/dev/dri` mount), `catalog/render.py` (the
|
||||
mechanism), `schemas/renderer-contract.md` `### hardware` (the contract).
|
||||
**Related:** `runbooks/plex-claim.md` — the other Plex procedure with a manual last mile.
|
||||
|
||||
## What this means here
|
||||
|
||||
Plex transcodes in **software** by default. On the UHD Graphics 630 iGPU of an i7-9700T, Quick Sync
|
||||
moves that work off the CPU — which matters on a three-node cluster where the CPU is also running
|
||||
everything else. Two independent things have to be true before that happens, and **only one of them
|
||||
is in git**:
|
||||
|
||||
| Half | Who does it | State |
|
||||
|---|---|---|
|
||||
| The render device is inside the container at `/dev/dri` | **GitOps.** `hardware.gpu: intel-quicksync` in `catalog/plex.yaml` renders a `dri` hostPath volume (`/dev/dri`, `type: DirectoryOrCreate`) and a read-write `/dev/dri` volumeMount on the app container. Flux applies it. | Done by merging that change. |
|
||||
| Plex is *told* to use it | **The operator, by hand, in the Plex web UI.** | **NOT done.** `HardwareAcceleratedCodecs` was **absent** from `Preferences.xml` when this page was written — the toggle has never been on. |
|
||||
|
||||
**There is no GitOps path to the second half, and that is verified rather than assumed.** A bounded
|
||||
grep across the running image's entire init surface — `/etc/cont-init.d`, `/etc/services.d`,
|
||||
`/etc/fix-attrs.d`, `/plex-common.sh`, `/usr/local/bin` — found **zero** `PLEX_PREFERENCE*` support.
|
||||
Every `setPref` the image performs is one of six hardcoded keys (`MachineIdentifier`,
|
||||
`ProcessedMachineIdentifier`, `PlexOnlineToken`, `customConnections`, `allowedNetworks`,
|
||||
`TranscoderTempDirectory`), none of them the transcode toggle. No environment variable, no ConfigMap
|
||||
and no `postBuild` substitution can set it. So do not go looking for the clever way to automate this;
|
||||
there isn't one, and the search has already been done.
|
||||
|
||||
What the device mount does **not** do, so nobody reads more into a green gate than is there: it does
|
||||
not enable hardware transcoding, it does not prove the device can be *opened* from inside a
|
||||
non-privileged pod (see "Likely causes" #2), and it renders no `nodeSelector` — Plex can land on any
|
||||
of the three nodes and all three carry the device.
|
||||
|
||||
## Running the commands in this runbook
|
||||
|
||||
`kubectl` does not work on the operator workstation; the toolchain lives only in the pinned dev
|
||||
container. Every command below runs as:
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
||||
|
||||
Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
|
||||
picks a current-context **per invocation**, and that default has already changed once. Do not run two
|
||||
`dev.sh` invocations at once.
|
||||
|
||||
**Never `cat` `Preferences.xml`.** It contains `PlexOnlineToken`, a plex.tv account credential. Every
|
||||
read of that file below greps a single named key — do not widen one.
|
||||
|
||||
## First commands
|
||||
|
||||
**1. Is the device mounted into the pod at all?** (Proves Flux applied the change, not that it works.)
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
|
||||
-o jsonpath="{.items[0].spec.containers[0].volumeMounts[*].mountPath}"; echo'
|
||||
```
|
||||
|
||||
`/dev/dri` in that list ⇒ the GitOps half landed. Absent ⇒ the running pod predates the merge; compare
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0 \
|
||||
-o jsonpath="{.status.lastAppliedRevision}"; echo'
|
||||
```
|
||||
|
||||
against the merge commit before blaming anything else.
|
||||
|
||||
**2. Is the device actually visible inside the container?**
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex exec deploy/plex -- ls -l /dev/dri'
|
||||
```
|
||||
|
||||
Expect `renderD128` (usually `card0` too), mode `crw-rw----`, group `render`. An **empty directory**
|
||||
here is the `DirectoryOrCreate` degradation: the kubelet created an empty `/dev/dri` because the node
|
||||
had none. That is the designed-safe outcome — the app tier still reconciles — but it means software
|
||||
transcoding, silently. Find which node the pod is on and ask why that node lost its iGPU:
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide'
|
||||
```
|
||||
|
||||
**3. Can Plex *open* it?** This is the question the manifest cannot answer — see "Likely causes" #2.
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex exec deploy/plex -- \
|
||||
bash -c "exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED"'
|
||||
```
|
||||
|
||||
`RW-OPEN-DENIED` (or `Operation not permitted` / `Permission denied`) with the device plainly visible
|
||||
in step 2 is the **device-cgroup** failure, not a file-permission failure. The two look identical from
|
||||
the outside and are not the same problem.
|
||||
|
||||
**4. Is the toggle on?** Named key only — never the whole file:
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex exec deploy/plex -- \
|
||||
grep -o "HardwareAcceleratedCodecs=\"[01]\"" \
|
||||
"/config/Library/Application Support/Plex Media Server/Preferences.xml" || echo ABSENT'
|
||||
```
|
||||
|
||||
`ABSENT` ⇒ never enabled (the state this page was written in). `="0"` ⇒ explicitly off. `="1"` ⇒ on,
|
||||
and the remaining question is whether it *works* — go to "How to verify".
|
||||
|
||||
## The one manual step
|
||||
|
||||
Plex web UI → **Settings → Transcoder → "Use hardware acceleration when available"** → tick → Save.
|
||||
(It sits under *Show Advanced*.) A transcode already in flight is unaffected; the next one picks it up.
|
||||
|
||||
The underlying preference key is **`HardwareAcceleratedCodecs`**. That name was confirmed by
|
||||
extracting strings from the 23 MB `Plex Media Server` binary in the running container — **not** from
|
||||
recall and not from a forum post. Real neighbouring keys from the same binary, so you can recognise
|
||||
them and not invent them:
|
||||
|
||||
| Key | What it is |
|
||||
|---|---|
|
||||
| `HardwareAcceleratedCodecs` | the toggle this page is about |
|
||||
| `HardwareAcceleratedEncoders` | hardware **encode**, as distinct from decode |
|
||||
| `HardwareDevicePath` | how Plex is pointed at a *specific* render node — relevant only if a host ever exposes more than one |
|
||||
| `TranscoderToneMapping`, `TranscoderToneMappingAgorithm` | HDR→SDR tone mapping. **That misspelling of "Algorithm" is upstream's, in the binary. It is not a typo in this page — do not "fix" it** here or anywhere else; the corrected spelling is not a key Plex reads. |
|
||||
| `TranscoderHEVCEncoding` | HEVC output |
|
||||
|
||||
**`HardwareAcceleratedCodecsEnableHEVC` does not exist.** It is widely cited and it is not in the
|
||||
binary. Recorded explicitly so nobody spends an evening hunting for it or, worse, writes it into
|
||||
`Preferences.xml` and concludes hardware transcoding is broken.
|
||||
|
||||
Editing `Preferences.xml` by hand is **not** the procedure. It is app state on the replicated Longhorn
|
||||
tier, it is in no git commit, and the UI writes the key correctly. Use the UI.
|
||||
|
||||
## How to verify — and why there is no alert
|
||||
|
||||
Start a playback that **forces a transcode** (a client that cannot direct-play the source, or pick a
|
||||
lower quality explicitly), then read Plex's own **Dashboard → Activity**:
|
||||
|
||||
- **`Transcode (hw)`** on the session ⇒ hardware transcoding. **This is the proof.**
|
||||
- plain **`Transcode`** ⇒ software. The feature is not working, whatever the manifest says.
|
||||
|
||||
**That dashboard is the *only* proof available, and nothing else will ever tell you.**
|
||||
`catalog/plex.yaml` declares no `metricsEndpoint`, so Plex renders **no ServiceMonitor at all** —
|
||||
there is no scrape, no transcode-mode metric, and therefore no alert that could fire. Do not open work
|
||||
to "alert on software-transcode fallback": with no metric behind it such a rule would be permanently
|
||||
inert, and this repo has already been burned at scale by rules that evaluated against nothing while
|
||||
looking like coverage (`runbooks/README.md`, "Alerts you should not trust the silence of"). If that
|
||||
gap is ever worth closing, the honest order is a real exporter first and a rule second.
|
||||
|
||||
Corroborating, not a substitute — the server log on the config PVC:
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex exec deploy/plex -- \
|
||||
sh -c "grep -i -m20 -e hardware -e vaapi -e hwdevice \
|
||||
\"/config/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log\""'
|
||||
```
|
||||
|
||||
Treat those strings as advisory: they are Plex's, not this repo's, and they move between versions.
|
||||
Absence of a match is **not** evidence of software transcoding; a VA-API device open is good evidence
|
||||
of hardware. The dashboard remains the verdict.
|
||||
|
||||
## Likely causes, most likely first
|
||||
|
||||
When the dashboard still says plain `Transcode`:
|
||||
|
||||
1. **The toggle is off.** Overwhelmingly the most likely answer, and the reason this page exists.
|
||||
First-command #4 returns `ABSENT` or `="0"`. Remedy: the manual step above. Check this **before**
|
||||
investigating anything technical.
|
||||
|
||||
2. **containerd's device cgroup is refusing the `open()` — UNVERIFIED, and the largest remaining
|
||||
technical risk in this feature.** Whether a non-privileged pod may `open()` a bind-mounted
|
||||
character device on this k3s/containerd build **has not been established**; it can only be settled
|
||||
by deploying and trying. The container inits as uid 0, which settles *file* permissions, but
|
||||
**uid 0 does not bypass the cgroup device gate.** Symptom: step 2 shows `renderD128` present and
|
||||
correctly owned, step 3 returns `RW-OPEN-DENIED`.
|
||||
**The documented fallback is the Intel Device Plugin for Kubernetes** (`gpu.intel.com/i915` as a
|
||||
resource request, no hostPath). Because the ServiceDefinition field expresses *intent* rather than
|
||||
a device path, that migration is a **`catalog/render.py` change plus a re-render — with no
|
||||
ServiceDefinition edit anywhere**; `catalog/plex.yaml` keeps saying exactly what it says today.
|
||||
**`privileged: true` is not an available escape hatch:** Kyverno `restrict-host-namespaces` is in
|
||||
**Enforce** mode and rejects it in app namespaces. Do not reach for it.
|
||||
|
||||
3. **The pod landed on a node without the device.** Step 2 shows an empty `/dev/dri`. All three prod
|
||||
nodes carried `renderD128` with `i915` bound when this was measured (2026-09-30), so this means a
|
||||
node changed — a rebuild, a BIOS reset, or a driver regression. `DirectoryOrCreate` is why the app
|
||||
tier still came up; the cost is that the loss is silent. **Fix the node, do not tighten the mount:**
|
||||
`Directory` would stall `apps-wave-0` (`wait: true`, with `apps-wave-1` depending on it) for a
|
||||
feature that is not load-bearing.
|
||||
|
||||
4. **The image changed under you.** See "The floating-image caveat". A `:latest` pull can remove the
|
||||
mechanism this feature leans on with no commit in this repo.
|
||||
|
||||
5. **The session was never transcoding.** Direct Play / Direct Stream never touches the transcoder, so
|
||||
`Transcode (hw)` cannot appear. Confirm the session actually says `Transcode` before treating it as
|
||||
a hardware failure.
|
||||
|
||||
6. **The source needs a codec Quick Sync on this generation cannot handle** — e.g. AV1 decode, which
|
||||
Coffee Lake's UHD 630 has no engine for. Plex falls back to software *per stream*. Symptom: some
|
||||
sessions show `(hw)` and others do not. Not a fault.
|
||||
|
||||
## The floating-image caveat
|
||||
|
||||
Everything this page states about the image's behaviour — the `cont-init.d` hook that adds the app
|
||||
user to the device's group, the absence of `PLEX_PREFERENCE*`, the six `setPref` keys, the
|
||||
`HardwareAcceleratedCodecs` string — was read from digest **`sha256:e0ab27395614…`**
|
||||
(`plexmediaserver 1.43.4.10903`).
|
||||
|
||||
The deployment runs **`plexinc/pms-docker:latest` with `imagePullPolicy: Always`**. A silent pull can
|
||||
therefore change the behaviour this feature depends on **with no commit in this repo**, while the
|
||||
manifest still looks identical. Symptom to watch for: hardware transcoding that worked and quietly
|
||||
stopped, with no git change to point at.
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
|
||||
-o jsonpath="{.items[0].status.containerStatuses[0].imageID}"; echo'
|
||||
```
|
||||
|
||||
If that digest is not the one above, **re-verify the hook before trusting this page**:
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex exec deploy/plex -- ls /etc/cont-init.d/'
|
||||
```
|
||||
|
||||
`45-plex-hw-transcode-and-connected-tuner` still present ⇒ the group mechanism is still the one
|
||||
described here. Gone or renamed ⇒ stop and re-measure. In particular the "no `supplementalGroups`
|
||||
needed" conclusion in `catalog/plex.yaml` and `schemas/renderer-contract.md` is only true while that
|
||||
hook exists.
|
||||
|
||||
## When to escalate or stop
|
||||
|
||||
- **Do not add `supplementalGroups` or a `podSecurityContext` to `catalog/plex.yaml`.** It would be
|
||||
**inert**, and it would mislead the next person. Measured in-container 2026-09-30: the image's
|
||||
`cont-init.d` hook stats each `/dev/dri` char device and `usermod -a -G`s the `plex` user into its
|
||||
group *inside* the container, and the service then execs via `s6-setuidgid plex`, which **replaces**
|
||||
the process group list from `/etc/group` rather than appending — `Plex Media Server` runs with
|
||||
`Groups: 100 1000`, root's gid `0` (which PID 1 does hold) gone. Anything Kubernetes supplies is
|
||||
discarded before Plex starts. `render.py --selftest` asserts that absence deliberately.
|
||||
- **Do not reach for `privileged: true`.** Kyverno `restrict-host-namespaces` is Enforce and will
|
||||
reject it. If the cgroup gate is the blocker, the answer is the Intel device plugin (cause #2).
|
||||
- **Do not `kubectl edit`/`patch` the Deployment to add or change the device mount.** Flux reconciles
|
||||
`main`; a hand-applied change is drift that the next reconcile reverts or, worse, adopts. Mutating
|
||||
`kubectl` and `flux reconcile` are **operator** actions, and the durable change is a git change
|
||||
through a branch, a PR and a green gate.
|
||||
- **Do not edit `Preferences.xml` on the PVC**, never `cat` it, and never `sops -d` anything into a
|
||||
terminal. It holds `PlexOnlineToken`.
|
||||
- **Do not delete the `plex-config` PVC to "start clean".** It is the library database, metadata and
|
||||
watch state (`policy: daily-offcluster`). The transcode toggle is one preference, not a reason to
|
||||
lose the DB.
|
||||
- **Stop and get the operator** if the device cannot be opened from the pod (cause #2 confirmed). The
|
||||
next step is a platform change — the Intel device plugin — with its own PR, not more poking at this
|
||||
workload.
|
||||
|
||||
## Known gaps, deliberately not fixed here
|
||||
|
||||
Recorded so they are findable, and so nobody assumes they were overlooked:
|
||||
|
||||
- **Transcode scratch is on the wrong medium, and it is invisible.** The image sets
|
||||
`TranscoderTempDirectory=/transcode`, which is **not a volume** — it is the container's writable
|
||||
overlay, i.e. the node's `/data`. That is NVMe on prod-2 but **SATA on prod-1 and prod-3**, so
|
||||
transcode scratch performance changes silently whenever the pod is rescheduled. Both candidate fixes
|
||||
need a **schema** change and are therefore out of scope here: pin Plex to prod-2 (needs a
|
||||
`nodeSelector` field the schema does not have), or give `/transcode` a `medium: Memory` emptyDir and
|
||||
transcode in RAM (~31 GiB per node; needs an emptyDir field the schema does not have). The RAM option
|
||||
is the better one, because it removes the node asymmetry outright instead of working around it.
|
||||
- **GuC/HuC firmware is not loaded** (`i915.enable_guc=0`). The cost is low-power H.264 **encode**
|
||||
quality only — decode, and the feature on this page, are unaffected. Changing it needs a kernel
|
||||
parameter and a **serialized rolling reboot**, which is an operator action (CLAUDE.md §5), and this
|
||||
cluster has a documented history of that going badly (a prod node hung on a warm reboot on
|
||||
2026-09-03). Not worth coupling to this change.
|
||||
|
||||
---
|
||||
Source of truth: `runbooks/plex-quicksync.md`. Index: `/runbooks/`
|
||||
The node measurements (all three prod nodes) and the in-container image measurements cited here were
|
||||
taken **2026-09-30** against `plexinc/pms-docker` digest `sha256:e0ab27395614…`
|
||||
(`plexmediaserver 1.43.4.10903`). Hardware transcoding had **not** been observed working at the time
|
||||
of writing, and the Plex-side toggle was still unset.
|
||||
Reference in new issue
Block a user