docs(obs): record why CI is red — approval-policy on the one conformance entry

The gate job reported failure while the local run was rc 3 with nothing failed. Measured cause is a single sensitive path (the KNOWN_EXTERNAL_MONITORS entry) with no non-author approving review, which ci/approval-policy.sh blocks fail-closed by design. Also records the general trap: approval-policy short-circuits to N/A off a pull_request event, so a clean LOCAL gate is a false reassurance for any branch touching a sensitive path. Only the PR run tells the truth. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
feat(obs): wire the gate-sidecar scrape, and fix the alert that would have gone silent doing it
2026-10-01 22:12:19 -05:00 · 2026-10-01 21:56:07 -05:00 · 2026-10-01 21:40:37 -05:00 · 2026-09-30 10:55:19 -05:00
17 changed files with 1095 additions and 786 deletions
@@ -1,104 +0,0 @@
 ---
 # Flux Kustomization: Intel GPU device plugin — reconciles ./platform/intel-gpu-plugin.
 # Makes the three prod nodes advertise `gpu.intel.com/i915`, which is the resource a later PR will
 # have `hardware.gpu: intel-quicksync` request in place of today's non-working /dev/dri hostPath.
 #
 # WHY ITS OWN KUSTOMIZATION, AND NOT ONE LINE IN platform/kustomization.yaml
 #
 # The cheap option was a line in the shared `platform` tier plus a new platform/ directory, and
 # neither path is sensitive in infra/gitea/approval-policy.json, so it would have needed no human
 # reviewer on the policy's terms. It is still the wrong place, for the reason the repo already
 # wrote down in platform/kustomization.yaml when Cilium was split out of that tier:
 #
 #   * `platform` is `wait: true` over eleven-plus components, so its Ready condition cannot tell
 #     you "the GPU plugin is wedged" from "longhorn is slow".
 #   * The only emergency valve would be `flux suspend kustomization platform` — freezing storage,
 #     TLS, DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a device
 #     plugin.
 #   * The tier's 10m interval / 5m timeout are explicitly called out there as WRONG for a 3-node
 #     DaemonSet rollout. This is a 3-node DaemonSet rollout.
 #
 # And the consequence is worse here than for an average component, because of what sits downstream:
 # `platform` gates platform-config, which gates apps-wave-0 (wait:true), which gates apps-wave-1.
 # A plugin DaemonSet that never went healthy inside that tier would stall every app on "dependency
 # not ready" — i.e. a component added to fix a SILENT NO-OP (software transcoding) would have
 # acquired the ability to cause a REAL OUTAGE. That trade is the whole argument.
 #
 # The usual objection to this placement is the approval cost, and it does not apply: Gitea enforces
 # `required_approvals: 1` with `block_admin_merge_override: true` on EVERY PR here regardless of
 # path, and authors cannot self-approve. So this change needed the operator's approval either way;
 # touching clusters/** costs nothing extra and buys a blast radius of exactly one component.
 #
 # dependsOn: [cilium], WHICH IS THE MINIMUM AND ALSO THE EARLIEST
 #
 # Not platform-config (what headlamp/crafty/homepage/oauth2-proxy use) — those are leaf UIs that
 # genuinely need Keycloak + wildcard TLS + split-horizon DNS + Gateway. This plugin needs none of
 # it. It never uses the network and never talks to the API server; it talks to the local kubelet
 # over a unix socket. Its only real prerequisite is a working CNI, so its pod can get a sandbox at
 # all, and `cilium` already depends on `flux-system` (the root Kustomization that applies the
 # cluster-vars ConfigMap this one substitutes from), so that edge is transitively satisfied too.
 #
 # Depending on `cilium` rather than `platform` also gets the resource advertised EARLIEST — this
 # reconciles in parallel with the `platform` tier instead of behind all eleven of its components.
 # That matters on a cold start once the consumer PR lands: a Plex pod requesting a resource no node
 # advertises is Unschedulable, and apps-wave-0 is wait:true. Reconciling early shrinks that window
 # instead of widening it. It is not a GUARANTEE — see the ordering note below, which is a real
 # constraint on the next PR and not a formality.
 #
 # Kyverno (in `platform`) is deliberately NOT a dependency: every policy that could match this
 # workload is scoped to namespaces labelled app-namespace=true, and this namespace is labelled
 # "false" (platform/intel-gpu-plugin/namespace.yaml). So there is no admission race to lose.
 #
 # wait: true — A HEALTH SIGNAL, NOT A GATE. This is the one place this component departs from the
 # headlamp/crafty leaf template, and the departure is deliberate. Those run wait:false because a
 # leaf UI can be legitimately not-Ready for a long time during a cold start while it waits on
 # Keycloak/DNS/TLS, so its health is not a useful signal. This DaemonSet has no such dependency: it
 # reads sysfs and opens a socket, so on healthy nodes it should be Ready in seconds, and
 # not-Ready genuinely means BROKEN. wait:true turns `flux get kustomization intel-gpu-plugin` into
 # a one-line answer to "did the plugin come up on all three nodes", which is exactly the question
 # the operator must answer before the consumer PR may merge.
 #
 # NOTHING IN THE GRAPH dependsOn THIS, so wait:true gates nothing — the blast radius of an unhealthy
 # plugin is this Kustomization's own Ready condition and nothing else. Anyone later adding
 # `dependsOn: [intel-gpu-plugin]` is choosing to make plugin health gate their thing; for the app
 # waves that would hand a device plugin the power to stall every app, which is the thing this file
 # exists to prevent. Think before adding that edge.
 #
 # ORDERING — DO NOT REORDER THE STACK. The renderer change (catalog/render.py emitting
 # `gpu.intel.com/i915: 1` for `hardware.gpu: intel-quicksync` instead of the hostPath) must NOT
 # land until this plugin is merged AND confirmed advertising the resource on all three nodes. If
 # the renderer requests a resource no node advertises, the Plex pod is Unschedulable; apps-wave-0 is
 # wait:true and apps-wave-1 dependsOn it, so that stalls the whole app tier — turning today's silent
 # no-op into a real outage. This PR's only job is to make the resource exist.
 apiVersion: kustomize.toolkit.fluxcd.io/v1
 kind: Kustomization
 metadata:
  name: intel-gpu-plugin
  namespace: flux-system
 spec:
  # Static vendored manifests with nothing to re-render; kustomize-controller is source-triggered,
  # so this is only the drift-correction floor.
  interval: 10m
  retryInterval: 2m
  # 10m, not the 5m the `platform` tier uses: a 3-node DaemonSet rollout under `wait: true` can
  # exceed 5m on a cold start (image pull on three nodes), and the Cilium split-out comment calls
  # out that exact mismatch as one of its reasons for leaving the shared tier.
  timeout: 10m
  prune: true
  # See the header: a health SIGNAL for a component nothing depends on, not a gate.
  wait: true
  dependsOn:
    - name: cilium   # the CNI — the only real prerequisite; transitively gives us flux-system too
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./platform/intel-gpu-plugin
  # Retained for parity with every other tier / future overlay knobs. This component references no
  # ${VARS} today, and its manifests contain no `$` literals for envsubst to eat (unlike
  # clusters/base/runbooks.yaml, which omits substitution for exactly that reason).
  postBuild:
    substituteFrom:
      - kind: ConfigMap
        name: cluster-vars
  # No SOPS decryption patch: this component carries no Secret, so clusters/dev adds nothing for it
  # (same as homepage/runbooks, unlike headlamp/crafty/oauth2-proxy/backup).
@@ -12,15 +12,6 @@ resources:
  # interval/timeout, and `flux suspend kustomization cilium` as an emergency valve that does not
  # freeze the other eleven platform components. See clusters/base/cilium.yaml.
  - cilium.yaml          # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
  # Intel GPU device plugin — its OWN Kustomization (dependsOn cilium, wait:true), NOT a line in
  # platform/kustomization.yaml. Same reasoning as the Cilium split-out, applied to a leaf: inside
  # that wait:true tier a wedged 3-node DaemonSet would be indistinguishable from eleven unrelated
  # components AND would stall platform-config + every apps wave on "dependency not ready" — so a
  # component added to fix a silent no-op (Plex software transcoding) could cause a real outage.
  # wait:true here is a HEALTH SIGNAL, not a gate: nothing in the graph dependsOn it.
  # It advertises `gpu.intel.com/i915`; the renderer change that REQUESTS that resource must land
  # strictly later. See clusters/base/intel-gpu-plugin.yaml.
  - intel-gpu-plugin.yaml # Flux Kustomization -> ./platform/intel-gpu-plugin (dependsOn cilium)
  - platform.yaml        # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
  - platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
  # SECOND-ORDER CDI configs — CRs whose CRD is created by a controller AT RUNTIME, not by any
@@ -129,6 +129,22 @@ KNOWN_EXTERNAL_MONITORS: dict[str, str] = {
        "a PodMonitor, so it resolves against POD ports, not Service ports — the Flux controller "
        "Deployments (and their `http-prom` container port) are installed by the Flux bootstrap "
        "manifests, not authored here. Live on prod: job flux-system/flux-system, 6 targets up.",
    "platform/oauth2-proxy/podmonitor.yaml:oauth2-proxy-sidecars":
        "a PodMonitor, so it resolves against POD ports, not Service ports. Unlike the other "
        "entries here the target is NOT chart-owned — the `metrics` (44180) container port is "
        "authored in this repo, on the oauth2-proxy sidecar in apps/wave-0/{jackett,qbittorrent,"
        "pihole}.yaml and apps/wave-1/{radarr,sonarr}.yaml. It is unauditable here because NO "
        "Service exposes 44180 BY DESIGN: adding one would mean changing catalog/render.py and "
        "every apps/wave-* render and golden, to publish an endpoint only Prometheus should dial. "
        "A PodMonitor needs no Service, which is the whole reason it was chosen. What assertion "
        "(2) would have checked was instead MEASURED on prod 2026-10-01, read-only: the pods "
        "declare ports [('http', 4180), ('metrics', 44180)], the selector label "
        "homelab.keithjackson.dev/auth-mode=forward-auth is on exactly those five gate pods, and "
        "port-forwarding 44180 returns http=200 with 10208 bytes of oauth2_proxy_* metrics. "
        "CAVEAT, deliberately recorded: those targets are DOWN on prod, because the app "
        "namespaces' default-deny NetworkPolicy drops the scrape from ns monitoring (same bug and "
        "fingerprint as platform/longhorn/networkpolicy-metrics.yaml documents). The port name "
        "resolves; the network path does not yet.",
 }
 MONITOR_KINDS = ("ServiceMonitor", "PodMonitor")
@@ -0,0 +1,289 @@
 ---
 date: 2026-09-30
 branch: feat/oauth2-proxy-auth-failure-visible
 title: The Jackett auth failure was invisible because the gates emit nothing — alert on the gap, not on a metric that does not exist
 ---
 ## What changed
 Jackett's UI broke in the way that costs the most: **HTTP 200 and a plausible page**. The
 operator's report named the property that matters — *"reloading the page never fixes the issue and
 the user is never re-prompted for login"*. Two sibling branches fix the auth defects. This one is
 about the fact that **no signal existed anywhere**, and it turns out that is not an oversight in
 the alert rules. It is an absence of data.
 **The measurement that decided the whole design** (prod, live Prometheus, 2026-09-30):
    count by (__name__) ({__name__=~"oauth2_proxy.*"})
      oauth2_proxy_requests_in_flight => 2
 Two series. Both from the **shared** forward-auth Deployment in the `oauth2-proxy` namespace.
 That is every oauth2-proxy metric this cluster holds. The six per-app gate sidecars — crafty,
 jackett, pihole, qbittorrent, radarr, sonarr — publish **nothing at all**: they run with
 `--http-address=0.0.0.0:4180` and no `--metrics-address`, so oauth2-proxy's metrics listener never
 starts; they declare only a `http:4180` container port; and no ServiceMonitor or PodMonitor in the
 cluster selects them.
 So the alert the brief asked for — *"a gated app is serving a sustained rate of auth redirects to
 a user who is authenticated"* — **is not writable**, and writing it anyway would have produced
 exactly the artefact this repo keeps getting burned by: a rule that is present, evaluating,
 permanently silent, and indistinguishable from health. It was not written.
 What was written instead alerts on the condition that made the incident invisible, from families
 that **are** scraped:
 - **`platform/oauth2-proxy/auth-visibility-alerts.yaml`** — `OAuth2ProxyGateUnobserved`:
      count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
        unless
      count by (namespace) (up{container="oauth2-proxy"})
  `for: 1h`, `severity: warning`. `kube_pod_container_info` is kube-state-metrics (a declared
  owner in `alert-metrics-have-scrape-targets`) and sees the container whether or not it exports
  anything; `up` is UNIVERSAL and carries a `container` label for any scrape target, so the rule
  needs no advance knowledge of what job name a future per-app scrape will use. Nothing in it
  touches `oauth2_proxy_*`, so it cannot be inert for the reason it is reporting. **It returns six
  namespaces against live prod right now** and excludes `oauth2-proxy`, which is scraped. It
  clears per-namespace as each gate gains a real scrape.
 - **`platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml`** — four states pinned:
  fires for an unscraped namespace, never for the scraped one, honours `for: 1h` (pending at 59m),
  and **stops** once a scrape appears under a *different* job name. Series shapes copied off live
  prod, not invented.
 - **`runbooks/oauth2-proxy-auth-failures.md`** — how to tell "the gate refused me" from "the group
  check refused me" from "the app refused me" from "the XHR silently got HTML", with the
  2026-09-30 jackett trace as the worked example. Registered in `runbooks/kustomization.yaml`'s
  `configMapGenerator` and listed in `runbooks/README.md`.
 - **`runbooks/README.md`** — a new row in *"Alerts you should not trust the silence of"* for the
  per-app sidecars, plus the measured reason.
 - **`platform/oauth2-proxy/README.md`** — what this component can and cannot see, and the exact
  three-part change that would close the gap.
 **Why the obvious alternative was rejected.** An alert on the *shared* proxy's
 `oauth2_proxy_requests_total{code="302"}` ratio is the one a reader expects to see here. It would
 need a new `SCRAPE_OWNERS` entry in `conformance/**` (a sensitive path needing human approval),
 and — decisively — **it would not have fired on this incident**: the shared instance is not in the
 data path for any sidecar-gated app, and its current pods export no `oauth2_proxy_requests_total`
 at all, having proxied nothing since they started (the counter exists in the TSDB only from an
 older ReplicaSet, `oauth2-proxy-78449b5657-*`, with `code` labels `302` and `401` and no `202`).
 Shipping it would have looked like coverage of the reported failure and been none.
 ## What I verified
 Everything below was measured against live prod through the pinned container, read-only. No
 `apply`, no `edit`, no `patch`, no `flux reconcile`.
 - **The full oauth2-proxy metric inventory.** `count by (__name__) ({__name__=~"oauth2_proxy.*"})`
  → `oauth2_proxy_requests_in_flight => 2`. `count by (__name__) ({job="oauth2-proxy"})` returns
  go_*/process_*/promhttp_* plus that one family and nothing else. **Only the shared instance is
  scraped**: `up{job="oauth2-proxy"}` → 2 targets, `10.0.1.189:44180` and `10.0.2.47:44180`, both
  `1`, `namespace="oauth2-proxy"`.
 - **The per-app sidecars are unscraped, and unscrapable as configured.** Every pod carrying an
  `oauth2-proxy` container, with its declared ports: the two shared replicas have
  `[('http',4180),('metrics',44180)]`; crafty, jackett, pihole, qbittorrent, radarr and sonarr all
  have `[('http',4180)]` only. The live jackett sidecar's args contain no `--metrics-address`.
  `kubectl get servicemonitors,podmonitors -A` lists 16 ServiceMonitors and 5 PodMonitors; the
  only oauth2-proxy entry is `oauth2-proxy/oauth2-proxy`.
 - **No existing alert is inert *for this reason*, because no alert references these metrics at
  all.** `grep -rn oauth2_proxy --include=*.yaml` over the tree hits only cookie-name strings in
  `platform/crafty/`. The 39 live PrometheusRules include `platform-health-identity`, whose four
  rules are about Keycloak readiness and the client reconciler and are all keyed on `kube_*`
  (scraped, fine). **The inertness here is the absence of a rule, not a broken one** — which is
  why the runbook README's inert-alerts table gains a row phrased like the etcd one.
 - **Keycloak cannot express it either.** `count by (__name__) ({__name__=~"keycloak_.*"})` →
  empty, and empty over `[30d]`. The `keycloak-service` scrape yields JVM/Agroal/JGroups metrics
  only; there are no user-event or login counters, so "a burst of re-logins for one client" is not
  observable from the IdP side.
 - **`oauth2_proxy_requests_total` has existed historically but not on the current pods.**
  `count_over_time(oauth2_proxy_requests_total[30d])` returns three series, all from
  `oauth2-proxy-78449b5657-*` (a prior ReplicaSet): `code="302"` on both replicas and `code="401"`
  on one, ~634 samples each (~5h of a 30d window). **No `code="202"` ever.**
 - **The alert expression fires against live data.** Run verbatim on prod:
  `count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by
  (namespace) (up{container="oauth2-proxy"})` → `pihole, jackett, crafty, qbittorrent, sonarr,
  radarr`, value 1 each; `oauth2-proxy` correctly absent.
 - **The log signal exists and is collected.** Loki (`loki-0`, SingleBinary) and a 3-node Alloy
  DaemonSet are running, and Alloy labels pod logs with `namespace`/`container`. LogQL over 24h,
  `{container="oauth2-proxy"}`: crafty 12036 lines, jackett 11669, oauth2-proxy 34559, pihole
  11520, qbittorrent 11520, radarr 11763, sonarr 11520. Filtered to `|= "No valid authentication
  in request"`: **crafty 81, jackett 39, radarr 67**; to `|= "AuthSuccess"`: **crafty 1, jackett
  4, radarr 1**. Those ratios (81:1, 10:1, 67:1) are the incident's signature and are in the
  runbook as the thing to compare against.
 - **Why `kubectl logs` hid it.** The jackett sidecar's stdout is ~11.5k lines/day and almost all
  of it is `GET "/ping" ... 200 2` from the kubelet probe, twice every 15s. A `--tail=40` shows
  forty `/ping` lines and looks perfectly healthy. The runbook's commands all filter.
 - **Runbook registration, both halves.** `- oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md`
  is in `runbooks/kustomization.yaml`'s `configMapGenerator.files`, and the page is listed under
  *Identity* in `runbooks/README.md`. The page clears the L0 quality floor (the four operator
  sections, >1500 bytes, mentions `dev.sh` and `--context k3s-prod`), and the alert's
  `runbook_url` points at exactly that name.
 - **Gate:** run in the pinned toolbox from this worktree with the `ENGINE` shim. Exit code and
  per-stage outcomes in the next section — read them there, not here.
 ## What I did NOT verify
 - **The alert has never fired in Alertmanager.** Nothing is applied; the PrometheusRule exists
  only in git on this branch. What is proven is that its expression returns the expected vector
  against live prod data and that promtool agrees about its behaviour over synthetic series.
  **Merging is what deploys it.**
 - **`OAuth2ProxyGateUnobserved` would NOT have fired *during* the 2026-09-30 jackett incident in
  any useful sense** — it describes a standing condition that has been true since the sidecars
  were deployed, so it would have been firing before, during and after. That is deliberate and it
  is the honest claim: it is a **gap tracker**, not an incident detector. It would have meant that
  on 2026-09-30 an operator asking "why did nothing alert?" had the answer already on the board
  instead of discovering it during the investigation. **No alert on this platform could have
  detected the incident itself, and none can today.**
 - **The alert that the brief actually asked for does not exist, and cannot be written yet.** The
  sustained-redirect-ratio rule needs `oauth2_proxy_requests_total` from the *per-app* sidecars.
  **Enabling that requires editing files owned by the two sibling branches** — the sidecar args in
  `apps/wave-0/jackett.yaml` (Mira Vance) and `apps/wave-0/{qbittorrent,pihole}.yaml`,
  `apps/wave-1/{radarr,sonarr}.yaml` (Theo Bright) — so it was **not attempted here**. The change
  is three things per app, together: `--metrics-address=0.0.0.0:44180`, a `metrics` containerPort
  plus Service port, and a ServiceMonitor. Once any one app has them, the rule belongs beside
  `OAuth2ProxyGateUnobserved`, and that app's namespace drops out of it automatically.
 - **The Loki alerting path is inert and I did not fix it.** Loki *has* a ruler stanza and the
  chart's `loki-sc-rules` sidecar is running with `LABEL=loki_rule`, `FOLDER=/rules` — so a
  ConfigMap-shipped LogQL rule would be delivered. But `GET /loki/api/v1/rules` answers **"unable
  to read rule dir /var/loki/rules/fake: no such file or directory"** (the ruler reads a different
  directory from the one the sidecar writes), and the rendered config contains **no
  `alertmanager_url`** — `grep -i 'alertmanager\|rule_path'` over the `loki` ConfigMap returns
  nothing. So a LogQL alert on `|= "No valid authentication in request"` — which *would* have
  caught this incident, and is the single highest-value follow-up here — **would evaluate and
  notify nobody**. Wiring it means `rulerConfig` (alertmanager_url + a matching rules directory)
  in `platform/monitoring/loki.yaml`, which is not a file this brief gave me. **I did not ship an
  inert rule ConfigMap to look productive.**
 - **Nothing was reproduced in a browser.** The 302-to-200-with-HTML mechanism is read off the
  sidecar access log and the response sizes (`/UI/Dashboard 200 44379`), not observed in devtools.
 - **Loki retention is 72h** (`platform/monitoring/loki.yaml`), so the 24h ratios above are a
  snapshot; anything older than three days is already gone. This is stated in the runbook because
  it bounds what an operator can still investigate.
 - **The gate's L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), so no runtime
  requirement touched by this branch is certified. See the exit code below — **rc 3 is
  UNDETERMINED, not green.**
 ### Follow-ups, in order of value
 1. **Wire the Loki ruler to Alertmanager** (`platform/monitoring/loki.yaml`). This is the only
   route to an alert that would have caught the actual 2026-09-30 failure, and the data is
   already being collected. The rule is one expression:
   `sum by (namespace) (rate({container="oauth2-proxy"} |= "No valid authentication in request" [15m]))
   > 5 * sum by (namespace) (rate({container="oauth2-proxy"} |= "AuthSuccess" [15m]))`.
 2. **Give one gate sidecar a metrics endpoint** and watch its namespace drop out of
   `OAuth2ProxyGateUnobserved`. That is the cheapest proof the alert is a tracker and not a nag,
   and it unlocks the ratio rule in Prometheus.
 3. **Consider whether `--set-xauthrequest` apps should get `--api-routes`** so API paths answer
   401 instead of 302. That is the fix that would make this failure *loud* rather than merely
   observable, and it belongs with whoever owns the sidecar args.
 ---
 # Round 2 — 2026-10-01: the scrape got wired, and found a bug in this branch's own alert
 Added to the same PR (#29) rather than a fourth concurrent branch. PR #28 merged in the meantime
 and `main` moved three PRs ahead; `origin/main` is merged in here, and the `runbooks/kustomization.yaml`
 conflict was resolved by **keeping both** runbook entries (mine and `oauth2-proxy-session`).
 ## What changed
 **`platform/oauth2-proxy/podmonitor.yaml` (new).** PR #28 gave the five fleet sidecars
 `--metrics-address=0.0.0.0:44180` and a named `metrics` port. No Service exposes 44180 on any of
 them, so a ServiceMonitor would have nothing to select — a **PodMonitor** targets pod ports directly
 and needs no renderer change. Selects on the pod label `homelab.keithjackson.dev/auth-mode:
 forward-auth`, which is on exactly the five fleet gate pods and absent from crafty's: the property,
 not an app list, so a sixth forward-auth app needs no edit.
 **`auth-visibility-alerts.yaml`: `up{container="oauth2-proxy"}` → `up{container="oauth2-proxy"} == 1`.
 This is a real bug fix, and it is the most important thing in this round.** A target that is
 discovered but failing still publishes `up`, with value **0**. The series exists, so
 `count by (namespace)` returned 1, `unless` removed the namespace, and `OAuth2ProxyGateUnobserved`
 would have **gone quiet while nothing was being scraped at all** — this branch's own disease,
 reproduced inside the alert written to cure it. Not hypothetical: the PodMonitor lands five targets
 whose scrape is dropped, so the bare form would have silently cleared on this very commit.
 **Two new unit-test scenarios** pinning that: a discovered-but-failing target must still fire, and a
 target that works and then breaks must re-fire rather than latch cleared.
 **Docs corrected rather than appended to.** The 2026-09-30 claim "the sidecars export no metrics at
 all" is now only half true and was rewritten in `runbooks/README.md`, `platform/oauth2-proxy/README.md`
 and the alert header, each showing the two-step history, because each step looked like the end of it.
 The runbook gains the policy-vs-no-target triage (step 1b) and the **radarr `/signalr` worked case**.
 **Deliberately NOT shipped: the ratio alert.** `oauth2_proxy_requests_total` now exists on the
 endpoint, but not in the TSDB, because the scrape is blocked. Writing the ratio rule today would make
 it inert for precisely the reason it was refused on 2026-09-30. It becomes writable the day a scrape
 succeeds, not the day a listener exists.
 ## What I verified
 - **The endpoint serves — by payload, not by flag.** Port-forwarded to the jackett pod's 44180:
  `http=200 bytes=10208`, exporting `oauth2_proxy_requests_in_flight`, `oauth2_proxy_requests_total`
  (`code="200" 3`, `code="302" 2`) and `oauth2_proxy_response_duration_seconds`. `--metrics-address`
  being in `args` and `/metrics` returning a body are different claims; both now hold.
 - **The scrape path is DROPPED, measured from the right place.** From grafana in ns `monitoring`
  (beside Prometheus, same policy position) to the jackett pod IP: `:44180` → `wget: download timed
  out`; `:4180` → **also** timed out; the shared proxy's `:44180` → 200 with `oauth2_proxy_*` present.
  Cause, read from the objects: ns jackett's `default-deny` is `podSelector: {}`,
  `policyTypes: [Ingress, Egress]`, with an `egress` rule and **no `ingress` key** (deny-all ingress);
  the only allowance is CNP `jackett-allow-gateway-ingress`, `fromEntities: [ingress]` to port 4180.
  The shared proxy is scrapable only because ns `oauth2-proxy` has **no NetworkPolicy at all**.
 - **No Service exposes 44180** on any of the five (enumerated).
 - **The pod label selector is exact:** `auth-mode=forward-auth` on all five fleet gate pods, absent
  from crafty's. Crafty's sidecar still has no `--metrics-address` (args read in full).
 - **Flux ordering forbids shipping the fix from here:** `oauth2-proxy dependsOn=['platform-config']`
  and `apps-wave-0 dependsOn=['platform-config']` — siblings, no ordering. A NetworkPolicy naming ns
  `jackett` from this tier would apply today and fail on a cold rebuild.
 - **The `== 1` guard is non-vacuous — proven by mutation.** Reverting it to the bare `up` makes the
  two new scenarios FAIL (`got: nil` at 1h5m and 2h5m) and leaves every other assertion passing;
  restoring it returns the suite to SUCCESS under pinned promtool v3.14.0.
 - **Gate:** see the next section for the exit code and skips.
 ## What I did NOT verify
 - **The PodMonitor has never been applied.** That it discovers the five pods is reasoned from the
  measured pod labels and port name, not observed — no target list has been read with it in place,
  because applying it is an operator action. **Expect `up == 0` on all five when it lands**, which is
  the documented state, not a regression.
 - **I did not fix the NetworkPolicy, and this PR therefore does not make the gates observable.** It
  makes them *discovered*, and makes the remaining gap loud instead of silent. The one narrow rule
  per app (ns monitoring → `auth-mode=forward-auth` → TCP 44180, per
  `platform/longhorn/networkpolicy-metrics.yaml`) has to be rendered with each app's own policy, in
  the catalog/renderer that is in flight on another branch. **It needs an owner.**
 - **Whether Prometheus will actually select a PodMonitor in ns `oauth2-proxy` targeting other
  namespaces.** `namespaceSelector.any: true` plus `podMonitorSelectorNilUsesHelmValues: false` and
  the fact that cross-namespace ServiceMonitors/PrometheusRules already work here say yes; it is not
  measured with this object.
 - **No session-lifecycle claim can be made from the current window.** `AuthSuccess` is **0** on both
  jackett and radarr since the PR #28 restart, so nobody has signed in and `--cookie-refresh=2m30s`
  has never executed a refresh. The cookie fix is unproven in the one dimension that matters most.
 - **The radarr `/signalr` finding is the coordinator's measurement, not mine** — I reproduced neither
  the 206/209 sample nor the jackett 0-in-13,612. They are recorded as reported.
 - **L2/L3 conformance still did not run** (`CLUSTER_AVAILABLE` unset). Nothing is certified.
 ### Round 2 addendum — CI is RED, by design, and the local gate cannot see why
 After pushing round 2, `gate / gate (pull_request)` reported **failure after 1m42s** while the local
 toolbox run was rc 3 with nothing failed. Measured cause, singular:
    sensitive paths touched:
      conformance/tests/l0-static/servicemonitor-ports-resolve/check.sh   (matched conformance/**)
    approving reviews: (no reviews at all)        author=gitea-admin
 `ci/approval-policy.sh` is fail-closed and exits 1 when a PR touches a `sensitive_paths` glob with no
 `APPROVED` review from a non-author — "a change to the gate cannot be gated by the gate". The single
 `KNOWN_EXTERNAL_MONITORS` entry is what trips it. **Nothing in L0/L1 failed.**
 **The local gate structurally cannot catch this, and said `approval-policy: green (N/A)`.** The stage
 short-circuits to N/A when `GITHUB_EVENT_NAME` is unset, which it always is off-CI. So for any branch
 touching a sensitive path, a clean local gate is a FALSE REASSURANCE — the PR run is the only honest
 signal. That is a general trap, not specific to this change, and it is worth knowing before someone
 concludes a red PR means broken code.
 To clear: a non-author approving review, then **re-run the gate** — approval does not re-trigger the
 status check, so the required context stays red until the job runs again. `required_approvals=1`
 already applied to every PR here, so the human was needed regardless; the conformance entry only
 turns the tick red while that is pending.
 The alternative is to drop the register entry, which means dropping `podmonitor.yaml` with it (the
 check fails without the declaration). I did not take it: the entry is the "declare the limit"
 mechanism the check explicitly asks for, it is shrink-only so it cannot rot quietly, and
 `flux-podmonitor.yaml` is the same case already in the register. But it is a real trade and the
 reviewer should make it knowingly.
@@ -1,198 +0,0 @@
 ---
 date: 2026-10-01
 branch: feat/intel-gpu-device-plugin
 title: Intel GPU device plugin, in its own Kustomization — the resource exists; nothing has run yet
 ---
 ## What changed
 Deployed the **Intel Device Plugin for Kubernetes (GPU)**, standalone DaemonSet, pinned at
 **v0.37.0**, so the prod nodes advertise `gpu.intel.com/i915`. This is the measured fix for the
 Plex hardware-transcoding blocker: the `/dev/dri` hostPath merged on `main` @ `cdf3e14` makes the
 device node *visible* and not *openable* (`EPERM` from the device cgroup at both uid 0 and uid
 1000; the container's runc `config.json` carries one deny-all rule and `linux.devices: None`). A
 hostPath is a filesystem operation and adds no cgroup allow rule. A device plugin's `Allocate`
 response is what makes kubelet populate `linux.devices` *and* the matching cgroup allow rule, and
 it is the only non-privileged way to get there — `privileged: true` is blocked by Kyverno
 `restrict-host-namespaces` under Enforce in app namespaces, and `plex` is labelled
 `app-namespace=true`.
 New files:
 - `platform/intel-gpu-plugin/intel-gpu-plugin.yaml` — the DaemonSet, **vendored** as
  `kubectl kustomize` output of upstream `deployments/gpu_plugin/base` at `v0.37.0`, under a
  provenance header carrying the exact re-vendor command and both sha256s (generated body
  `80ebd070…`, raw upstream source `c8773918…`). Generated rather than byte-copied because
  upstream mixes YAML sequence-indentation styles within the one document and `ci/yamllint.yaml`
  sets `indent-sequences: consistent` — a verbatim copy is a hard lint error that blocks the gate.
  That is not a guess: the first gate run of this branch returned **rc 1 BLOCKED** on exactly that
  error, and regenerating through kustomize is the fix (and is also the re-vendor idiom
  `platform/external-snapshotter` documents). Vendored at all, rather than referenced as a remote
  kustomize base, for the other reason `external-snapshotter` records: a remote base is
  `git fetch`ed on every reconcile and those fetches have flaked a whole tier.
 - `platform/intel-gpu-plugin/namespace.yaml` — a PLATFORM namespace, explicitly
  `homelab.keithjackson.dev/app-namespace: "false"` (following `vpn-egress`/`kyverno`).
 - `platform/intel-gpu-plugin/kustomization.yaml` — namespace scoping plus the image pin.
 - `platform/intel-gpu-plugin/README.md` — the measurement, the placement argument, the
  non-choices, and the post-merge verification sequence.
 - `clusters/base/intel-gpu-plugin.yaml` — **its own Flux Kustomization**: `dependsOn: [cilium]`,
  `wait: true`, `interval: 10m`, `timeout: 10m`, `prune: true`.
 - One wired line (plus its rationale comment) in `clusters/base/kustomization.yaml`.
 **Placement: its own Kustomization, not a line in the shared `platform` tier.** The shared tier was
 cheaper and needed no human reviewer on `approval-policy.json`'s terms, but `platform` is
 `wait: true` over eleven-plus components and the repo already argued this case when Cilium was
 split out: its Ready condition cannot distinguish "the plugin is wedged" from "longhorn is slow",
 the only emergency valve is freezing the entire tier, and the tier's 10m/5m interval/timeout are
 called out there as *wrong for a 3-node DaemonSet rollout* — which is exactly what this is. The
 consequence is worse than average here, because `platform` gates `platform-config` gates
 `apps-wave-0` (`wait: true`) gates `apps-wave-1`: a plugin that never went healthy inside that tier
 would stall every app, so a component added to fix a **silent no-op** would have gained the ability
 to cause a **real outage**. The approval cost of touching `clusters/**` is nil, because
 `required_approvals: 1` with `block_admin_merge_override: true` applies to every PR here regardless
 of path and authors cannot self-approve.
 `dependsOn: [cilium]` is the minimum *and* the earliest: the plugin never uses the network and
 never talks to the API server (it talks to the local kubelet over a unix socket), so a working CNI
 is its only real prerequisite, and `cilium` already depends on `flux-system`, which applies the
 `cluster-vars` ConfigMap this Kustomization substitutes from. Depending on `cilium` rather than
 `platform` also means it reconciles *in parallel with* the platform tier rather than behind all
 eleven of its components, which shrinks the cold-start window in which a node has not yet
 advertised the resource.
 `wait: true` is a deliberate departure from the `headlamp`/`crafty` leaf template. Those use
 `wait: false` because a leaf UI can be legitimately not-Ready for a long time during a cold start,
 making its health a useless signal. This DaemonSet reads sysfs and opens a socket, so not-Ready
 genuinely means broken — and since **nothing in the graph `dependsOn` it** (verified against the
 rendered graph), `wait: true` gates nothing while turning
 `flux get kustomization intel-gpu-plugin` into a one-line answer to the question that must be
 answered before the consumer PR may merge.
 Deliberate non-choices, each argued in the README: standalone DaemonSet rather than the operator
 (no CRD, no controller-manager, no cert-manager webhook, no second namespace to keep healthy);
 `--shared-dev-num` left at the upstream default of 1 (a higher count is a lie the scheduler
 believes — there is no GPU-side partitioning behind it, so it converts a loud "Unschedulable" into
 a quiet "transcodes are slow"); `--health-management` left OFF, because it is the one flag that
 *would* open a device and would reintroduce the exact `EPERM` this fixes; no NFD overlay (all three
 nodes are identical); no `supplementalGroups` anywhere, because the `pms-docker` cont-init hook
 already adds the `plex` user to each device's group and that works identically on a
 plugin-injected device.
 **Scope held deliberately:** `catalog/render.py`, `catalog/plex.yaml` and `apps/wave-0/plex.yaml`
 are untouched. Switching Plex from the hostPath to a `gpu.intel.com/i915: 1` request is a strictly
 later PR, and **it must not land until this plugin is merged AND confirmed advertising the resource
 on all three nodes** — a request for a resource no node advertises makes the Plex pod
 Unschedulable, and `apps-wave-0` is `wait: true` with `apps-wave-1` depending on it, so that would
 stall the whole app tier and turn today's silent no-op into a real outage.
 ## What I verified
 Mechanism, read from the v0.37.0 source rather than assumed:
 - **The plugin only lists devices; it never opens one.** The scan path uses `os.ReadDir`,
  `os.ReadFile`, `os.Stat` and `os.Readlink` over `/sys/class/drm` and `/dev/dri`. `grep` for
  `os.Open|OpenFile|ioctl|syscall|unix\.` across `cmd/gpu_plugin/gpu_plugin.go` (908 lines) returns
  **nothing**. Listing and `stat()` are what demonstrably work today, so the plugin needs no cgroup
  grant and no extra privilege — it does not need more than the upstream manifest gives it.
 - **The fix mechanism is real:** `createDeviceSpecsFromDrmFiles` emits
  `pluginapi.DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}` per DRM node, which is the
  thing kubelet turns into `linux.devices` plus a cgroup allow rule.
 - **The resource name is `gpu.intel.com/i915` by derivation, not by hope:** `scan()` calls
  `devTree.AddDevice(devProps.driver(), ...)`, so the suffix is the kernel driver name, and these
  nodes have `i915` bound.
 - `--shared-dev-num` default is `1` (`flag.IntVar(..., 1, ...)`); `--health-management` default is
  `false` and `setupLevelZeroService` returns early unless it is set.
 - Missing `/dev/dri/by-path` is non-fatal (`newDevicePlugin` warns, sets `bypathFound = false`).
 Image pin, verified against the registry:
 - `docker.io/intel/intel-gpu-plugin:0.37.0` exists; manifest digest
  `sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1` (single amd64
  manifest, not an index), fetched from `registry-1.docker.io` on 2026-10-01. `v0.37.0` is the
  latest upstream release (2026-09-16) and is not a prerelease.
 - The Docker Hub tag list carries `0.35.0 / 0.36.0 / 0.37.0` — **image tags have no `v` prefix**
  while GitHub release tags do.
 Policy scope, read from the policies rather than assumed:
 - All four app-scoped Kyverno policies (`restrict-host-namespaces`, `require-egress-class-label`,
  `require-auth-mode-label`, `require-default-deny-baseline`) are gated on
  `namespaceSelector`/`selector` `matchLabels: homelab.keithjackson.dev/app-namespace: "true"`.
  The new namespace is labelled `"false"`, so none of them match it.
 - Measured on prod: `plex` **is** `app-namespace=true`, confirming `privileged: true` is genuinely
  unavailable there; `kyverno` and `vpn-egress` are the two namespaces carrying the label as
  `"false"`; the other platform namespaces leave it unset. No `intel*` namespace exists, so no
  collision.
 Cluster before-state (read-only, `kubectl` in the pinned toolbox with the prod kubeconfig
 bind-mounted read-only — never copied into the worktree):
 - Three nodes `k3s-prod-{1,2,3}`, each `v1.36.4+k3s1`, Debian 13 trixie, `containerd://2.3.4-k3s1.36`.
 - Every extended resource currently advertised is `devices.kubevirt.io/{kvm,sev,tun,vhost-net}` on
  all three nodes. **No node advertises `gpu.intel.com/*`** — queried explicitly, returned
  "NONE". That is the before-state this change is meant to move.
 - Those `devices.kubevirt.io/*` resources are themselves advertised by a device plugin, which is
  measured evidence that the device-plugin API and the default kubelet socket path
  (`/var/lib/kubelet/device-plugins`) work on these nodes.
 Offline rendering (the main proof available to a PR):
 - `kubectl kustomize platform/intel-gpu-plugin` renders two objects: `Namespace/intel-gpu-plugin`
  carrying `app-namespace=false`, and `DaemonSet/intel-gpu-plugin` in namespace
  `intel-gpu-plugin`, with the image resolved to `docker.io/intel/intel-gpu-plugin:0.37.0`.
 - `kubectl kustomize clusters/base` renders, and the new Flux Kustomization appears as
  `intel-gpu-plugin  ./platform/intel-gpu-plugin  wait=true  dependsOn=cilium`. Both overlays
  (`clusters/prod`, `clusters/dev`) also render and carry it.
 - Across the whole rendered graph, **no Kustomization lists `intel-gpu-plugin` in `dependsOn`** —
  so `wait: true` gates nothing.
 - The vendored DaemonSet is a **formatting-only** transform of upstream, proven rather than
  asserted: `yq -P 'sort_keys(..)'` of the generated file and of the raw upstream file `diff`
  clean, so passing it through kustomize changed no content. The committed body's sha256 matches
  the one recorded in its own provenance header (`80ebd070…`), so the file has not drifted from
  what was generated.
 Gate (pinned toolbox, from this worktree, with the `ENGINE` shim bind-mounting the main checkout's
 gitdir and `DEV_KUBECONFIG=/dev/null`): see the PR description for the per-stage table. **`rc 3`
 (UNDETERMINED), with exactly the three expected skips** — `spec-immutability` (sibling spec repo is
 not beside a worktree), `gitea-settings` (no CI token), `conformance` (no cluster in CI). Every
 other stage passed. **rc 3 is not green** and is not reported as such.
 ## What I did NOT verify
 **Nothing about this component's runtime behaviour. It has never run.**
 - **The plugin has never executed.** No pod has been scheduled, no image has been pulled, no scan
  has happened. It cannot run until the operator merges this and Flux reconciles — `kubectl apply`
  and `flux reconcile` are operator actions (CLAUDE.md §5), so this is expected, not an oversight.
 - **The resource has never been advertised.** `gpu.intel.com/i915` does not exist on any node and
  I have not seen it exist. The claim that it *will* be `i915` rather than `xe` is derived from
  source plus the established fact that `i915` is bound — it is a derivation, not an observation.
 - **The DaemonSet has never been admitted.** That the Kyverno policies do not match this namespace
  is read off the policy manifests; no admission request has actually been evaluated. Nor has the
  `kubectl kustomize` output ever been server-side dry-run against the API — rendering is not
  admission.
 - **The `open()` test has NOT been re-run.** The `EPERM` is exactly as it was. Nothing in this PR
  touches Plex, so Plex is still software-transcoding and will continue to until the later renderer
  PR lands. The decisive evidence is an `open()` as **uid 1000 through `s6-setuidgid plex`**;
  passing as uid 0 is weaker and must not be accepted as proof.
 - **A green gate proves the manifests render, nothing more.** And this gate was not green: it was
  `rc 3`. Note also that the Gitea `gate` job runs `./ci/gate.sh || [ "$?" -eq 3 ]`
  (`.gitea/workflows/gate.yml:138`), so a green tick on this PR cannot distinguish rc 0 from rc 3,
  and `vars.CLUSTER_AVAILABLE` is unset so no L2/L3 conformance runs in CI.
 - **`seLinuxOptions: type: container_device_plugin_t`** is carried verbatim from upstream. These
  nodes are Debian 13 and I did not measure whether SELinux is enabled or whether that type exists
  there. Expected to be inert (runc skips labelling when SELinux is disabled), but unverified — if
  the pods fail to start with an SELinux error, that line is the first suspect.
 - **Renovate has not been run** against this pin. The inline-marker idiom is copied from
  `platform/kubevirt` / `platform/external-snapshotter`, but whether Renovate normalises the
  `v`-prefix mismatch (release `v0.37.0` vs image `0.37.0`) for this dep is unknown. If it proposes
  `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in `renovate.json`.
 - **Resource limits unexamined under load.** The upstream 115Mi/100m limits are taken as-is; I have
  no measurement of this plugin's footprint on these nodes. Worth a glance given
  `keycloak-operator` is OOMing at 91% of its limit elsewhere in this cluster.
 - **`clusters/dev` renders it, and there is no dev cluster** (the operator confirmed the
  `k3s-dev-*` nodes are gone). That is inert rather than wrong — the unified graph (R-ENV-02) has
  both overlays render the same base — but it is not a thing I could test.
 - **Branch protection / required-checks state not re-measured** in this session. CLAUDE.md §2b
  says re-measure before relying on it; I did not, so treat the approval/gating claims in §2 as
  unconfirmed-by-me.
@@ -1,225 +0,0 @@
 # platform/intel-gpu-plugin
 **Intel Device Plugin for Kubernetes (GPU)** — the standalone GPU plugin DaemonSet from
 [`intel/intel-device-plugins-for-kubernetes`](https://github.com/intel/intel-device-plugins-for-kubernetes),
 pinned at **v0.37.0**. It makes every node advertise **`gpu.intel.com/i915`** so a workload can
 *request* a GPU instead of hoping a `hostPath` is enough.
 - **Serves:** hardware transcoding for Plex (`hardware.gpu: intel-quicksync`), and any later
  consumer of an Intel iGPU.
 - **Tier:** its **own** Flux Kustomization
  ([`clusters/base/intel-gpu-plugin.yaml`](../../clusters/base/intel-gpu-plugin.yaml),
  `dependsOn: cilium`, `wait: true`) — **not** a line in `platform/kustomization.yaml`. See
  *Placement* below.
 - **Namespace:** `intel-gpu-plugin` — a PLATFORM namespace, labelled
  `homelab.keithjackson.dev/app-namespace: "false"`, so the app-scoped Kyverno policies
  (default-deny generation, auth-mode, egress-class, host-namespaces) do not apply.
 - **No RBAC, no ServiceAccount, no CRDs, no webhook, no API-server access.** A device plugin talks
  to the *local kubelet* over a unix socket.
 ## What this component owns
 | File | Purpose |
 |---|---|
 | `namespace.yaml` | `intel-gpu-plugin` ns, explicitly `app-namespace=false` |
 | `intel-gpu-plugin.yaml` | the DaemonSet, **vendored** (machine-generated) from upstream v0.37.0 |
 | `kustomization.yaml` | namespace scoping + the pinned image (`images:` transformer) |
 ## The problem this fixes
 Plex was given a `/dev/dri` hostPath (merged on `main` @ `cdf3e14`). The device is **visible** in
 the container and **cannot be opened**. Measured in the running pod on `k3s-prod-2`:
 ```
 uid 0,    O_RDWR -> EPERM "Operation not permitted"
 uid 1000, O_RDWR -> EPERM "Operation not permitted"
 ```
 `EPERM` from the **device cgroup**, not `EACCES` from file permissions — proven three ways:
 1. root on the **host** opens the same device with `rc=0`;
 2. the node is unified `cgroup2fs`;
 3. the container's runc `config.json` reads
   `linux.resources.devices: [{'allow': False, 'access': 'rwm'}]` and `linux.devices: None` —
   one deny-all rule, no allow entry for major 226, and the runtime was never asked to grant a
   device.
 A hostPath is a **filesystem** operation. It makes the device *node* appear and adds no cgroup
 allow rule, because Kubernetes cannot infer one from a volume.
 **A device plugin is the mechanism that closes exactly this gap.** Verified in the v0.37.0 source
 (`cmd/gpu_plugin/gpu_plugin.go`, `createDeviceSpecsFromDrmFiles`), the `Allocate` response carries
 per DRM node:
 ```go
 specs = append(specs, pluginapi.DeviceSpec{
    HostPath: devPath, ContainerPath: devPath, Permissions: "rw",
 })
 ```
 and it is that `DeviceSpec` which makes kubelet populate `linux.devices` **and** the matching
 cgroup allow rule. It is the only **non-privileged** way to get there. `privileged: true` would
 also work and is not available: `platform/kyverno/policies/restrict-host-namespaces.yaml` blocks it
 under Enforce in app namespaces, and `plex` is labelled `app-namespace=true` (measured
 2026-10-01). Correctly so.
 ## Placement — why its own Kustomization
 The cheap option was one line in the shared `platform` tier. Neither that file nor a new
 `platform/<name>/` directory is a sensitive path in `infra/gitea/approval-policy.json`, so it would
 have needed no human reviewer *on the policy's terms*. It is still the wrong place, and the repo
 already argued the case when Cilium was split out (read the comment in
 `platform/kustomization.yaml`):
 * `platform` is `wait: true` over eleven-plus components, so its Ready condition cannot distinguish
  "the GPU plugin is wedged" from "longhorn is slow".
 * The only emergency valve would be `flux suspend kustomization platform` — freezing storage, TLS,
  DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a device plugin.
 * The tier's 10m/5m interval/timeout are called out there as **wrong for a 3-node DaemonSet
  rollout**. This is a 3-node DaemonSet rollout.
 The consequence is worse than average here because of what sits downstream: `platform` gates
 `platform-config`, which gates `apps-wave-0` (`wait: true`), which gates `apps-wave-1`. A plugin
 that never went healthy inside that tier would stall **every app** on "dependency not ready" — so a
 component added to fix a *silent no-op* (software transcoding) would have acquired the ability to
 cause a *real outage*.
 The usual objection — that `clusters/**` needs a human approval — costs nothing: Gitea enforces
 `required_approvals: 1` with `block_admin_merge_override: true` on **every** PR here regardless of
 path, and authors cannot self-approve. The approval was required either way.
 ### `wait: true` is a health signal, not a gate
 This departs from the `headlamp`/`crafty` leaf template, deliberately. Those run `wait: false`
 because a leaf UI can be legitimately not-Ready for a long time during a cold start while it waits
 on Keycloak/DNS/TLS, making its health useless as a signal. This DaemonSet has no such dependency —
 it reads sysfs and opens a socket, so on healthy nodes it is Ready in seconds and **not-Ready
 genuinely means broken**. `wait: true` makes `flux get kustomization intel-gpu-plugin` a one-line
 answer to "did it come up on all three nodes", which is precisely the question that must be
 answered before the consumer PR may merge.
 Nothing in the graph `dependsOn` this, so `wait: true` gates nothing. **Think hard before adding
 `dependsOn: [intel-gpu-plugin]`** anywhere — for the app waves that would hand a device plugin the
 power to stall every app, which is the thing this placement exists to prevent.
 ## Deliberate non-choices
 **Standalone DaemonSet, not the operator.** The operator adds a CRD (`GpuDevicePlugin`), a
 controller-manager Deployment, cert-manager-backed admission webhooks and a second namespace —
 three more things that can be unhealthy — to buy declarative management of a DaemonSet that is
 already declared in git. For three identical nodes and one consumer that is strictly more blast
 radius for no gain.
 **It lists devices; it never opens them** — worth being certain about, since *opening* is exactly
 what fails today. Verified in v0.37.0 source: the scan path uses only `os.ReadDir`, `os.ReadFile`,
 `os.Stat` and `os.Readlink` over `/sys/class/drm` and `/dev/dri`. There is no `os.Open`, no
 `OpenFile`, no `ioctl` and no raw syscall against a DRM node anywhere in
 `cmd/gpu_plugin/gpu_plugin.go`. Listing and `stat()` demonstrably work today — that is how the Plex
 probe found the device before failing to open it — so the plugin needs no cgroup grant of its own
 and no privilege. Accordingly `/dev/dri` and `/sys/class/drm` are mounted **readOnly**.
 * **`--health-management` stays OFF** (upstream default). It is the one flag that *would* need to
  open a device: it starts a Level-Zero sidecar (`setupLevelZeroService` returns early unless the
  flag is set). Enabling it would reintroduce the exact `EPERM` this component exists to fix.
 * `/dev/dri/by-path` being absent is non-fatal — `newDevicePlugin` warns and sets
  `bypathFound = false`.
 **`--shared-dev-num` left at the upstream default of 1** (i.e. not set). The flag multiplies each
 card into N schedulable device IDs, letting N containers share one GPU. Not raised, because:
 * Plex is the only consumer today and wants one device;
 * the count is a **lie the scheduler believes** — there is no GPU-side partitioning or quota behind
  it, so N containers get the same `renderD128` and contend for the same two media engines
  (`vcs0`/`vecs0`). Raising it converts "Unschedulable", which is loud and obvious, into
  "transcodes are mysteriously slow", which is neither;
 * it is a one-line, zero-migration change on the day a second consumer exists — and *then* the
  right number will be known rather than guessed.
 **No NFD overlay.** All three nodes are identical (HP EliteDesk 800 G5 Mini, i7-9700T, UHD Graphics
 630, PCI `8086:3e98`, `i915` bound), so a nodeSelector on a GPU feature label would gate on
 something true everywhere. If a GPU-less node ever joins, add upstream's `nfd` overlay *then*
 rather than relying on the plugin's behaviour on such a node.
 **No `supplementalGroups`, anywhere.** The `pms-docker` image's own cont-init hook stats each
 `/dev/dri` char device and adds the `plex` user to its group — measured live as
 `Groups: 44 100 992 1000` on the running Plex process, where GID 992 is the host `render` group.
 That mechanism is image-side and works identically on a plugin-injected device.
 ## Resource name
 `gpu.intel.com/i915`. Not a guess: `scan()` calls `devTree.AddDevice(devProps.driver(), ...)`, so
 the suffix is the **kernel driver name**. These nodes have `i915` bound (not `xe`), so the resource
 is `i915`. A future kernel moving CoffeeLake to `xe` would silently rename it to `gpu.intel.com/xe`
 and a consumer's request would stop matching — worth knowing before that kernel arrives.
 ## Versioning
 Pinned to **v0.37.0** via the `images:` transformer in `kustomization.yaml`; the DaemonSet itself is
 vendored as `kubectl kustomize` output of the upstream base (see its header for the exact re-vendor
 command and both sha256s). It is *generated* rather than a byte copy because upstream mixes YAML
 sequence-indentation styles within the one document and `ci/yamllint.yaml` sets
 `indent-sequences: consistent` — a verbatim copy is a hard lint error that blocks the gate
 (measured: it did). `kubectl kustomize` normalises the formatting, and the result was proven to be
 a formatting-only change: `yq -P 'sort_keys(..)'` of the generated file and of the raw upstream file
 diff clean. This is also the re-vendor idiom `platform/external-snapshotter` documents.
 Vendored rather than referenced as a remote kustomize base for the same reason as
 `platform/external-snapshotter`: a remote base is `git fetch`ed on **every** reconcile, those
 fetches intermittently exceeded kustomize-controller's 27s timeout, and transient GitHub slowness
 then rendered as "kustomize build failed". A local file makes the build offline and deterministic —
 which also means the CI gate's `kustomize build` proves the manifests rather than proving GitHub
 was up.
 Renovate tracks it through the inline-marker idiom (`# renovate: datasource=github-releases
 depName=...` immediately above the version literal), because `renovate.json` sets
 `"docker": {"enabled": false}` and `"kubernetes": {"enabled": false}` — an image reference in a
 plain manifest is tracked by nothing. **Caveat:** upstream *release* tags carry a `v` (`v0.37.0`)
 while *image* tags do not (`0.37.0`). Whether Renovate normalises that for this dep is unverified;
 if it ever proposes `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in
 `renovate.json`, not a hand-edit.
 ## Ordering — do not reorder the stack
 The renderer change (`catalog/render.py` emitting `gpu.intel.com/i915: 1` for
 `hardware.gpu: intel-quicksync` instead of the hostPath) **must not land until this plugin is
 merged AND confirmed advertising the resource on all three nodes.** If the renderer requests a
 resource no node advertises, the Plex pod is **Unschedulable**; `apps-wave-0` is `wait: true` and
 `apps-wave-1` `dependsOn` it, so that stalls the whole app tier — turning today's silent no-op into
 a real outage.
 ## Post-merge verification
 Nothing about this component's runtime behaviour can be verified from a PR: it does not exist until
 the operator merges and Flux reconciles. Afterwards:
 ```bash
 D=./infra/devcontainer/dev.sh
 # 1. Flux applied it, and (wait:true) the DaemonSet went healthy.
 $D run bash -c 'flux get kustomization intel-gpu-plugin'
 $D run bash -c 'kubectl --context k3s-prod -n intel-gpu-plugin get ds intel-gpu-plugin'
 #    want: DESIRED=3 CURRENT=3 READY=3 AVAILABLE=3
 # 2. The resource is advertised on ALL THREE nodes. This is the gate on the renderer PR.
 $D run bash -c 'kubectl --context k3s-prod get nodes -o json \
  | jq -r ".items[] | \"\(.metadata.name) i915=\(.status.allocatable[\"gpu.intel.com/i915\"] // \"ABSENT\")\""'
 #    want: three lines, each i915=1   (before this change: ABSENT on all three)
 # 3. If a node shows ABSENT, read that node's plugin logs.
 $D run bash -c 'kubectl --context k3s-prod -n intel-gpu-plugin logs -l app=intel-gpu-plugin --tail=50 --prefix'
 ```
 ### The decisive test
 Resource-advertised is necessary but **not** sufficient — it proves the plugin registered, not that
 the cgroup allow rule reaches Plex. The decisive test is re-running the `open()` probe **as uid 1000
 through `s6-setuidgid plex`** inside the Plex pod, *after* the renderer PR has switched Plex to
 request the resource:
 ```bash
 $D run bash -c 'kubectl --context k3s-prod -n plex exec deploy/plex -- \
  s6-setuidgid plex python3 -c "import os; os.close(os.open(\"/dev/dri/renderD128\", os.O_RDWR)); print(\"rc=0 OK\")"'
 ```
 **Passing as uid 0 is weaker evidence and must not be accepted as proof** — Plex transcodes as uid
 1000, and the two differ in group membership, which is the half the image's cont-init hook supplies.
@@ -1,117 +0,0 @@
 ---
 # VENDORED, MACHINE-GENERATED, from intel/intel-device-plugins-for-kubernetes @ v0.37.0:
 #   deployments/gpu_plugin/base
 #
 # RE-VENDOR on a version bump (and bump `newTag` in kustomization.yaml to match):
 #   kubectl kustomize \
 #     "https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/gpu_plugin/base?ref=vX.Y.Z" \
 #     > platform/intel-gpu-plugin/intel-gpu-plugin.yaml
 #   # then re-add this header, and READ THE DIFF — upstream has changed the securityContext
 #   # and the volume set between minor releases before.
 #
 # Generated 2026-10-01 at v0.37.0; sha256 of the body below (header excluded):
 #   80ebd070f983f3116f4004f719d9c872220997ec766b4ab7e7290585ac8ef842
 # The raw upstream source file reviewed to produce it
 # (deployments/gpu_plugin/base/intel-gpu-plugin.yaml) was sha256
 #   c8773918c2370e711d4ba84120a9971a290bf75ecf81dc520782bcccd7ef24b6
 # and the two were proven EQUIVALENT, not merely similar: `yq -P 'sort_keys(..)'` of each diffs
 # clean, so passing it through kustomize changed formatting only and no content.
 #
 # WHY GENERATED RATHER THAN A BYTE COPY of the upstream file: upstream mixes YAML sequence
 # indentation styles within the one document (`containers:` non-indented, `env:` indented), and
 # ci/yamllint.yaml sets `indentation: indent-sequences: consistent`, so a verbatim copy is a hard
 # lint ERROR and blocks the merge gate — measured, this file failed exactly that way first time
 # round. `kubectl kustomize` normalises the output, which is also precisely the re-vendor idiom
 # platform/external-snapshotter documents for the same reason.
 #
 # WHY VENDORED AT ALL rather than a remote kustomize base (`...gpu_plugin/base?ref=v0.37.0` left in
 # `resources:`): the same reason platform/external-snapshotter is vendored. A remote base is
 # `git fetch`ed by Flux's kustomize-controller on EVERY reconcile, those fetches intermittently
 # exceeded the controller's 27s timeout, and a transient GitHub slowness then rendered as
 # "kustomize build failed" on the whole tier. A local file makes the build OFFLINE and
 # deterministic — which also means `kustomize build` in the CI gate proves the real thing instead
 # of proving GitHub was up. (platform/kubevirt still carries the remote-base pattern and its latent
 # fragility; not this component's problem.)
 #
 # DO NOT HAND-EDIT THE BODY. Every local change belongs in kustomization.yaml as a transformer or
 # patch, so the next re-vendor stays a mechanical regenerate-and-diff instead of a three-way merge.
 # The image tag below is re-pinned (and fully qualified) by the `images:` transformer there. The
 # rationale for each thing deliberately NOT changed — no privileged, no supplementalGroups, no
 # -shared-dev-num, no NFD overlay, no health-management — is in README.md.
 apiVersion: apps/v1
 kind: DaemonSet
 metadata:
  labels:
    app: intel-gpu-plugin
  name: intel-gpu-plugin
 spec:
  selector:
    matchLabels:
      app: intel-gpu-plugin
  template:
    metadata:
      labels:
        app: intel-gpu-plugin
    spec:
      containers:
      - env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: HOST_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        image: intel/intel-gpu-plugin:0.37.0
        imagePullPolicy: IfNotPresent
        name: intel-gpu-plugin
        resources:
          limits:
            cpu: 100m
            memory: 115Mi
          requests:
            cpu: 40m
            memory: 45Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          seLinuxOptions:
            type: container_device_plugin_t
          seccompProfile:
            type: RuntimeDefault
        volumeMounts:
        - mountPath: /dev/dri
          name: devfs
          readOnly: true
        - mountPath: /sys/class/drm
          name: sysfsdrm
          readOnly: true
        - mountPath: /var/lib/kubelet/device-plugins
          name: kubeletsockets
        - mountPath: /var/run/cdi
          name: cdipath
      nodeSelector:
        kubernetes.io/arch: amd64
      volumes:
      - hostPath:
          path: /dev/dri
        name: devfs
      - hostPath:
          path: /sys/class/drm
        name: sysfsdrm
      - hostPath:
          path: /var/lib/kubelet/device-plugins
        name: kubeletsockets
      - hostPath:
          path: /var/run/cdi
          type: DirectoryOrCreate
        name: cdipath
  updateStrategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
@@ -1,99 +0,0 @@
 ---
 # Intel GPU device plugin for Kubernetes — makes the nodes advertise `gpu.intel.com/i915`.
 #
 # WHAT THIS FIXES, and why nothing cheaper works. Plex already has a `/dev/dri` hostPath mount
 # (merged on main @ cdf3e14). The device node is VISIBLE in the container and cannot be OPENED.
 # Measured in the running pod on k3s-prod-2:
 #     uid 0,    O_RDWR -> EPERM "Operation not permitted"
 #     uid 1000, O_RDWR -> EPERM "Operation not permitted"
 # EPERM from the DEVICE CGROUP, not EACCES from file permissions — proven three ways: root on the
 # HOST opens the same device rc=0; the node is unified cgroup2fs; and the container's runc
 # config.json carries one deny-all rule and nothing else:
 #     linux.resources.devices: [{'allow': False, 'access': 'rwm'}]
 #     linux.devices: None
 # No allow entry for major 226, and the runtime was never asked to grant a device. A hostPath is a
 # FILESYSTEM operation: it makes the device node appear, and adds no cgroup allow rule, because
 # Kubernetes cannot infer one from a volume.
 #
 # A device plugin is the mechanism that closes exactly that gap. Verified by reading the v0.37.0
 # source (cmd/gpu_plugin/gpu_plugin.go, createDeviceSpecsFromDrmFiles): the Allocate response
 # carries `pluginapi.DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}` per DRM node, and it
 # is that DeviceSpec which makes kubelet populate `linux.devices` AND the matching cgroup allow
 # rule. It is the only NON-PRIVILEGED way to get there. `privileged: true` would also work and is
 # not available: platform/kyverno/policies/restrict-host-namespaces.yaml blocks it under Enforce in
 # app namespaces, and `plex` is labelled app-namespace=true (measured 2026-10-01). Correctly so.
 #
 # STANDALONE DAEMONSET, NOT THE OPERATOR. Upstream ships both. The operator adds a CRD
 # (GpuDevicePlugin), a controller-manager Deployment, cert-manager-backed admission webhooks and a
 # second namespace — i.e. three more things that can be unhealthy — to buy declarative management
 # of a DaemonSet we are declaring in git anyway. For three identical nodes and one consumer that is
 # strictly more blast radius for no gain. The DaemonSet has no CRDs, no controller, no webhook, and
 # no API-server access at all: a device plugin talks to the LOCAL kubelet over a unix socket, which
 # is why there is no ServiceAccount or RBAC here (and therefore no `**/rbac.yaml` sensitive path).
 #
 # LISTING, NOT OPENING — the thing worth being sure about, since opening is precisely what fails
 # today. Verified in v0.37.0 source: the scan path uses only os.ReadDir / os.ReadFile / os.Stat /
 # os.Readlink over /sys/class/drm and /dev/dri. There is no os.Open, no OpenFile, no ioctl and no
 # syscall of any kind against a DRM node anywhere in cmd/gpu_plugin/gpu_plugin.go. Listing and
 # stat() demonstrably work today (that is how the Plex probe found the device before failing to
 # open it), so the plugin needs no cgroup grant of its own and no privilege.
 #   * `--health-management` is left OFF (upstream default). It is the one flag that WOULD need to
 #     open a device: it starts a Level-Zero sidecar (setupLevelZeroService returns early unless the
 #     flag is set). Turning it on would reintroduce the exact EPERM this component exists to fix.
 #   * /dev/dri and /sys/class/drm are mounted readOnly, consistent with list-and-stat only.
 #   * /dev/dri/by-path absence is non-fatal (newDevicePlugin warns and sets bypathFound=false).
 #
 # RESOURCE NAME: `gpu.intel.com/i915`. Not a guess — `scan()` does
 # `devTree.AddDevice(devProps.driver(), ...)`, so the suffix is the KERNEL DRIVER name. These nodes
 # have i915 bound (not xe) on UHD Graphics 630 / PCI 8086:3e98, so the resource is i915. A future
 # kernel moving CoffeeLake to xe would silently rename it to `gpu.intel.com/xe` and the consumer's
 # request would stop matching — worth knowing before that kernel arrives.
 #
 # `--shared-dev-num` LEFT AT THE UPSTREAM DEFAULT OF 1 (so: not set). The flag multiplies each
 # card into N schedulable device IDs (`for i := 0; i < sharedDevNum; i++`), letting N containers
 # share one GPU. Deliberately not raised:
 #   * Plex is the only consumer and wants one device.
 #   * The count is a LIE the scheduler believes. There is no GPU-side partitioning or quota behind
 #     it — N containers get the same renderD128 and contend for the same two media engines
 #     (vcs0/vecs0). Raising it converts "Unschedulable", which is loud and obvious, into
 #     "transcodes are mysteriously slow", which is neither.
 #   * It is a one-line, zero-migration change on the day a second consumer actually exists, and at
 #     that point the right number will be known rather than guessed.
 # No NFD overlay either: all three nodes are identical (established cluster fact), so a nodeSelector
 # on a GPU feature label would gate on something that is true everywhere. If a GPU-less node ever
 # joins, add upstream's nfd overlay THEN rather than relying on the plugin's behaviour there.
 #
 # NO supplementalGroups, anywhere, on purpose. The pms-docker image's own cont-init hook stats each
 # /dev/dri char device and adds the `plex` user to its group — measured live as `Groups: 44 100 992
 # 1000` on the running Plex process, where GID 992 is the host `render` group. That mechanism is
 # image-side and works identically on a plugin-injected device, so adding a supplementalGroup here
 # or on the consumer would duplicate a thing that already works.
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 namespace: intel-gpu-plugin
 resources:
  - namespace.yaml
  - intel-gpu-plugin.yaml
 images:
  # PINNED, never `latest`. Upstream release v0.37.0 (2026-09-16); verified present on Docker Hub
  # 2026-10-01, manifest digest sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1
  # (single amd64 manifest, not an index). Recorded here rather than pinned by digest because this
  # repo pins upstream platform components by VERSION TAG (platform/kubevirt, external-snapshotter);
  # digest pinning is reserved for the conformance runner, where it is a security boundary.
  # `newName` fully qualifies the registry: upstream writes the bare `intel/intel-gpu-plugin`, which
  # relies on containerd's docker.io default rather than saying so.
  #
  # RENOVATE: the inline-marker idiom (customManagers in renovate.json), not the docker datasource —
  # renovate.json sets `"docker": {"enabled": false}` and `"kubernetes": {"enabled": false}`, so an
  # image reference in a plain manifest is tracked by NOTHING. The marker must sit on the line
  # IMMEDIATELY above the version literal for the regex to match, which is why it is down here on
  # `newTag` and not up in this header.
  # CAVEAT, stated because it will bite someone: the GitHub release tags carry a `v`
  # (`v0.37.0`) and the IMAGE tags do not (`0.37.0` — confirmed against the Docker Hub tag list).
  # Whether Renovate normalises that for this dep is UNVERIFIED here (no Renovate run available).
  # If it ever proposes `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in
  # renovate.json, not a hand-edit of this line.
  - name: intel/intel-gpu-plugin
    newName: docker.io/intel/intel-gpu-plugin
    # renovate: datasource=github-releases depName=intel/intel-device-plugins-for-kubernetes
    newTag: "0.37.0"
@@ -1,34 +0,0 @@
 ---
 # intel-gpu-plugin — namespace for the Intel GPU device plugin DaemonSet.
 #
 # PLATFORM INFRA, NOT AN APPLICATION NAMESPACE, and that distinction is load-bearing here rather
 # than cosmetic. Every Kyverno policy that could object to this DaemonSet is scoped by a
 # `namespaceSelector: matchLabels: homelab.keithjackson.dev/app-namespace: "true"` — verified by
 # reading the policies, not assumed:
 #   * platform/kyverno/policies/restrict-host-namespaces.yaml  (hostNetwork/PID/IPC + privileged)
 #   * platform/kyverno/policies/require-egress-class-label.yaml
 #   * platform/kyverno/policies/require-auth-mode-label.yaml
 #   * platform/kyverno/policies/require-default-deny-baseline.yaml (a *generate* rule, so an
 #     app-namespace also gets a default-deny NetworkPolicy minted into it)
 # Labelling this namespace "false" therefore puts it out of scope for all four. That is the same
 # mechanism restrict-host-namespaces' own header describes: "platform components that do [need host
 # access] live in platform namespaces that do NOT carry the app-namespace label".
 #
 # Measured on prod 2026-10-01: `kyverno` and `vpn-egress` are the two namespaces that carry the
 # label explicitly as "false"; the rest of the platform namespaces leave it unset. Both are out of
 # scope identically — an unset label does not match `"true"`. We set it EXPLICITLY, following
 # vpn-egress, because "someone chose this" and "nobody thought about it" should not look the same
 # in git, and because this component's correctness depends on the answer.
 #
 # NOTE this plugin does NOT actually need any of the things those policies forbid: it sets no
 # hostNetwork/hostPID/hostIPC and is not privileged (see intel-gpu-plugin.yaml). The namespace
 # placement is not a workaround for a policy — there is nothing to work around. It is here because
 # a node-level platform DaemonSet is not an application, and because the app-namespace fabric
 # (default-deny NetworkPolicy, egress-class, auth-mode) describes obligations that are meaningless
 # for a process that talks only to the local kubelet over a unix socket and never uses the network.
 apiVersion: v1
 kind: Namespace
 metadata:
  name: intel-gpu-plugin
  labels:
    homelab.keithjackson.dev/app-namespace: "false"
@@ -88,3 +88,72 @@ This proves the R-IDP-02 property (unauthenticated cannot reach the upstream) on
 3. Log in as a **non-platform-admins** user -> expect **403** (authenticated but not authorized).
 4. Confirm the session cookie is `.home.keithjackson.dev`-scoped (single sign-on across the other
   protected UIs).
 ## Observability: what this component can and cannot see (measured 2026-09-30)
 `servicemonitor.yaml` scrapes **this namespace's** deployment and nothing else. On prod that is
 two targets, `job="oauth2-proxy"`, both `up=1`. The complete set of oauth2-proxy-specific series
 in the TSDB is:
    count by (__name__) ({__name__=~"oauth2_proxy.*"})
      oauth2_proxy_requests_in_flight => 2
 **The per-app gate sidecars are not scraped, and there is nothing to scrape.** Six apps run an
 `oauth2-proxy` container of their own — crafty, jackett, pihole, qbittorrent, radarr, sonarr —
 each with `--http-address=0.0.0.0:4180` and **no `--metrics-address`**, so oauth2-proxy's metrics
 listener never starts. They declare only a `http:4180` container port, no `metrics` port exists on
 their Services, and `kubectl get servicemonitors,podmonitors -A` selects none of them. Any alert
 written against `oauth2_proxy_requests_total{namespace="jackett"}` would be structurally inert.
 This matters because the gate's characteristic failure is invisible by construction: an
 unauthenticated **XHR** gets a `302` to Keycloak rather than a `401`, the browser follows it
 against a live SSO session with no prompt, and the call resolves **200 carrying an HTML page**
 where the app expected JSON. Nothing in the request path reports an error. On 2026-09-30 jackett
 spent a burst of requests in exactly that state, re-established a session, and lost it 26 seconds
 into a 5-minute session, while `jackett-down` sat at 1 replica available the whole time.
 `auth-visibility-alerts.yaml` therefore alerts on **the gap**, not on the incident:
 `OAuth2ProxyGateUnobserved` compares `kube_pod_container_info{container="oauth2-proxy"}` against
 `up{container="oauth2-proxy"}` — two already-scraped families — and names every namespace running
 a gate that nothing observes. It fires for six namespaces today and clears per-namespace as each
 one gains a scrape. `runbooks/oauth2-proxy-auth-failures.md` is the by-hand substitute meanwhile,
 and it reads the sidecar logs out of Loki (Alloy collects them; 72h retention).
 **To close the gap for an app**, three things are needed together, on that app's own manifest:
 `--metrics-address=0.0.0.0:44180` on the sidecar, a `metrics` containerPort plus the matching
 Service port, and a ServiceMonitor selecting it. Only then does the auth-churn rule — sustained
 `code="302"` as a share of `oauth2_proxy_requests_total`, held across a window longer than the
 session lifetime — become writable rather than decorative.
 ### Update 2026-10-01: the listener exists; the scrape path does not
 PR #28 added `--metrics-address=0.0.0.0:44180` and a named `metrics` container port to the five
 fleet sidecars (jackett, radarr, sonarr, qbittorrent, pihole). `podmonitor.yaml` in this directory
 discovers them by the pod label `homelab.keithjackson.dev/auth-mode: forward-auth` — a **PodMonitor**
 rather than a ServiceMonitor because no Service exposes 44180 and adding one would mean changing the
 renderer.
 The endpoint is real: port-forwarded straight to the jackett pod, `/metrics` returns **http=200,
 10208 bytes**, exporting `oauth2_proxy_requests_in_flight`, `oauth2_proxy_requests_total` (with
 `code` labels) and `oauth2_proxy_response_duration_seconds`. That is verified by payload, not by the
 flag appearing in `args` — a distinction worth keeping, because the two claims came apart here.
 **The scrape is nevertheless dropped.** From a pod in ns `monitoring`, both `:44180` and `:4180` on
 the jackett pod time out; the shared proxy's `:44180` answers. The only difference is that ns
 `oauth2-proxy` has **no NetworkPolicy at all**, while each app namespace has a `default-deny` whose
 `policyTypes` include `Ingress` with no `ingress` rules (deny-all) plus a single CNP admitting
 `fromEntities: [ingress]` to port 4180 — the Gateway's Envoy and nothing else. Prometheus matches no
 clause.
 So the targets read `up == 0` and the TSDB is still empty for those namespaces. This is the same bug
 and the same fingerprint as `platform/longhorn/networkpolicy-metrics.yaml` documents for Longhorn,
 and the same narrow fix applies: ns `monitoring` → the gate pod label → TCP 44180 only. It cannot
 ship from this directory, because `oauth2-proxy` and `apps-wave-0` are **siblings** in the Flux graph
 (both `dependsOn: [platform-config]`), so a NetworkPolicy naming ns `jackett` would apply on today's
 cluster and fail on a cold rebuild. It belongs with each app's own rendered policy.
 Consequence for `auth-visibility-alerts.yaml`: its `unless` clause requires `up == 1`, not merely a
 present `up` series. A discovered-but-failing target publishes `up` with value 0, so the bare form
 would have cleared `OAuth2ProxyGateUnobserved` the moment the PodMonitor landed — reporting the gates
 as observed while nothing was being scraped. The unit test pins that scenario; deleting the `== 1`
 fails it and nothing else.
@@ -0,0 +1,177 @@
 ---
 # promtool unit tests for OAuth2ProxyGateUnobserved (platform/oauth2-proxy/auth-visibility-alerts.yaml).
 #
 # WHY. The rule is an `unless` between two aggregations over DIFFERENT metric families, and the
 # failure modes of that shape are both silent: a label that does not survive `count by` makes the
 # right-hand side match nothing (the alert fires for the scraped namespace too, i.e. permanently),
 # and a right-hand side that over-matches makes it fire for nobody (i.e. inert). Neither is
 # visible by reading the expression, and neither is visible on a live cluster where the answer
 # happens to look plausible. So all four states are pinned: fires for an unscraped namespace,
 # stays quiet for a scraped one, honours `for: 1h`, and STOPS once a scrape appears.
 #
 # HOW TO RUN. The one repo-wide harness, by the companion convention:
 #   ./infra/devcontainer/dev.sh run bash -c 'platform/monitoring/tests/run.sh platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml'
 # promtool is pinned in infra/devcontainer/versions.env; never run it off the host PATH.
 #
 # THE SERIES BELOW ARE REAL SHAPES read off live prod Prometheus on 2026-09-30 before being
 # copied here — kube_pod_container_info as kube-state-metrics emits it (job="kube-state-metrics",
 # its own instance/pod labels, image/image_id/container_id) and `up` as the oauth2-proxy
 # ServiceMonitor emits it (endpoint="metrics", port 44180). The point of using the real shapes is
 # that `count by (namespace)` has to discard all of them; a test with tidy two-label series would
 # pass while the real expression collapsed differently.
 #
 # ASSERTIONS USE `ALERTS` RATHER THAN exp_alerts: the description annotation is long prose and
 # comparing it verbatim would couple this file to every wording change while proving nothing the
 # ALERTS series does not already prove.
 rule_files:
  - auth-visibility-alerts.rules.yaml
 evaluation_interval: 1m
 tests:
  # ---------------------------------------------------------------------------------------------
  # 1. THE STATE PROD IS ACTUALLY IN (2026-09-30): jackett runs a gate sidecar with no scrape;
  #    the shared oauth2-proxy namespace runs two that ARE scraped. Asserts the alert fires for
  #    jackett and NOT for oauth2-proxy, and that `for: 1h` is honoured (pending at 59m).
  - interval: 1m
    name: an unscraped gate namespace fires after 1h; the scraped one never does
    input_series:
      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
        values: '1x130'
      # The shared instance: kube-state-metrics sees both replicas...
      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://aa11bb22",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000001"}'
        values: '1x130'
      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://cc33dd44",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000002"}'
        values: '1x130'
      # ...and both replicas are genuinely scraped, so `up` carries container="oauth2-proxy" there.
      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.189:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="oauth2-proxy"}'
        values: '1x130'
      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.2.47:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="oauth2-proxy"}'
        values: '1x130'
    promql_expr_test:
      # Not yet firing at 59m — `for: 1h` has not elapsed. Guards against someone "simplifying"
      # the hold away and turning every rollout into a page.
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
        eval_time: 59m
        exp_samples: []
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett"}'
        eval_time: 59m
        exp_samples:
          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett",severity="warning"}'
            value: 1
      # Firing for jackett past the hold...
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
        eval_time: 65m
        exp_samples:
          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
            value: 1
      # ...and never for the namespace that IS scraped. This is the assertion that catches an
      # `unless` whose right-hand side stopped matching: without it, a broken rule that fires for
      # every gate namespace would pass the test above.
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="oauth2-proxy"}'
        eval_time: 65m
        exp_samples: []
      # The count must be 1 per namespace-with-a-gate, not 1 overall: a `count by (namespace)`
      # that lost its `by` would collapse to a single unlabelled series.
      - expr: 'count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by (namespace) (up{container="oauth2-proxy"})'
        eval_time: 65m
        exp_samples:
          - labels: '{namespace="jackett"}'
            value: 1
  # ---------------------------------------------------------------------------------------------
  # 2. THE FIX LANDING. jackett's sidecar gains --metrics-address and a ServiceMonitor 70m in, so
  #    `up` starts carrying container="oauth2-proxy" for namespace="jackett" under a DIFFERENT job
  #    name than the shared proxy's. The alert must stop. This is what makes the rule a gap
  #    tracker rather than a permanent nag, and it pins that the match is on `container`, not on
  #    a hardcoded job="oauth2-proxy" (which would keep firing forever after a correct fix).
  - interval: 1m
    name: the alert clears once the sidecar is actually scraped
    input_series:
      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
        values: '1x130'
      # Absent for the first 70 minutes, then present. `_x` is "no sample"; the scrape did not
      # exist yet.
      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.213:44180",job="jackett-auth",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="jackett-auth-metrics"}'
        values: '_x70 1x60'
    promql_expr_test:
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett"}'
        eval_time: 65m
        exp_samples:
          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
            value: 1
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="jackett"}'
        eval_time: 80m
        exp_samples: []
  # ---------------------------------------------------------------------------------------------
  # 3. THE REGRESSION GUARD FOR `== 1` — the scenario that exposed a real bug on 2026-10-01.
  #
  #    A target is DISCOVERED but every scrape FAILS: `up` exists with value 0. This is not
  #    hypothetical — it is the exact state podmonitor.yaml creates today, because the app
  #    namespaces' default-deny drops the scrape (measured; see podmonitor.yaml).
  #
  #    With the original `unless count by (namespace) (up{container="oauth2-proxy"})`, the `up`
  #    SERIES EXISTS, so `count by (namespace)` returns 1, `unless` removes the namespace, and the
  #    alert GOES QUIET while nothing is being scraped at all — a green signal standing in for an
  #    absent one, which is the precise disease this whole rule was written to cure. Deleting the
  #    `== 1` makes THIS test fail and every other test in this file still pass, which is the
  #    property that makes it worth having.
  - interval: 1m
    name: a discovered-but-FAILING target does not count as observed (up==0 must still fire)
    input_series:
      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-6877fcd9f5-ht9d9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
        values: '1x130'
      # The PodMonitor target exists and never succeeds: NetworkPolicy drops the packets, so the
      # scrape times out and `up` is published as 0 rather than going absent.
      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.0.189:44180",job="oauth2-proxy-sidecars",namespace="jackett",pod="jackett-6877fcd9f5-ht9d9"}'
        values: '0x130'
    promql_expr_test:
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett"}'
        eval_time: 65m
        exp_samples:
          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
            value: 1
      # And the inner clause itself yields NOTHING while the scrape is failing — which is the
      # mechanism the alert depends on, asserted directly so a future reader can see why the
      # `== 1` matters without re-deriving it. (Scenario 4 below covers the other direction:
      # a succeeding scrape really does clear the alert, so `== 1` has not made the rule
      # impossible to satisfy.)
      - expr: 'count by (namespace) (up{container="oauth2-proxy"} == 1)'
        eval_time: 65m
        exp_samples: []
  # ---------------------------------------------------------------------------------------------
  # 4. A FLAPPING target: up goes 1 -> 0 and stays there. The alert must come BACK, not latch
  #    cleared. Covers the ordering bug where a once-observed namespace is permanently excluded.
  - interval: 1m
    name: a target that starts working and then fails re-fires
    input_series:
      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="radarr",pod="radarr-7d55cbffb9-42hmh",service="kube-prometheus-stack-kube-state-metrics",uid="3333cccc-0000-0000-0000-000000000001"}'
        values: '1x250'
      # Healthy for the first 60m, then every scrape fails for the rest.
      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.77:44180",job="oauth2-proxy-sidecars",namespace="radarr",pod="radarr-7d55cbffb9-42hmh"}'
        values: '1x60 0x180'
    promql_expr_test:
      # Quiet while it was working.
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="radarr"}'
        eval_time: 55m
        exp_samples: []
      # Firing again an hour after it broke.
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="radarr"}'
        eval_time: 125m
        exp_samples:
          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="radarr",severity="warning"}'
            value: 1
  # ---------------------------------------------------------------------------------------------
  # 5. NO GATE, NO ALERT. A namespace with no oauth2-proxy container must never appear, or the
  #    rule would page for every app on the platform the day someone widened the matcher.
  - interval: 1m
    name: namespaces without a gate sidecar are never reported
    input_series:
      - series: 'kube_pod_container_info{container="plex",container_id="containerd://feed0001",endpoint="http",image="ghcr.io/example/plex:1",image_id="ghcr.io/example/plex@sha256:feed0001",image_spec="ghcr.io/example/plex:1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="plex",pod="plex-0",service="kube-prometheus-stack-kube-state-metrics",uid="2222bbbb-0000-0000-0000-000000000001"}'
        values: '1x130'
    promql_expr_test:
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved"}'
        eval_time: 65m
        exp_samples: []
@@ -0,0 +1,137 @@
 ---
 # oauth2-proxy AUTH-VISIBILITY alerting (R-OBS-04).
 #
 # ============================================================================================
 # WHY THIS FILE EXISTS — THE 2026-09-30 JACKETT FAILURE THAT NOTHING OBSERVED
 # ============================================================================================
 # Jackett's UI stopped working. Reloading never fixed it and the user was never re-prompted to
 # log in. The sidecar's own access log had the whole story:
 #
 #   [oauthproxy.go:1026] No valid authentication in request. Initiating login.
 #   GET "/api/v2.0/indexers/yts/config?_=..."          HTTP/1.1 ... 302 400
 #   GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
 #   ... ten of these at 15:30:53, then a fresh [AuthSuccess] at 15:31:06,
 #   then "No valid authentication in request" again 26 SECONDS LATER, inside the same session.
 #
 # NOTHING ANYWHERE ELSE RECORDED IT. oauth2-proxy answers an unauthenticated XHR with a 302 to
 # Keycloak rather than a 401; the browser follows it silently, the live SSO session satisfies it
 # without a prompt, and the XHR resolves 200 carrying a 44 KB HTML dashboard where the app's
 # JavaScript wanted JSON. From the browser, from the user, and from any uptime check, the service
 # is healthy. `jackett-down` (apps/wave-0/jackett.yaml) watches
 # kube_deployment_status_replicas_available, which was 1 throughout and would have been 1 if the
 # gate had rejected every request ever made.
 #
 # ============================================================================================
 # WHY THIS FILE DOES NOT CONTAIN THE OBVIOUS ALERT — MEASURED, prod, 2026-09-30
 # ============================================================================================
 # The alert you would want is "a gated app is serving a sustained rate of auth redirects to a
 # user who is already authenticated". It is NOT WRITABLE against this cluster's TSDB, and the
 # reason is this file's real subject:
 #
 #   THE PER-APP GATE SIDECARS EXPORT NO METRICS AT ALL.
 #
 #   $ promtool query instant :9090 'count by (__name__) ({__name__=~"oauth2_proxy.*"})'
 #     oauth2_proxy_requests_in_flight => 2          <- the SHARED proxy's two replicas. That is ALL.
 #
 #   PARTLY SUPERSEDED 2026-10-01, and the correction matters more than the original. PR #28 gave
 #   the five FLEET sidecars `--metrics-address=0.0.0.0:44180` and a named `metrics` port, and the
 #   endpoint genuinely serves (verified by payload: http=200, 10208 bytes, all three
 #   `oauth2_proxy_*` families present — not merely by the flag being in `args`). podmonitor.yaml
 #   in this directory now discovers them. But the scrape is still DROPPED by the app namespaces'
 #   default-deny ingress, so the TSDB remains empty for those namespaces; and crafty still has no
 #   metrics listener at all. The rule below is unchanged in purpose and still has work to do.
 #
 #   Every sidecar (crafty, jackett, pihole, qbittorrent, radarr, sonarr) ran with
 #   `--http-address=0.0.0.0:4180` and NO `--metrics-address`, so oauth2-proxy's metrics listener
 #   was never started; the container declared only port `http:4180`, there was no `metrics` port on
 #   any of their Services, and there was no ServiceMonitor or PodMonitor for any of them anywhere
 #   in the cluster (measured: `kubectl get servicemonitors,podmonitors -A`). Six gates, zero
 #   telemetry. An alert written against `oauth2_proxy_requests_total{namespace="jackett"}` would
 #   have been PERMANENTLY INERT — present, evaluating, silent, and indistinguishable from health.
 #   This repo has shipped that mistake before (VeleroNoRecentBackup, the Longhorn rules), which is
 #   why `conformance/tests/l0-static/alert-metrics-have-scrape-targets` exists, and why the
 #   temptation was refused here rather than ratcheted into a register.
 #
 #   The SHARED proxy IS scraped (job="oauth2-proxy", 2 targets, up=1) but it is not in the data
 #   path for any sidecar-gated app — each app gates itself — and its current pods export no
 #   `oauth2_proxy_requests_total` at all, having proxied nothing since they started. An alert on
 #   the shared instance would not have fired on this incident either. Saying so is the point.
 #
 # So this file alerts on the CONDITION THAT MADE THE INCIDENT INVISIBLE, not on the incident.
 # That condition is measurable today, from two already-scraped metric families, and it is the
 # precondition for ever being able to alert on the incident itself. It self-clears the moment a
 # sidecar gains a metrics endpoint and a scrape — at which point the auth-churn rule becomes
 # writable and belongs here beside it. See runbooks/oauth2-proxy-auth-failures.md, which is the
 # by-hand equivalent an operator uses in the meantime.
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
 metadata:
  name: oauth2-proxy-auth-visibility
  namespace: oauth2-proxy
  labels:
    release: kube-prometheus-stack     # kube-prometheus-stack's ruleSelector
    app.kubernetes.io/name: oauth2-proxy
 spec:
  groups:
    - name: oauth2-proxy.auth-visibility
      rules:
        - alert: OAuth2ProxyGateUnobserved
          # A namespace runs an oauth2-proxy gate container, and Prometheus holds no scrape
          # target for that container in that namespace.
          #
          # BOTH SIDES ARE ALREADY-SCRAPED FAMILIES, deliberately:
          #   * kube_pod_container_info  — kube-state-metrics, the declared owner in
          #     alert-metrics-have-scrape-targets. It sees the container whether or not the
          #     container exports anything, which is exactly the property needed here.
          #   * up                       — UNIVERSAL. `up` carries the `container` label for any
          #     ServiceMonitor/PodMonitor target, so this works for a future per-app scrape
          #     without knowing what job name it will pick.
          # Nothing in this expression touches `oauth2_proxy_*`, so the rule cannot itself be
          # inert for the reason it is reporting.
          #
          # MEASURED ON PROD 2026-09-30 — this expression returned six namespaces:
          #   {namespace="crafty"} {namespace="jackett"} {namespace="pihole"}
          #   {namespace="qbittorrent"} {namespace="radarr"} {namespace="sonarr"}
          # and excluded namespace="oauth2-proxy", whose two replicas ARE scraped. It fires on
          # the state the cluster is in; it is not a rule waiting for a hypothetical.
          #
          # `== 1` IS LOAD-BEARING, AND IT IS A BUG FIX (2026-10-01). This clause was originally
          # a bare `up{container="oauth2-proxy"}`, which was WRONG in the one way that matters
          # here: a target that is DISCOVERED BUT FAILING still publishes `up`, with value 0. So
          # the series exists, `count by (namespace)` returns 1, `unless` removes the namespace,
          # and the alert GOES QUIET while nothing whatsoever is being scraped. That is this
          # branch's own disease — a green signal standing in for an absent one — reproduced
          # inside the alert written to cure it. It was not hypothetical: podmonitor.yaml lands
          # five targets whose scrape is DROPPED by NetworkPolicy (measured; see that file), so
          # the bare form would have silently cleared on the very next commit. `== 1` means
          # "observed" requires a scrape that actually succeeds.
          #
          # A failing target is therefore reported by BOTH this rule and kube-prometheus-stack's
          # generic TargetDown. That duplication is deliberate: TargetDown says "a target is
          # down", this says "an authentication gate is unobserved", and during an incident those
          # are different sentences. The overlap is the cheap half of the trade.
          expr: |
            count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
              unless
            count by (namespace) (up{container="oauth2-proxy"} == 1)
          # 1h, not 5m: a rollout briefly has a running sidecar whose target has not been
          # discovered yet, and this condition is a standing architectural gap, never an
          # emergency. Long enough that no deploy can produce it.
          for: 1h
          labels:
            severity: warning
          annotations:
            summary: "oauth2-proxy gate in {{ $labels.namespace }} exports no metrics"
            description: >-
              The {{ $labels.namespace }} namespace runs an oauth2-proxy gate container and
              Prometheus has NO scrape target for it, so every authentication decision that gate
              makes — every redirect to Keycloak, every 403 on a group check, every session that
              silently fails to hold — is invisible to metrics and to alerting. This is not a
              theoretical gap: on 2026-09-30 the jackett gate answered a burst of the app's own
              XHR calls with 302s to Keycloak, re-established a session, and lost it again 26
              seconds later, and the only record anywhere was a line in the sidecar's stdout.
              No alert could have fired, because there is nothing to alert on. Until a scrape
              exists the runbook's LogQL queries are the substitute; the fix is
              `--metrics-address=0.0.0.0:44180` on the sidecar plus a `metrics` port and a
              ServiceMonitor, after which the auth-churn rule becomes writable.
            runbook_url: "https://${BASE_DOMAIN}/runbooks/oauth2-proxy-auth-failures"
@@ -14,3 +14,12 @@ resources:
  - oauth2-proxy-credentials.sops.yaml
  - httproute.yaml       # forward-auth sign-in/callback ingress (was referenced in-place by platform-config)
  - servicemonitor.yaml  # Prometheus scrape (was referenced in-place by platform-config)
  # Alerting on whether the gates are OBSERVABLE at all. Travels with the servicemonitor above
  # for the same reason it does: monitoring.coreos.com CRDs are registered by the `platform`
  # tier, which is Ready before this tier's platform-config dependency completes.
  - auth-visibility-alerts.yaml
  # Discovers the PER-APP gate sidecars' :44180 metrics endpoints by pod label (PR #28 added the
  # listener; no Service exposes it, so this is a PodMonitor, not a ServiceMonitor). The scrape is
  # still denied by the app namespaces' default-deny ingress — read podmonitor.yaml's header
  # before concluding the targets should be up.
  - podmonitor.yaml
@@ -0,0 +1,109 @@
 ---
 # PodMonitor for the PER-APP oauth2-proxy gate sidecars (R-OBS-04).
 #
 # ============================================================================================
 # WHY A PodMonitor AND NOT A ServiceMonitor
 # ============================================================================================
 # PR #28 gave all five fleet sidecars `--metrics-address=0.0.0.0:44180` and a named `metrics`
 # container port. It did NOT add a Service port, and measured on prod 2026-10-01 none of the five
 # Services exposes 44180:
 #
 #   jackett      jackett:     [('port-9117', 9117), ('authproxy', 4180)]
 #   radarr       radarr:      [('port-7878', 7878), ('authproxy', 4180)]
 #   sonarr       sonarr:      [('port-8989', 8989), ('authproxy', 4180)]
 #   qbittorrent  qbittorrent: [('port-8080', 8080), ('authproxy', 4180)]
 #   pihole       pihole:      [('port-80',     80), ('authproxy', 4180)]
 #
 # A ServiceMonitor selects Service PORTS, so it would have nothing to select. A PodMonitor targets
 # pod ports directly and bypasses Services entirely — so this needs no change to the Services, and
 # therefore no change to `catalog/render.py` or the `apps/wave-*` renders and goldens.
 #
 # SELECTOR: the `homelab.keithjackson.dev/auth-mode: forward-auth` POD LABEL, not a namespace list.
 # Measured 2026-10-01: that label is present on exactly the five fleet gate pods (jackett, radarr,
 # sonarr, qbittorrent, pihole) and absent from crafty's, which is a separately-managed deployment
 # (platform/crafty) that still has no `--metrics-address` at all. So the selector describes the
 # PROPERTY being scraped rather than enumerating today's apps, and a sixth forward-auth app is
 # picked up with no edit here. `namespaceSelector.any: true` because those five namespaces are
 # created by the apps-wave-0/1 tiers, which this tier does not and must not depend on.
 #
 # ============================================================================================
 # READ THIS BEFORE BELIEVING THE TARGETS ARE UP: THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY
 # ============================================================================================
 # This PodMonitor is NECESSARY AND NOT SUFFICIENT. The endpoint serves, and the path from ns
 # monitoring is blocked. Both halves measured on prod 2026-10-01, read-only:
 #
 #   THE ENDPOINT IS HEALTHY — port-forward straight to the jackett pod's 44180:
 #     GET /metrics -> http=200 bytes=10208
 #     # TYPE oauth2_proxy_requests_in_flight gauge
 #     # TYPE oauth2_proxy_requests_total counter
 #     # TYPE oauth2_proxy_response_duration_seconds histogram
 #     oauth2_proxy_requests_total{code="200"} 3
 #     oauth2_proxy_requests_total{code="302"} 2
 #
 #   THE SCRAPE PATH IS DROPPED — from a pod in ns monitoring (grafana, which sits beside
 #   Prometheus and so occupies the same policy position) to the jackett pod IP:
 #     http://10.0.0.189:44180/metrics   -> wget: download timed out
 #     http://10.0.0.189:4180/ping       -> wget: download timed out     (even the "allowed" port)
 #     http://<shared-proxy>:44180/metrics -> 200, oauth2_proxy_* present (control: ns oauth2-proxy
 #                                            has NO NetworkPolicy, which is the only reason the
 #                                            shared instance has ever been scrapable)
 #
 #   WHY: ns jackett's `default-deny` NetworkPolicy is `podSelector: {}` with
 #   `policyTypes: [Ingress, Egress]` and an `egress` rule ONLY — no `ingress` key, which in
 #   NetworkPolicy semantics is deny-all-ingress. The only ingress allowance is the CNP
 #   `jackett-allow-gateway-ingress`: `fromEntities: [ingress]` to port 4180 — the Gateway's Envoy
 #   and nothing else. Prometheus matches no clause and its packets are dropped, not refused.
 #
 # SO THE TARGETS BELOW WILL READ `up == 0` UNTIL AN INGRESS ALLOWANCE LANDS. That is a known,
 # measured, documented state — NOT a regression introduced here, and NOT something to silence.
 # It is deliberately left visible: `OAuth2ProxyGateUnobserved` in auth-visibility-alerts.yaml
 # requires `up == 1`, precisely so a discovered-but-failing target does not read as "observed"
 # (see that file's header — this exact trap is why the `== 1` is there).
 #
 # THIS IS THE SAME BUG LONGHORN HAD, and the platform has a documented fix idiom for it:
 # platform/longhorn/networkpolicy-metrics.yaml, whose header records the identical fingerprint
 # measured on 2026-09-13 — ServiceMonitor in place, 3 targets DISCOVERED, `up` 0/0/0, TSDB empty,
 # `lastError: context deadline exceeded` (a drop, not a refusal). The fix there is a narrow
 # additive `networking.k8s.io/v1` NetworkPolicy: ns monitoring -> the pod label, that one port.
 #
 # WHY THAT POLICY IS NOT IN THIS DIRECTORY. It would have to be five objects, one per app
 # namespace, and those namespaces are created by apps-wave-0/1. Measured Flux graph, 2026-10-01:
 #   oauth2-proxy  dependsOn=['platform-config']        <- this tier
 #   apps-wave-0   dependsOn=['platform-config']        <- SIBLING, no ordering between them
 #   apps-wave-1   dependsOn=['apps-wave-0']
 # They are siblings, so on a cold bootstrap there is no guarantee ns jackett exists when this tier
 # applies, and a NetworkPolicy naming a missing namespace fails to apply. It would work on today's
 # running cluster and break on a rebuild — the "works now, broken on restore" shape this repo
 # keeps paying for. Longhorn's precedent is safe only because longhorn-system is created by the
 # same tier that ships its policy. The allowance therefore belongs WITH THE APPS, rendered next to
 # the rest of each app's network policy — i.e. in the catalog/renderer, which is in flight on
 # another branch. Tracked in the handoff; it is one narrow rule per app, not a redesign.
 apiVersion: monitoring.coreos.com/v1
 kind: PodMonitor
 metadata:
  name: oauth2-proxy-sidecars
  namespace: oauth2-proxy
  labels:
    release: kube-prometheus-stack     # kube-prometheus-stack's podMonitorSelector
    app.kubernetes.io/name: oauth2-proxy
    homelab.keithjackson.dev/purpose: metrics-scrape
 spec:
  namespaceSelector:
    # The gate sidecars live in the app namespaces, not this one. `any` rather than a fixed list so
    # a new forward-auth app needs no edit here — the pod label below is what scopes this.
    any: true
  selector:
    matchLabels:
      # The PROPERTY, not the app list. Present on exactly the five fleet gate pods (measured
      # 2026-10-01); absent from crafty, which has no metrics listener yet.
      homelab.keithjackson.dev/auth-mode: forward-auth
  podMetricsEndpoints:
    # By PORT NAME (`metrics`, 44180), never by number: the name is what PR #28 declared on the
    # container, and a numeric port here would silently stop matching if the port were renumbered.
    - port: metrics
      path: /metrics
      interval: 30s
      # 10s, matching the drop signature above: a DENIED path does not refuse, it hangs until the
      # scrape times out. Leaving the default would make each failing target hold a scrape slot for
      # the full interval.
      scrapeTimeout: 10s
@@ -51,6 +51,36 @@ rule groups still evaluate against nothing and **cannot fire**:
 | Flux (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` |
 | Hubble (`hubble_drop_total`) | `HubbleDeniedFlowSpike` |
 | etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all |
 | per-app oauth2-proxy sidecars (`oauth2_proxy_*`) — updated 2026-10-01 | no authentication alerting exists at all for crafty, jackett, pihole, qbittorrent, radarr, sonarr |
 The oauth2-proxy row is the newest, and it has **moved twice in two days** — worth reading in order,
 because each step looked like the end of it.
 **2026-09-30 — nothing to scrape.** `count by (__name__) ({__name__=~"oauth2_proxy.*"})` returned
 exactly one family, `oauth2_proxy_requests_in_flight`, from the two replicas of the **shared**
 forward-auth instance. All six per-app gate sidecars ran without `--metrics-address`, so their metrics
 listener never started, and no ServiceMonitor or PodMonitor selected them.
 **2026-10-01 — the listener exists, and the scrape is still blocked.** PR #28 added
 `--metrics-address=0.0.0.0:44180` and a named `metrics` port to the five fleet sidecars, and the
 endpoint genuinely serves (port-forwarded: `http=200`, 10208 bytes, all three `oauth2_proxy_*`
 families, `oauth2_proxy_requests_total{code="302"}` present). `platform/oauth2-proxy/podmonitor.yaml`
 now discovers them by pod label. **But the app namespaces' `default-deny` NetworkPolicy drops the
 scrape**: from a pod in ns `monitoring`, both `:44180` and `:4180` on the jackett pod time out, while
 the shared proxy's `:44180` answers — the only difference being that ns `oauth2-proxy` has no
 NetworkPolicy at all. So the targets are discovered and `up == 0`, and the TSDB is still empty for
 those namespaces. **This is the same bug Longhorn had**, with the same fingerprint; the fix idiom is
 `platform/longhorn/networkpolicy-metrics.yaml` (ns monitoring → one pod label → one port), and for
 these apps it has to be rendered alongside each app's own policy rather than shipped from the
 oauth2-proxy tier, which is a *sibling* of `apps-wave-0` in the Flux graph and so cannot rely on those
 namespaces existing yet.
 `OAuth2ProxyGateUnobserved` (`platform/oauth2-proxy/auth-visibility-alerts.yaml`) alerts on the gap
 itself from `kube_pod_container_info` and `up` — two families that *are* scraped. Note its `up == 1`:
 a discovered-but-failing target publishes `up` with value **0**, so the earlier bare form would have
 gone quiet the moment the PodMonitor landed, while nothing was being scraped. That near-miss is the
 whole reason to distrust a cleared alert here. Crafty remains outside all of this — no metrics
 listener at all. See `oauth2-proxy-auth-failures`.
 Each affected runbook says so on its own page and gives the by-hand equivalent. See `targets-down`.
@@ -108,6 +138,14 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
 - **`keycloak-db`** — Keycloak's CloudNativePG database is unavailable.
 - **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
  every green signal missed.
 - **`oauth2-proxy-auth-failures`** — a gated app "is up but does not work". Linked from
  `OAuth2ProxyGateUnobserved`. The gate answers an unauthenticated **XHR** with a 302 to Keycloak
  rather than a 401, the browser follows it silently against a live SSO session, and the call
  resolves **200 carrying HTML** where the app's JavaScript wanted JSON — so the user is never
  re-prompted and every uptime check stays green. Tells the gate's refusal from the group check
  from the app's own login, from the sidecar log and from Loki, with the 2026-09-30 jackett trace
  as the worked example. **Read the caveat**: the per-app gate sidecars export no metrics at all,
  so these logs (72h of them) are the only record this class of failure leaves.
 ### Operator procedures (no alert links here)
 - **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only
@@ -85,10 +85,17 @@ configMapGenerator:
      - node-io=node-io.md
      - node-memory=node-memory.md
      - node-memory-swap=node-memory-swap.md
      # Linked from OAuth2ProxyGateUnobserved, and the diagnostic procedure for any gated app
      # that "is up but does not work" — the gate answers an unauthenticated XHR with a 302,
      # not a 401, so the failure arrives as an HTTP 200 carrying the wrong body.
      - oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md
      # Operator procedure, not alert-linked. Reading the per-app oauth2-proxy sidecar log to tell
      # Keycloak, the sidecar and the app's own login apart. Cookie collision between two sidecars
      # presents as random session loss in ONE app's log and is only legible with both side by side,
      # so the page has to be reachable at 3am without going looking for it in git.
      # SIBLING PAGE to oauth2-proxy-auth-failures above, and they are deliberately both here: this
      # one is the SESSION-LIFECYCLE procedure (why a session died), that one is the DIAGNOSTIC
      # TRIAGE (which of the three layers refused, and how to tell a 302-to-200 from a real success).
      - oauth2-proxy-session=oauth2-proxy-session.md
      # Operator procedure, not an alert page (no alert links to it — the
      # alert-runbooks-resolvable check treats that as advisory). It is served
@@ -0,0 +1,244 @@
 # oauth2-proxy auth failures — telling the three layers apart
 Linked from `OAuth2ProxyGateUnobserved`, and the page to open whenever a gated app "is up but
 does not work": a UI that loads and then shows nothing, a page that never finishes, a list that
 stays empty, a save that silently does not save. The alert is about the *visibility* gap; this
 page is the diagnostic procedure, and it works whether or not the alert ever fired.
 ## What this means here
 Every app behind Keycloak on this platform has **three** places a request can be refused, and
 from a browser they look identical — because two of the three answer with **HTTP 200**.
 | Layer | What refuses | What the browser sees |
 |---|---|---|
 | 1. The gate (oauth2-proxy sidecar) | no valid session cookie | **302 to Keycloak**, then usually **200** |
 | 2. The gate's group check | session valid, wrong group | 403, or a redirect loop |
 | 3. The app itself | the app's own login / API key | the app's own 401 or login page |
 **Layer 1 is the one that hides.** oauth2-proxy answers an unauthenticated request with a
 redirect, not a 401 — correct for a browser navigation, disastrous for an **XHR**. The app's
 JavaScript asks for JSON, the gate answers `302`, the browser follows it silently, the live
 Keycloak SSO session satisfies it with no prompt, and the XHR resolves **200 with an HTML
 dashboard in the body**. `fetch()` sees a success. The user sees an empty page. Reloading does
 nothing, and nobody is ever asked to log in, because from the browser's point of view nothing
 failed.
 The worked example, jackett on 2026-09-30, straight from the sidecar log:
    [oauthproxy.go:1026] No valid authentication in request. Initiating login.
    GET "/api/v2.0/indexers/yts/config?_=..."          HTTP/1.1 ... 302 400
    GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
    ... ten of these at 15:30:53
    [AuthSuccess] Authenticated via OAuth2: Session{email:... created:2026-09-30 15:31:06
                  expires:2026-09-30 15:36:06 refresh_token:true groups:[platform-admins]}
    GET "/UI/Dashboard"        200 44379
    GET "/api/v2.0/indexers?_=..." 200 1501229
    ... and 26 seconds later, inside that same 5-minute session:
    [oauthproxy.go:1026] No valid authentication in request. Initiating login.
 A session created at 15:31:06, valid until 15:36:06, not honoured at 15:31:32. That is the
 signature of the session being **re-established over and over instead of held** — a cookie that
 another proxy is overwriting, or a cookie that never comes back at all — not of Keycloak being
 down and not of the app being broken.
 **Read this before you trust any silence (updated 2026-10-01).** The five fleet gate sidecars now
 *do* run a metrics listener on `:44180`, and `platform/oauth2-proxy/podmonitor.yaml` discovers
 them — but **the scrape is dropped by the app namespaces' `default-deny` NetworkPolicy**, so those
 targets read `up == 0` and the TSDB holds nothing for them. Crafty has no listener at all. Only
 the shared `oauth2-proxy` namespace instance is genuinely scraped, and it is not in the data path
 for any of these apps. So there is still nothing to graph and nothing that could have alerted.
 **The logs below remain the only record that this class of failure happened.** Check which state
 you are in before you trust a quiet dashboard — "First commands" step 1 tells you in one query.
 ## First commands
 `kubectl` does not exist on the workstation; everything runs in the pinned container, and always
 with an explicit context.
 **1. Which namespaces have an unobserved gate** (this is what the alert measures):
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
     -c prometheus -- /bin/promtool query instant http://localhost:9090 \
     "count by (namespace) (kube_pod_container_info{container=\"oauth2-proxy\"}) unless count by (namespace) (up{container=\"oauth2-proxy\"} == 1)"'
 ```
 **1b. If a namespace is listed, is it "no target" or "target failing"?** The two need different
 fixes and the alert deliberately does not distinguish them:
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
     -c prometheus -- /bin/promtool query instant http://localhost:9090 \
     "up{container=\"oauth2-proxy\"}"'
 ```
 No series for the namespace → nothing discovers it (no PodMonitor match; check the pod carries
 `homelab.keithjackson.dev/auth-mode=forward-auth` and a named `metrics` port). A series with
 value **0** → discovered but every scrape fails; get the reason from the target's `lastError`:
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   curl -s localhost:19090/api/v1/targets | python3 -c "
 import json,sys
 for t in json.load(sys.stdin)[\"data\"][\"activeTargets\"]:
    if t[\"labels\"].get(\"container\")==\"oauth2-proxy\":
        print(t[\"labels\"].get(\"namespace\"), t[\"health\"], t.get(\"lastError\",\"\")[:90])"'
 ```
 `context deadline exceeded` is a **drop**, not a refusal — that is NetworkPolicy, not a dead
 endpoint. Confirm the endpoint itself is fine by port-forwarding straight to the pod, which
 bypasses policy entirely:
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n jackett port-forward \
     $(kubectl --context k3s-prod -n jackett get pod -l app.kubernetes.io/name=jackett -o name | head -1) \
     14418:44180 & sleep 5; curl -s -o /dev/null -w "http=%{http_code} bytes=%{size_download}\n" \
     http://127.0.0.1:14418/metrics'
 ```
 `http=200` with a few KB means the listener is healthy and the path is what is broken. As of
 2026-10-01 that is the state of all five fleet apps, and the fix is an ingress allowance from ns
 `monitoring` to port 44180 — see `platform/longhorn/networkpolicy-metrics.yaml` for the idiom.
 **2. The sidecar log — but you must grep it.** The gate's stdout is ~11,500 lines a day per app
 and almost all of it is the kubelet health probe (`GET "/ping" ... 200`). A plain
 `kubectl logs --tail=100` shows you nothing but `/ping` and looks perfectly healthy. Filter:
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n jackett logs deploy/jackett -c oauth2-proxy --since=2h \
     | grep -v "/ping" | grep -E "No valid authentication|AuthSuccess|403|401|callback"'
 ```
 Substitute the namespace: `crafty`, `jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`.
 **3. The same question across every gate at once, from Loki.** Alloy collects all of these, so
 you can ask about the whole fleet and about a window that has already scrolled out of
 `kubectl logs`. Port-forward Loki and query it:
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/loki 13100:3100 & sleep 5; \
   curl -s -G --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"No valid authentication in request\" [24h]))" \
     http://127.0.0.1:13100/loki/api/v1/query | python3 -m json.tool | grep -E "namespace|value" '
 ```
 and the denominator, which is what makes the number mean something:
 ```
   ... --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"AuthSuccess\" [24h]))" ...
 ```
 **How to read the ratio.** A healthy sign-in is roughly **one or two** `No valid authentication`
 lines per `AuthSuccess` — the first request of a session is unauthenticated by definition. A
 ratio in the tens means sessions are not surviving. Measured on prod 2026-09-30 (24h):
 | namespace | `No valid authentication` | `AuthSuccess` | ratio |
 |---|---|---|---|
 | crafty | 81 | 1 | 81:1 |
 | radarr | 67 | 1 | 67:1 |
 | jackett | 39 | 4 | ~10:1 |
 | pihole, qbittorrent, sonarr | 0 | 0 | (unused in the window) |
 All three of those are the failure, not the baseline.
 **The cleanest worked case of all — radarr, measured 2026-10-01.** After the cookie fix landed,
 jackett dropped to **0** `No valid authentication` lines in 13,612 (from 39), and `removing
 session` / `cookie signature not valid` went to **0 on both apps** (radarr had 8 and 8 before).
 The eviction fingerprint is gone. But radarr still logged **206** rejections — and sampling 209 of
 them, **every single one was the same request**:
    GET "/signalr/messages/negotiate" ... 302    <- browser UA, over and over
 That is one left-open browser tab. Its cookie is gone, its SignalR socket keeps trying to
 reconnect, and every attempt is silently 302'd to Keycloak. Nobody is at the keyboard, so nobody
 is ever re-prompted; the tab just sits there looking like a working page that has stopped
 updating. **This is the entire failure class in one line**: a non-navigation request answered with
 a redirect, forever, with no error surfaced anywhere.
 So when you read a rejection count, **always group by request path before concluding anything**:
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n radarr logs deploy/radarr -c oauth2-proxy --since=24h \
     | grep -v "/ping" | grep "No valid authentication" -A 1 \
     | grep -oE "\"[^\"]+\" HTTP" | sort | uniq -c | sort -rn | head'
 ```
 One path dominating means an orphaned client, not a broken gate — annoying but harmless, and it
 clears when the tab is closed. Many different paths from one session means the session itself is
 not surviving, which is the real defect. A per-app `--api-route` is the structural fix for the
 first shape: it makes those paths answer **401** instead of 302, so the client fails loudly
 instead of looping. Note that a count of **0 `AuthSuccess`** alongside a high rejection count
 means nobody has signed in since the last restart — so `--cookie-refresh` has never run, and no
 session-lifecycle conclusion can be drawn from that window at all.
 **4. Did the gate admit the user, or did the app refuse them?** Look for the `[AuthSuccess]` line
 and what follows it. If `[AuthSuccess]` appears and the very next thing is the *app's own* login
 page or the app's 401, the gate did its job and layer 3 is your problem — that is the Crafty
 shape, and `runbooks/crafty-admin-credential.md` covers it.
 **5. Cookie names, when sessions do not survive.** Two oauth2-proxy instances emitting the
 **same** cookie name on the **same** domain hold different cookie-secrets, so each overwrites the
 other's cookie and the loser logs exactly the `No valid authentication` line above. List what
 each gate is configured to emit (arguments only — never print a Secret's data):
 ```
 infra/devcontainer/dev.sh run bash -c \
  'for ns in crafty jackett pihole qbittorrent radarr sonarr; do printf "%-12s " "$ns"; \
     kubectl --context k3s-prod -n $ns get deploy $ns \
       -o jsonpath="{range .spec.template.spec.containers[?(@.name==\"oauth2-proxy\")]}{.args}{end}" \
     | tr "," "\n" | grep -E "cookie-name|cookie-domain" | tr "\n" " "; echo; done'
 ```
 A gate with no `--cookie-name` is using oauth2-proxy's default `_oauth2_proxy`. If two of them
 share that default **and** a zone-wide `--cookie-domain=.home.keithjackson.dev`, they are
 evicting each other, and the symptom is intermittent and depends on which app you visited last.
 ## Likely causes, most likely first
 1. **Cookie collision between two gates on one domain.** Default cookie names plus a zone-wide
   cookie domain. Signature: high `No valid authentication`-to-`AuthSuccess` ratio, sessions
   dying seconds after being created, symptom depends on browsing order, and *nothing* wrong at
   Keycloak. Fix is a unique `--cookie-name` and a host-scoped `--cookie-domain` per gate — see
   `decisions/2026-09-29-crafty-oauth2-proxy-signin.md`, which found and fixed exactly this.
 2. **The cookie is too large to come back.** With `--pass-access-token` the session carries the
   access token; a fat Keycloak token pushes the cookie past 4 KB and it gets split into
   `_oauth2_proxy_0`, `_oauth2_proxy_1`… If a chunk is dropped the session is unreadable and the
   gate logs the same line. Look for numbered cookie names in the log, and for a session that
   never survives a single navigation.
 3. **The XHR path is not the navigation path.** The app's API is under a prefix the gate handles
   differently, or the app issues cross-origin XHRs the cookie does not accompany. Signature:
   `/UI/...` returns 200 and `/api/...` returns 302 in the *same second*, which is precisely the
   jackett trace above.
 4. **Session lifetime shorter than the app's polling interval,** with no refresh. The jackett
   session above was a 5-minute session. If `refresh_token:true` is present and sessions still
   die early, the refresh is failing, not missing.
 5. **Group check, not authentication.** `--allowed-group` is a floor: a valid Keycloak session in
   the wrong group is refused *after* a successful login, which reads as a loop. Look for 403 and
   for the `groups:[...]` list on the `[AuthSuccess]` line.
 6. **The app, not the gate.** `[AuthSuccess]` followed by the app's own login page or 401.
 ## When to escalate or stop
 - **Stop and hand over if the fix is a manifest change.** Everything above is read-only
  diagnosis. Changing a `--cookie-name`, adding `--metrics-address`, or editing an alert goes
  through a branch and a PR; `kubectl edit` on a gate is drift that the next Flux reconcile
  reverts, and in the meantime you have an unreproducible cluster.
 - **Stop if you find yourself about to print a Secret.** The cookie-secret and client-secret live
  in `oauth2-proxy-credentials` and the per-app equivalents. Compare *key names* and whether two
  Secrets differ; never dump `.data` or `.stringData`. A password has been leaked into a
  transcript here exactly this way.
 - **Escalate to the platform owner if Keycloak itself is the suspect** — check `KeycloakDown` and
  `KeycloakDatabaseDown` first (`runbooks/keycloak-down.md`, `runbooks/keycloak-db.md`). In the
  2026-09-30 incident Keycloak was healthy throughout and blaming it would have cost days.
 - **The honest limit of this page.** With no metrics from the sidecars there is no history beyond
  Loki's **72h** retention (`platform/monitoring/loki.yaml`, `retention_period: 72h`). If the
  report is older than three days, the evidence is gone and the only remaining move is to
  reproduce it. That is the cost of the gap `OAuth2ProxyGateUnobserved` is tracking, and it is
  the reason to close it rather than silence it.
Author	SHA1	Message	Date
jakman85andClaude Opus 5	1e71b578d5	docs(obs): record why CI is red — approval-policy on the one conformance entry gate / gate (pull_request) Successful in 1m41s Details gate / conformance (pull_request) Has been skipped Details The gate job reported failure while the local run was rc 3 with nothing failed. Measured cause is a single sensitive path (the KNOWN_EXTERNAL_MONITORS entry) with no non-author approving review, which ci/approval-policy.sh blocks fail-closed by design. Also records the general trap: approval-policy short-circuits to N/A off a pull_request event, so a clean LOCAL gate is a false reassurance for any branch touching a sensitive path. Only the PR run tells the truth. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-01 22:12:19 -05:00
jakman85andClaude Opus 5	bce101d03f	feat(obs): wire the gate-sidecar scrape, and fix the alert that would have gone silent doing it gate / gate (pull_request) Failing after 1m42s Details gate / conformance (pull_request) Has been skipped Details PR #28 gave the five fleet oauth2-proxy sidecars --metrics-address=0.0.0.0:44180 and a named `metrics` port. This wires a scraper to them and, in the process, found a real bug in the alert this branch added. PodMonitor, not ServiceMonitor. Measured: no Service exposes 44180 on any of the five (jackett/radarr/sonarr/qbittorrent/pihole all publish only their app port + authproxy 4180), so a ServiceMonitor would select nothing. A PodMonitor targets pod ports directly and needs no change to catalog/render.py or the apps/wave-* renders and goldens. Selector is the pod label homelab.keithjackson.dev/auth-mode=forward-auth — present on exactly those five gate pods, absent from crafty's, which still has no metrics listener. The property, not an app list. THE BUG: `up{container="oauth2-proxy"}` -> `up{container="oauth2-proxy"} == 1`. A target that is discovered but FAILING still publishes `up`, with value 0. The series exists, so count by (namespace) returned 1, `unless` removed the namespace, and OAuth2ProxyGateUnobserved went QUIET while nothing was being scraped at all — this branch's own disease reproduced inside the alert written to cure it. Not hypothetical: the PodMonitor lands five targets whose scrape is dropped, so the bare form would have silently cleared on this very commit. Proven by mutation: reverting the `== 1` fails the two new unit-test scenarios (got: nil) and leaves every other assertion green. The scrape does NOT work yet, and that is documented rather than papered over. From a pod in ns monitoring to the jackett pod, :44180 AND :4180 both time out; the shared proxy's :44180 answers. ns jackett's default-deny is podSelector:{} with policyTypes:[Ingress,Egress] and NO ingress key (deny-all), and the one allowance is fromEntities:[ingress] to 4180 — the Gateway's Envoy only. The endpoint itself is healthy: port-forwarded, /metrics returns http=200, 10208 bytes, all three oauth2_proxy_* families including requests_total{code="302"}. Same bug and fingerprint as platform/longhorn/networkpolicy-metrics.yaml records for Longhorn; the fix is one narrow rule per app, and it cannot ship from this tier because oauth2-proxy and apps-wave-0 are Flux SIBLINGS (both dependsOn platform-config), so naming ns jackett here would apply today and fail on a cold rebuild. Needs an owner in the renderer. Touches conformance/** : one KNOWN_EXTERNAL_MONITORS entry, which is the mechanism the failing check explicitly instructs ("DECLARE it ... with who creates it") and which flux-podmonitor.yaml already uses. It records that the port name was measured on prod instead of resolved from a Service, and that the targets are currently down. That makes this PR require a non-author review on the sensitive-path rule. Also: ratio alert still NOT shipped. oauth2_proxy_requests_total now exists on the endpoint but not in the TSDB, so writing it today would make it inert for exactly the reason it was refused on 2026-09-30. Runbook gains policy-vs-no-target triage and the radarr /signalr/messages/negotiate worked case — one orphaned browser tab, 206 identical 302s, nobody ever re-prompted. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-01 21:56:07 -05:00
jakman85	ba79a1b59b	Merge remote-tracking branch 'origin/main' into feat/oauth2-proxy-auth-failure-visible # Conflicts: # runbooks/kustomization.yaml	2026-10-01 21:40:37 -05:00
jakman85andClaude Opus 5	fc7ec01628	feat(obs): make oauth2-proxy auth failures visible — alert on the gap, not on absent metrics gate / gate (pull_request) Successful in 1m42s Details gate / conformance (pull_request) Has been skipped Details Jackett failed on 2026-09-30 in the most expensive way available: HTTP 200 and a plausible page. oauth2-proxy answers an unauthenticated XHR with a 302 to Keycloak rather than a 401; the browser follows it silently against a live SSO session and the call resolves 200 carrying a 44 KB HTML dashboard where the app's JavaScript wanted JSON. The user is never re-prompted, reloading never helps, and every uptime check stays green. `jackett-down` watches replicas_available, which was 1 throughout. The reason nothing alerted turns out not to be a missing rule. Measured on prod: count by (__name__) ({__name__=~"oauth2_proxy."}) oauth2_proxy_requests_in_flight => 2 Two series, both from the SHARED forward-auth Deployment. The six per-app gate sidecars (crafty, jackett, pihole, qbittorrent, radarr, sonarr) publish nothing at all: no --metrics-address, so the metrics listener never starts; no `metrics` container port; no ServiceMonitor or PodMonitor selects them. The alert the incident calls for — sustained auth redirects to an authenticated user — is therefore NOT WRITABLE, and writing it anyway would have shipped a rule that is present, evaluating, permanently silent and indistinguishable from health. So alert on the condition that made the incident invisible, from families that ARE scraped: OAuth2ProxyGateUnobserved count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by (namespace) (up{container="oauth2-proxy"}) kube_pod_container_info is kube-state-metrics (a declared scrape owner) and sees the container whether or not it exports anything; `up` is UNIVERSAL and carries a `container` label for any target, so the rule needs no advance knowledge of what job name a future per-app scrape picks. Nothing in it touches oauth2_proxy_, so it cannot be inert for the reason it is reporting. Run verbatim against live prod it returns six namespaces and correctly excludes oauth2-proxy. It clears per namespace as each gate gains a real scrape. Also: - promtool companion test pinning all four states: fires unscraped, quiet when scraped, honours for:1h, and STOPS once a scrape appears under a different job name. - runbooks/oauth2-proxy-auth-failures.md — tell the gate's refusal from the group check from the app's own login from an XHR that silently got HTML, with the 2026-09-30 jackett trace as the worked example and the Loki queries to use while there are no metrics. Registered in the configMapGenerator and README. - runbooks/README.md gains an inert-alerts row for the unscraped sidecars. Rejected: an alert on the shared proxy's oauth2_proxy_requests_total ratio. It needs a conformance/ SCRAPE_OWNERS entry (human approval) and, decisively, would not have fired on this incident — the shared instance is not in any sidecar-gated app's data path and its current pods have proxied nothing. It would have looked like coverage and been none. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-30 10:55:19 -05:00