feat(obs): make oauth2-proxy auth failures visible — alert on the gap, not on absent metrics

Jackett failed on 2026-09-30 in the most expensive way available: HTTP 200 and a
plausible page. oauth2-proxy answers an unauthenticated XHR with a 302 to Keycloak
rather than a 401; the browser follows it silently against a live SSO session and
the call resolves 200 carrying a 44 KB HTML dashboard where the app's JavaScript
wanted JSON. The user is never re-prompted, reloading never helps, and every uptime
check stays green. `jackett-down` watches replicas_available, which was 1 throughout.

The reason nothing alerted turns out not to be a missing rule. Measured on prod:

    count by (__name__) ({__name__=~"oauth2_proxy.*"})
      oauth2_proxy_requests_in_flight => 2

Two series, both from the SHARED forward-auth Deployment. The six per-app gate
sidecars (crafty, jackett, pihole, qbittorrent, radarr, sonarr) publish nothing at
all: no --metrics-address, so the metrics listener never starts; no `metrics`
container port; no ServiceMonitor or PodMonitor selects them. The alert the incident
calls for — sustained auth redirects to an authenticated user — is therefore NOT
WRITABLE, and writing it anyway would have shipped a rule that is present,
evaluating, permanently silent and indistinguishable from health.

So alert on the condition that made the incident invisible, from families that ARE
scraped:

  OAuth2ProxyGateUnobserved
    count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
      unless
    count by (namespace) (up{container="oauth2-proxy"})

kube_pod_container_info is kube-state-metrics (a declared scrape owner) and sees the
container whether or not it exports anything; `up` is UNIVERSAL and carries a
`container` label for any target, so the rule needs no advance knowledge of what job
name a future per-app scrape picks. Nothing in it touches oauth2_proxy_*, so it
cannot be inert for the reason it is reporting. Run verbatim against live prod it
returns six namespaces and correctly excludes oauth2-proxy. It clears per namespace
as each gate gains a real scrape.

Also:
- promtool companion test pinning all four states: fires unscraped, quiet when
  scraped, honours for:1h, and STOPS once a scrape appears under a different job name.
- runbooks/oauth2-proxy-auth-failures.md — tell the gate's refusal from the group
  check from the app's own login from an XHR that silently got HTML, with the
  2026-09-30 jackett trace as the worked example and the Loki queries to use while
  there are no metrics. Registered in the configMapGenerator and README.
- runbooks/README.md gains an inert-alerts row for the unscraped sidecars.

Rejected: an alert on the shared proxy's oauth2_proxy_requests_total ratio. It needs
a conformance/ SCRAPE_OWNERS entry (human approval) and, decisively, would not have
fired on this incident — the shared instance is not in any sidecar-gated app's data
path and its current pods have proxied nothing. It would have looked like coverage
and been none.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

.gitea/workflows/gate.yml

@@ -131,6 +131,10 @@ jobs:
           # merging. Green here means the token resolved and the live Gitea matched the
           # declaration in infra/gitea/expected-branch-protection.json.
           REQUIRE_GITEA_SETTINGS: "true"
+          # ci/approval-policy.sh needs to know WHICH PR it is looking at, so it can read that
+          # PR's changed files and its reviews. Absent on a push event, where the stage is N/A.
+          GITHUB_EVENT_NAME: ${{ github.event_name }}
+          PR_NUMBER: ${{ github.event.pull_request.number }}
         run: ./ci/gate.sh || [ "$?" -eq 3 ]
 
       # NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4

CLAUDE.md

@@ -14,7 +14,28 @@ There is no other path. Specifically:
 
 - **Do not `git push origin main`.**
 - **Do not `git push origin HEAD:main`.** Same thing wearing a hat.
-- **Do not merge your own PR.** A human approves and clicks merge.
+- **Do not merge your own PR.** See the approval model below — and either way, the merge is
+  someone else's action, never yours.
+
+### Who approves
+
+**The CI/CD pipeline is the approver.** A green gate is sufficient for an ordinary change; the
+branch requires no human approval for one. That is a deliberate transfer of authority, on the
+operator's stated principle: *if the pipeline is not good enough to be trusted for this, it is not
+good enough to do its job and needs improving* — so a weak gate is a bug to fix, not a reason to
+add a human.
+
+**One exception, enforced by `ci/approval-policy.sh`:** a change to the pipeline itself, the
+admission policies, the Flux wiring, the SOPS recipients, the RBAC, or this file needs an
+approving review from someone who is not its author. The reason is narrow and structural — **a
+change to the gate cannot be gated by the gate.** A PR that weakens a check would be approved by
+the weakened check. Everything else merges on green.
+
+The path list is data, with a reason per entry: `infra/gitea/approval-policy.json`. It is itself a
+sensitive path, so the policy cannot be widened or narrowed without a human.
+
+That stage is **fail-closed** — if it cannot evaluate, it blocks. Unlike `gitea-settings`, where a
+missing credential is a lock-out, here a skip would be a bypass.
 - **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
   anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
 

ci/Makefile

@@ -35,6 +35,9 @@ decisions-log: ## handoff log: DECISIONS-LOG.md frozen, decisions/ entries well-
 l1: ## L1 rendering checks (no cluster; live now)
 	@"$(CI_DIR)l1.sh"
 
+approval-policy: ## CI is the approver; sensitive paths still need a human (needs PR_NUMBER + token)
+	@./approval-policy.sh
+
 gitea-settings: ## Gitea branch protection + runners match infra/gitea/expected-branch-protection.json (SKIPS w/o GITEA_SETTINGS_TOKEN)
 	@./gitea-settings.sh
 

ci/approval-policy.sh

@@ -0,0 +1,117 @@
+#!/usr/bin/env bash
+# ci/approval-policy.sh — the CI/CD pipeline is the approver, EXCEPT for changes that alter
+# what the pipeline verifies or the security posture it rests on.
+#
+# THE POLICY, and why it is shaped this way.
+#
+# Operator intent (2026-09-29): "CI/CD pipeline is the end-all-be-all for approval. If the CI
+# pipeline is not good enough to be trusted for this, then it is not good enough to do its
+# intended job and needs to be improved. Manual approvers should only be for architecturally or
+# security relevant changes -- which is essentially the CI pipeline itself and kyverno policies."
+#
+# So `required_approvals` on the branch drops to 0 and the GATE decides. That is a real transfer
+# of authority, and it has exactly one hole: **a change to the gate cannot be gated by the gate.**
+# A PR that weakens a check would be approved by the weakened check. The same applies to the
+# admission policies, the Flux wiring that decides what reconciles, the SOPS recipients, and the
+# RBAC. Those are the "architecturally or security relevant" set, and this stage is what still
+# requires a human for them.
+#
+# The list is data, not code: infra/gitea/approval-policy.json, with a stated reason per entry so
+# it stays reviewable. Changing that list is itself a sensitive change (infra/gitea/**), so the
+# policy cannot be widened without a human.
+#
+# FAIL-CLOSED, DELIBERATELY DIFFERENT FROM gitea-settings.sh. That stage skips when its token is
+# missing, because an unwired credential blocking every PR is a lock-out. Here the reverse is
+# true: if this cannot evaluate, an unreviewed change to the gate itself could merge. A skip is a
+# BYPASS, not a gap. So: cannot determine -> BLOCK.
+#
+# EXIT CODES
+#   0  no sensitive paths touched (gate alone is sufficient), OR sensitive paths touched and a
+#      non-author APPROVED review is present
+#   1  sensitive paths touched without such a review, OR the policy could not be evaluated
+set -euo pipefail
+CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
+# shellcheck disable=SC1091
+source "${CI_DIR}/lib.sh"
+
+log "Stage: approval policy (CI approves; humans approve changes to CI)"
+
+DECL="${REPO_ROOT}/infra/gitea/approval-policy.json"
+[ -f "$DECL" ] || die "approval policy declaration missing: ${DECL}"
+require_tool jq
+require_tool python3
+
+# Not a PR -> nothing to gate. A push to main is already merged, and direct pushes are refused
+# by branch protection, so this is genuinely not-applicable rather than a skipped check.
+EVENT="${GITHUB_EVENT_NAME:-${EVENT_NAME:-}}"
+PR="${PR_NUMBER:-}"
+if [ "$EVENT" != "pull_request" ] || [ -z "$PR" ] || [ "$PR" = "null" ]; then
+  ok "not a pull_request event (event='${EVENT:-none}') — approval policy N/A"
+  exit 0
+fi
+
+GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
+GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
+TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
+if [ -z "$TOKEN" ]; then
+  err "no SETTINGS_API_TOKEN — cannot read this PR's changed files or its reviews."
+  err "  BLOCKING. A skip here would let an unreviewed change to the gate itself merge."
+  summary "- approval-policy: **BLOCKED — cannot evaluate** (no token; a skip here is a bypass)"
+  exit 1
+fi
+api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
+
+FILES_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/files?limit=1000" || true)"
+PR_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}" || true)"
+REVIEWS_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/reviews" || true)"
+for v in FILES_JSON PR_JSON REVIEWS_JSON; do
+  jq -e . >/dev/null 2>&1 <<<"${!v}" || {
+    err "could not read ${v} from Gitea for PR #${PR} — BLOCKING rather than assuming."
+    summary "- approval-policy: **BLOCKED — Gitea unreadable**"
+    exit 1; }
+done
+
+AUTHOR="$(jq -r '.user.login // empty' <<<"$PR_JSON")"
+[ -n "$AUTHOR" ] || { err "could not determine PR author — BLOCKING"; exit 1; }
+
+# NOTE: the PR file list goes in via an ENV VAR, not a second stdin redirect. `python3 - <<EOF`
+# already consumes stdin for the script; adding `<<<"$json"` silently wins and the script body is
+# never read. Caught before shipping, by running it.
+SENSITIVE="$(FILES_JSON="$FILES_JSON" DECL="$DECL" python3 -c '
+import json, os, fnmatch
+decl = json.load(open(os.environ["DECL"]))
+files = [f.get("filename", "") for f in json.loads(os.environ["FILES_JSON"])]
+pats = decl["sensitive_paths"]
+hits = sorted({f for f in files for p in pats
+               if fnmatch.fnmatch(f, p) or (p.endswith("/**") and f.startswith(p[:-2]))})
+print("\n".join(hits))
+')"
+
+if [ -z "$SENSITIVE" ]; then
+  ok "no architecturally/security-relevant paths touched — the GATE is the approver."
+  summary "- approval-policy: **green** (CI approves; no sensitive paths touched)"
+  exit 0
+fi
+
+echo "  sensitive paths touched:" >&2
+printf '    %s\n' $SENSITIVE >&2
+
+APPROVER="$(jq -r --arg a "$AUTHOR" \
+  '[.[] | select(.state == "APPROVED") | select(.user.login != $a) | .user.login] | first // empty' \
+  <<<"$REVIEWS_JSON")"
+
+if [ -n "$APPROVER" ]; then
+  ok "approved by '${APPROVER}' (author '${AUTHOR}') — sensitive change is human-reviewed."
+  summary "- approval-policy: **green** (sensitive paths approved by \`${APPROVER}\`)"
+  exit 0
+fi
+
+err "This PR changes the pipeline or the security posture it rests on, and has NO approving"
+err "  review from anyone other than its author ('${AUTHOR}')."
+err "  A change to the gate cannot be gated by the gate — that is the one thing CI cannot"
+err "  honestly approve for itself. Get a human review, or move the change out of:"
+printf '    %s\n' $SENSITIVE >&2
+err "  The list lives in infra/gitea/approval-policy.json, with a reason per entry."
+summary "- approval-policy: **BLOCKED — sensitive paths need a non-author approval**"
+exit 1

ci/gate.sh

@@ -113,6 +113,7 @@ run_stage lint          "${CI_DIR}/lint.sh"
 run_stage l0            "${CI_DIR}/l0.sh"
 run_stage render-golden "${CI_DIR}/render-golden.sh"
 run_stage l1            "${CI_DIR}/l1.sh"
+run_stage approval-policy "${CI_DIR}/approval-policy.sh"
 run_stage gitea-settings "${CI_DIR}/gitea-settings.sh"
 run_stage conformance   "${CI_DIR}/conformance.sh"
 
@@ -190,6 +191,20 @@ case "${OUTCOME[spec-immutability]}" in
   pass) ok   "spec-immutability: green" ;;
 esac
 
+# approval-policy: the branch requires NO human approvals (required_approvals: 0) -- the GATE
+# is the approver, per operator intent. This stage is the single exception: a change to the
+# pipeline itself, the admission policies, the Flux wiring, the SOPS recipients or the RBAC
+# cannot honestly be approved BY the thing it changes, so those paths still need a human.
+#
+# It never SKIPS -- it exits 0 or 1 only, and blocks when it cannot evaluate. That is the
+# opposite of gitea-settings' tolerance and it is deliberate: there, a missing credential is a
+# lock-out; here, a missing credential would let an unreviewed change to the gate merge.
+case "${OUTCOME[approval-policy]}" in
+  fail) err "approval-policy: a sensitive change lacks a non-author approval -> blocks merge"; BLOCK=1 ;;
+  skip) err "approval-policy: did not run -- it is fail-closed by design, so treat as a block"; BLOCK=1 ;;
+  pass) ok  "approval-policy: green" ;;
+esac
+
 # gitea-settings: the ONLY check that can see Gitea's own database (branch protection +
 # runner registration). A FAIL is real drift and always blocks -- that is the whole point.
 # A SKIP means no token or unreachable Gitea, which is an UNWIRED CREDENTIAL, not a defect

conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh

@@ -237,6 +237,31 @@ SCRAPE_OWNERS: list[dict] = [
                "backed up (VeleroScheduleNeverBackedUp / SchedulePaused / NoVolumeDataBackup); "
                "Velero itself publishes nothing until a backup is attempted",
     },
+    {
+        "owner": "kube-prometheus-stack: kube-state-metrics custom-resource-state for Flux revisions",
+        "prefixes": ("kube_flux_",),
+        "file": KPS,
+        "require": [
+            r"customResourceState:\s*\n\s*enabled:\s*true",
+            r"metricNamePrefix:\s*kube_flux_kustomization",
+            r"metricNamePrefix:\s*kube_flux_gitrepository",
+            # The RBAC is MANDATORY, not cosmetic — the same measured failure mode as velero.io
+            # above: without read access to these two API groups KSM publishes the
+            # custom-resource-state metrics as an EMPTY SET and logs a forbidden error, which would
+            # leave FluxRolloutInProgress/FluxRolloutStuck inert while looking configured.
+            r'apiGroups:\s*\["kustomize\.toolkit\.fluxcd\.io"\]',
+            r'apiGroups:\s*\["source\.toolkit\.fluxcd\.io"\]',
+            # A config-only edit does not restart KSM (the chart sets no checksum annotation over
+            # the ConfigMap), so the pod annotation that forces the roll is part of the claim: drop
+            # it and a future change to the flux collector set ships green and never takes effect.
+            r"homelab\.keithjackson\.dev/crs-config-rollout",
+        ],
+        "forbid": [r"customResourceState:\s*\n\s*enabled:\s*false"],
+        "why": "kube_flux_* is the ONLY revision-carrying signal in the TSDB — gotk_* has no "
+               "revision label anywhere (measured 2026-09-29) and the Flux event stream emits "
+               "ReconciliationSucceeded on every interval tick whether or not anything changed, so "
+               "without these metrics nothing can tell a deploy starting from an idle cluster",
+    },
     {
         "owner": "kube-prometheus-stack: bundled kube-state-metrics",
         "prefixes": ("kube_",),

decisions/2026-09-29-flux-deploy-notifications.md

@@ -0,0 +1,144 @@
+---
+date: 2026-09-29
+branch: feat/flux-deploy-notifications
+title: Flux deploy notifications — the operator now gets mail when a git change starts rolling out and when it completes
+---
+
+## What changed
+
+The operator was getting **no notification when Flux began applying a git change, and none when it
+finished**. Root cause, measured rather than inferred: `notification-controller` v1.4.0 is running
+(`1/1` ready) and the `Provider`/`Alert`/`Receiver` CRDs are installed and serving `v1beta3`, but
+there are **zero `Provider` and zero `Alert` objects in the entire cluster** (`kubectl get
+providers,alerts -A` → `No resources found`). Nothing in git ever created any. The five `gotk_*`
+rules in `platform/monitoring/alerts.yaml` only cover *failures* (`FluxReconciliationFailure`,
+`FluxResourceSuspended`) — nothing covered the healthy path.
+
+**The obvious fix does not work, and this is the substance of the change.** Neither Flux event can be
+filtered into a deploy signal:
+
+- `ReconciliationSucceeded` is a **heartbeat, not a change signal**. 16 Kustomizations, 15 on
+  `interval: 10m` → **101 such events in one hour**, with the message `Reconciliation finished in
+  324.161192ms, next run in 10m0s` — byte-identical in shape whether the apply changed anything or
+  not. An `Alert` at `eventSeverity: info` would have mailed ~100 times an hour.
+- `Progressing` is worse. Five Kustomizations re-apply the same object on **every** tick — event
+  counts **2505 / 2504 / 2501 / 2495 / 2490** (`apps-vms` → `NetworkPolicy/unifi/default-deny`,
+  `platform` → `CustomResourceDefinition/cdis.cdi.kubevirt.io`, `apps-wave-0/1/2` →
+  `NetworkPolicy/{gitea,nextcloud,seerr}/default-deny`). Anything keyed on `Progressing` would mail
+  five times every ten minutes, forever.
+- `gotk_*` carries **no revision label anywhere** — `gotk_reconcile_condition` 120 series,
+  `gotk_suspend_status` 40, not one with a revision.
+
+The applied **revision** is the only signal that moves when and only when git moves, and it lives in
+object status alone. So:
+
+1. **`platform/monitoring/kube-prometheus-stack.yaml`** — two resources added to the *existing*
+   `kube-state-metrics.customResourceState` config (the Velero-Schedule path, already enabled):
+   `kube_flux_kustomization_info{revision}` + `kube_flux_kustomization_suspended` (from
+   `status.lastAppliedRevision` / `spec.suspend`) and `kube_flux_gitrepository_info{revision}` (from
+   `status.artifact.revision`). Plus the two mandatory read-only `rbac.extraRules` entries, and a
+   `podAnnotations` roll-counter — the chart sets **no checksum annotation** over that ConfigMap, so
+   a config-only edit would have left KSM running the old collector set with the new metrics absent
+   and the new alerts silently inert. Named `kube_flux_*`, never `gotk_*`, for the same reason
+   `kube_velero_schedule_*` is not `velero_*`: these are KSM series about Kubernetes objects.
+   Cost: 33 series.
+2. **`platform/monitoring/alerts.yaml`**, group `platform-health.flux` — `FluxRolloutInProgress`
+   (`severity: info`, **no `for:`**) fires the moment any non-suspended Kustomization's applied
+   revision is not the revision the GitRepository fetched from `main`, and **clears when all have
+   caught up**. Fired mail = "a deploy started"; **resolved mail = "it completed"** — the completion
+   half is delivered by `send_resolved`, not by a second rule. `FluxRolloutStuck` (`severity:
+   warning`, `for: 30m`) is the same expression as the escalation for a rollout that started and
+   never converged.
+3. **Alertmanager** — a dedicated `severity = "info"` route and `email-deploy` receiver, subject
+   `[FLUX] applying|applied — …`. Without it a routine deploy would have arrived subject-lined
+   `[WARNING]` via the catch-all receiver and become indistinguishable from a real fault.
+   `group_by: [alertname]` collapses all 16 Kustomizations into one mail per rollout;
+   `group_interval: 2m` so completion is not held 4m behind the event.
+4. **`platform/monitoring/tests/flux-rollout.test.yaml`** — 6 promtool scenarios, both directions.
+5. **`runbooks/flux-rollout.md`** (+ its `configMapGenerator` entry) and a `kube_flux_` entry in
+   `conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh` whose `require` patterns
+   pin `customResourceState`, both `metricNamePrefix`es, both `apiGroups` and the roll-counter — so
+   deleting any of them fails the gate instead of silently re-inerting the alerts.
+
+Two guards in the expression, each with a test whose only job is to fail if that clause is removed:
+`and on () (count(kube_flux_gitrepository_info…) > 0)` (without it, a missing exporter makes
+`unless on (revision) <empty>` remove nothing and **all 16 Kustomizations alert forever** — the
+permanently-firing mode this repo shipped once already), and `unless on (namespace, name)
+(kube_flux_kustomization_suspended == 1)` (a suspended Kustomization pins an old revision by design
+and would hold the rollout open, so the "applied" mail would never arrive).
+
+## What I verified
+
+Measured against live prod (`--context k3s-prod`) **before** writing anything, and quoted here as
+measured:
+
+- `notification-controller` `1/1`, image `ghcr.io/fluxcd/notification-controller:v1.4.0`;
+  `alerts`/`providers`/`receivers.notification.toolkit.fluxcd.io` all serve `v1beta3` (storage);
+  `get providers,alerts -A` → **`No resources found`**. This is the defect.
+- Event counts above, from `kubectl get events -A -o json` filtered to `toolkit.fluxcd.io`
+  involvedObjects: 101 `Kustomization/ReconciliationSucceeded`, 10 `Progressing`, 15
+  `DependencyNotReady`, 4 `GitRepository/GitOperationSucceeded`, 1 `NewArtifact`.
+- The `gotk_*` family **is** in the TSDB now (`flux-podmonitor` works): 6 targets up in job
+  `flux-system/flux-system`, `gotk_reconcile_condition` 120 series, `gotk_suspend_status` 40
+  (`{kind="Kustomization"}` = 16) — and none carries a revision.
+- `customResourceState` already works end to end: `kube_velero_schedule_{created,paused,
+  protects_volume_data,info}` = 2 series each.
+- Both CRDs serve **`v1` as the storage version**; `GitRepository/flux-system`
+  `.status.artifact.revision` = `main@sha1:b93594b7c85aeec4d28e008b239d61d181e1e6de`, `spec.interval`
+  `1m`; all 16 Kustomizations report that same `lastAppliedRevision`; **none** sets `spec.suspend`
+  (which is why `nilIsZero: true` is mandatory).
+- Reachability for the follow-up below: `flux-system/allow-egress` is `egress: [{}]` (allow-all) and
+  the `monitoring` namespace has **no NetworkPolicy at all**, so nothing blocks flux-system →
+  `kube-prometheus-stack-alertmanager` (`10.43.60.242:9093`).
+- `promtool test rules` on the new suite: **SUCCESS**, 53 rules parsed, promtool 3.14.0 from the
+  pinned toolbox. All 6 scenarios pass, including both guard tests and the convergence test that
+  proves the alert empties itself (the completion mail).
+- Full gate in the pinned toolbox: `lint`, `l0`, `render-golden`, `l1`, `decisions-log`,
+  `spec-immutability` all **green** — `l0` includes `alert-metrics-have-scrape-targets`,
+  `alert-rules-unit-tested` and `alert-runbooks-resolvable`, i.e. the new owner entry, the new tests
+  and the new runbook all cleared their own checks. Coverage 51/51.
+
+## What I did NOT verify
+
+- **No mail has been sent.** Nothing here has run on the cluster: the metrics do not exist yet, the
+  rules are not loaded, and the `email-deploy` receiver has never delivered. The end-to-end claim
+  ("a merge produces an `[FLUX] applying` mail and then an `[FLUX] applied` mail") is **UNVERIFIED**
+  and can only be proven by merging this and watching the *next* commit. Agents cannot apply, and
+  CI has no cluster (`vars.CLUSTER_AVAILABLE` unset → L2/L3 did not run).
+- **The KSM custom-resource-state paths are asserted, not observed.** I verified the CRD storage
+  versions and that each status field is populated on the live objects, but I have not seen KSM emit
+  `kube_flux_*`. If a path is wrong KSM emits the metric with the label **omitted**, `revision!=""`
+  then drops the series, and the alerts go quiet rather than wrong — the safe direction, but the
+  first thing to check after merge:
+  `count by (__name__) ({__name__=~"kube_flux.*"})` must return **three** names.
+- **KSM memory was not re-measured.** Limit is 384Mi against a working set of ~111–123Mi; this adds
+  two small informers (16 + 1 objects). 91 OOMKills followed the *last* CRS addition at a 128Mi
+  limit, so re-measure `max_over_time(container_memory_working_set_bytes{container="kube-state-metrics"}[6h])`
+  after this settles rather than assuming the headroom holds.
+- **Alertmanager's `.Status` subject template is untested.** `[FLUX] applying|applied` relies on
+  `.Status` being `firing`/`resolved` in the notification template data. Wrong, and the mail still
+  arrives with a poorer subject — cosmetic, but unproven.
+- **The `notification-controller` `Provider`/`Alert` half is designed but NOT shipped.** It was cut
+  deliberately: its unique value over the rules above is the human-readable **commit subject**
+  (`GitRepository`/`NewArtifact`, count 1 per commit, message `stored artifact for commit '<subject>'`,
+  revision in `source.toolkit.fluxcd.io/revision` metadata), and notification-controller has **no
+  SMTP provider**, so it must post into Alertmanager via `type: alertmanager` at
+  `http://kube-prometheus-stack-alertmanager.monitoring.svc:9093/api/v2/alerts`. I could not
+  establish **what label set that provider actually emits** (whether it sets `alertname`, and whether
+  `severity` is `info`) without applying it, and the `email-deploy` route matches on `severity`. So
+  the route design would have rested on a guess. Follow-up: apply it, read the resulting labels off
+  Alertmanager, then wire the route — and put it in its **own `wait: false` Flux Kustomization**
+  (never `platform` or `platform-config`, both `wait: true`), because a notification mechanism that
+  cannot go Ready must never be able to stall the delivery graph, which is the twice-shipped
+  cold-start deadlock this repo already documents.
+- **The 2500-count drift loop is NOT fixed, and is not mine.** Five Kustomizations have been
+  re-applying the same object every 10 minutes for ~25 days: something outside git is mutating
+  `NetworkPolicy/{unifi,gitea,nextcloud,seerr}/default-deny` and
+  `CustomResourceDefinition/cdis.cdi.kubevirt.io` after Flux applies them. Cause **UNVERIFIED** —
+  I only measured the event counts. It is a real defect (constant write load on the API server and
+  etcd, on a cluster with a known etcd I/O problem on prod-1) and it deserves its own branch. The
+  alerts here are immune to it by construction: they read revisions, not events.
+- **Branch protection / runner state re-measured? No.** The `gitea-settings` stage **SKIPPED**
+  (`SETTINGS_API_TOKEN` unset), so whether `main` is still protected and the gate still required is
+  UNVERIFIED by this run. The gate's overall verdict is therefore **rc 3, not rc 0** — every stage
+  that ran is green, and one stage did not run.

decisions/2026-09-30-flux-deploy-notification-inhibited.md

@@ -0,0 +1,95 @@
+---
+date: 2026-09-30
+branch: fix/flux-deploy-notification-inhibited
+title: The deploy notification routed correctly and was then inhibited before sending — `severity: info` is a suppressed class in kube-prometheus-stack
+---
+
+## What changed
+
+`FluxRolloutInProgress` (shipped hours earlier in PR #25) fired correctly on the very first real
+deploy and **its mail was suppressed before Alertmanager sent it**. One label changed:
+
+- `platform/monitoring/alerts.yaml` — `FluxRolloutInProgress` now carries `notification: deploy` and
+  **no `severity` label at all**, where it previously carried `severity: info`.
+- `platform/monitoring/kube-prometheus-stack.yaml` — the deploy route matches
+  `notification = "deploy"` instead of `severity = "info"`. Receiver, subject and grouping unchanged.
+- `platform/monitoring/tests/flux-rollout.test.yaml` — the three `FluxRolloutInProgress` `exp_labels`
+  blocks updated. `FluxRolloutStuck` keeps `severity: warning` and is unaffected.
+
+**The mechanism, measured rather than reasoned about.** kube-prometheus-stack's `general.rules`
+ships:
+
+    InfoInhibitor:
+      group by (namespace) (ALERTS{severity="info"} == 1)
+        unless on (namespace) group by (namespace)
+          (ALERTS{alertname!="InfoInhibitor", alertstate="firing", severity=~"warning|critical"} == 1)
+
+which fires for any namespace holding a firing `severity="info"` alert **and nothing else wrong in
+it**. The chart's third `inhibit_rule` then drops exactly that alert:
+
+    source_matchers: [alertname="InfoInhibitor"]
+    target_matchers: [severity="info"]
+    equal:           [namespace]
+
+For a deploy notification this is the worst possible interaction, because the suppressed case is the
+*only* case that matters: on a healthy rollout of a healthy cluster, `flux-system` holds no warning
+or critical alert, so `InfoInhibitor` fires and the deploy mail is inhibited. It would have been
+delivered only when something else in `flux-system` was already broken.
+
+Upstream's intent is explicit and reasonable — `severity: info` means "only worth showing next to a
+real problem". A deploy notice is not that, so it must not claim that severity. `notification` is a
+label **no** `inhibit_rule` in the chart targets (all three match on severity values), and dropping
+`severity` also stops `InfoInhibitor` firing for `flux-system` at all rather than merely working
+around it. Both files carry the full reasoning inline, and `alerts.yaml`'s design header now names
+the severity label as a third thing that must not regress — "do not tidy this by adding a severity"
+— because the next reader's instinct will be to add one back.
+
+## What I verified
+
+Against live prod, after PR #25 merged as `64c72fd` and Flux converged:
+
+- **The metrics work.** `kube_flux_kustomization_info` 16, `kube_flux_kustomization_suspended` 16,
+  `kube_flux_gitrepository_info` 1 = **33 series, exactly as designed**. The `revision` label IS
+  populated (`main@sha1:296d6d3…` / `…64c72fd…`), so the `status.lastAppliedRevision` /
+  `status.artifact.revision` paths are correct — previously UNVERIFIED.
+- **The `podAnnotations` roll mechanism works.** The KSM pod carries
+  `homelab.keithjackson.dev/crs-config-rollout: 1` and is a new pod
+  (`…-7b56d98f46-56n8j`, replacing the 2026-09-19 one), so the config-only-change trap was
+  genuinely closed rather than merely documented.
+- **The rule works, in both directions.** It fired for **12** lagging Kustomizations while 4 had
+  converged (`count by (revision)` → 12 old / 4 new), each alert carrying its own `name` and
+  `revision`; then the rollout converged (`count(count by (revision) (…))` → **1**) and the alert
+  **cleared**. Firing and self-clearing are both confirmed on real data, which is the completion
+  notification's whole mechanism.
+- **The routing works.** `amtool config routes` shows
+  `{notification="deploy"} → receiver: email-deploy` under the default route, and
+  `amtool config routes test notification=deploy` returns `email-deploy`.
+- **The inhibition was real.** `InfoInhibitor` went active `2026-09-30T04:58:48Z` — the same moment
+  the alert began firing — and `amtool alert query` showed **no** `FluxRolloutInProgress` in
+  Alertmanager, active or `--inhibited`. `alertmanager_notifications_failed_total{integration=email}`
+  is 0 for every reason, so nothing failed to send; it was never dispatched.
+- promtool: **SUCCESS**, 53 rules, 6 scenarios, promtool 3.14.0 in the pinned toolbox.
+
+## What I did NOT verify
+
+- **No deploy mail has been observed in the inbox — this fix is UNVERIFIED end to end.** The
+  suppression was proven (nothing in Alertmanager, zero send failures) and the routing was proven
+  (`amtool config routes test` → `email-deploy`), but the delivered message is not. It can only be
+  proven by merging this and watching the next commit produce `[FLUX] applying` and then
+  `[FLUX] applied`. Same structural gap as PR #25, one layer further along.
+- **Whether the single email during the window was ours.**
+  `alertmanager_notifications_total{integration="email"}` went 226 → 227 while other alerts
+  (`Watchdog`, `CNPGPrimariesCoLocated`) were also active, and that counter carries no receiver
+  label, so the increment cannot be attributed. I am NOT claiming our mail was or was not sent — the
+  attribution is genuinely unavailable from this metric.
+- **Whether `InfoInhibitor` interacts with `FluxRolloutStuck`.** Stuck is `severity: warning`;
+  inhibit_rule 1 (`critical` → `warning|info`) requires an equal `alertname`, so it should not
+  apply — reasoned, not measured. Worth confirming the first time a rollout genuinely hangs.
+- **The `[FLUX] applying|applied` subject template** is still untested; `.Status` is assumed to be
+  `firing`/`resolved` in the notification template data.
+- **`required_approvals` disagreement, unresolved and not mine to settle.** Live branch protection
+  reads `required_approvals: 1`, while the `ci/gate.sh` comment introduced by PR #17 describes the
+  model as `required_approvals: 0` ("the gate is the approver"). `ci/gitea-settings.sh` may report
+  this as drift against `infra/gitea/expected-branch-protection.json`. Flagged only.
+- **The drift loop is still unexplained.** Five Kustomizations re-applying the same object every
+  10 minutes (event counts ~2500) — untouched, cause unknown, still deserves its own branch.

decisions/2026-09-30-oauth2-proxy-auth-failure-visible.md

@@ -0,0 +1,175 @@
+---
+date: 2026-09-30
+branch: feat/oauth2-proxy-auth-failure-visible
+title: The Jackett auth failure was invisible because the gates emit nothing — alert on the gap, not on a metric that does not exist
+---
+
+## What changed
+
+Jackett's UI broke in the way that costs the most: **HTTP 200 and a plausible page**. The
+operator's report named the property that matters — *"reloading the page never fixes the issue and
+the user is never re-prompted for login"*. Two sibling branches fix the auth defects. This one is
+about the fact that **no signal existed anywhere**, and it turns out that is not an oversight in
+the alert rules. It is an absence of data.
+
+**The measurement that decided the whole design** (prod, live Prometheus, 2026-09-30):
+
+    count by (__name__) ({__name__=~"oauth2_proxy.*"})
+      oauth2_proxy_requests_in_flight => 2
+
+Two series. Both from the **shared** forward-auth Deployment in the `oauth2-proxy` namespace.
+That is every oauth2-proxy metric this cluster holds. The six per-app gate sidecars — crafty,
+jackett, pihole, qbittorrent, radarr, sonarr — publish **nothing at all**: they run with
+`--http-address=0.0.0.0:4180` and no `--metrics-address`, so oauth2-proxy's metrics listener never
+starts; they declare only a `http:4180` container port; and no ServiceMonitor or PodMonitor in the
+cluster selects them.
+
+So the alert the brief asked for — *"a gated app is serving a sustained rate of auth redirects to
+a user who is authenticated"* — **is not writable**, and writing it anyway would have produced
+exactly the artefact this repo keeps getting burned by: a rule that is present, evaluating,
+permanently silent, and indistinguishable from health. It was not written.
+
+What was written instead alerts on the condition that made the incident invisible, from families
+that **are** scraped:
+
+- **`platform/oauth2-proxy/auth-visibility-alerts.yaml`** — `OAuth2ProxyGateUnobserved`:
+
+      count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
+        unless
+      count by (namespace) (up{container="oauth2-proxy"})
+
+  `for: 1h`, `severity: warning`. `kube_pod_container_info` is kube-state-metrics (a declared
+  owner in `alert-metrics-have-scrape-targets`) and sees the container whether or not it exports
+  anything; `up` is UNIVERSAL and carries a `container` label for any scrape target, so the rule
+  needs no advance knowledge of what job name a future per-app scrape will use. Nothing in it
+  touches `oauth2_proxy_*`, so it cannot be inert for the reason it is reporting. **It returns six
+  namespaces against live prod right now** and excludes `oauth2-proxy`, which is scraped. It
+  clears per-namespace as each gate gains a real scrape.
+- **`platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml`** — four states pinned:
+  fires for an unscraped namespace, never for the scraped one, honours `for: 1h` (pending at 59m),
+  and **stops** once a scrape appears under a *different* job name. Series shapes copied off live
+  prod, not invented.
+- **`runbooks/oauth2-proxy-auth-failures.md`** — how to tell "the gate refused me" from "the group
+  check refused me" from "the app refused me" from "the XHR silently got HTML", with the
+  2026-09-30 jackett trace as the worked example. Registered in `runbooks/kustomization.yaml`'s
+  `configMapGenerator` and listed in `runbooks/README.md`.
+- **`runbooks/README.md`** — a new row in *"Alerts you should not trust the silence of"* for the
+  per-app sidecars, plus the measured reason.
+- **`platform/oauth2-proxy/README.md`** — what this component can and cannot see, and the exact
+  three-part change that would close the gap.
+
+**Why the obvious alternative was rejected.** An alert on the *shared* proxy's
+`oauth2_proxy_requests_total{code="302"}` ratio is the one a reader expects to see here. It would
+need a new `SCRAPE_OWNERS` entry in `conformance/**` (a sensitive path needing human approval),
+and — decisively — **it would not have fired on this incident**: the shared instance is not in the
+data path for any sidecar-gated app, and its current pods export no `oauth2_proxy_requests_total`
+at all, having proxied nothing since they started (the counter exists in the TSDB only from an
+older ReplicaSet, `oauth2-proxy-78449b5657-*`, with `code` labels `302` and `401` and no `202`).
+Shipping it would have looked like coverage of the reported failure and been none.
+
+## What I verified
+
+Everything below was measured against live prod through the pinned container, read-only. No
+`apply`, no `edit`, no `patch`, no `flux reconcile`.
+
+- **The full oauth2-proxy metric inventory.** `count by (__name__) ({__name__=~"oauth2_proxy.*"})`
+  → `oauth2_proxy_requests_in_flight => 2`. `count by (__name__) ({job="oauth2-proxy"})` returns
+  go_*/process_*/promhttp_* plus that one family and nothing else. **Only the shared instance is
+  scraped**: `up{job="oauth2-proxy"}` → 2 targets, `10.0.1.189:44180` and `10.0.2.47:44180`, both
+  `1`, `namespace="oauth2-proxy"`.
+- **The per-app sidecars are unscraped, and unscrapable as configured.** Every pod carrying an
+  `oauth2-proxy` container, with its declared ports: the two shared replicas have
+  `[('http',4180),('metrics',44180)]`; crafty, jackett, pihole, qbittorrent, radarr and sonarr all
+  have `[('http',4180)]` only. The live jackett sidecar's args contain no `--metrics-address`.
+  `kubectl get servicemonitors,podmonitors -A` lists 16 ServiceMonitors and 5 PodMonitors; the
+  only oauth2-proxy entry is `oauth2-proxy/oauth2-proxy`.
+- **No existing alert is inert *for this reason*, because no alert references these metrics at
+  all.** `grep -rn oauth2_proxy --include=*.yaml` over the tree hits only cookie-name strings in
+  `platform/crafty/`. The 39 live PrometheusRules include `platform-health-identity`, whose four
+  rules are about Keycloak readiness and the client reconciler and are all keyed on `kube_*`
+  (scraped, fine). **The inertness here is the absence of a rule, not a broken one** — which is
+  why the runbook README's inert-alerts table gains a row phrased like the etcd one.
+- **Keycloak cannot express it either.** `count by (__name__) ({__name__=~"keycloak_.*"})` →
+  empty, and empty over `[30d]`. The `keycloak-service` scrape yields JVM/Agroal/JGroups metrics
+  only; there are no user-event or login counters, so "a burst of re-logins for one client" is not
+  observable from the IdP side.
+- **`oauth2_proxy_requests_total` has existed historically but not on the current pods.**
+  `count_over_time(oauth2_proxy_requests_total[30d])` returns three series, all from
+  `oauth2-proxy-78449b5657-*` (a prior ReplicaSet): `code="302"` on both replicas and `code="401"`
+  on one, ~634 samples each (~5h of a 30d window). **No `code="202"` ever.**
+- **The alert expression fires against live data.** Run verbatim on prod:
+  `count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by
+  (namespace) (up{container="oauth2-proxy"})` → `pihole, jackett, crafty, qbittorrent, sonarr,
+  radarr`, value 1 each; `oauth2-proxy` correctly absent.
+- **The log signal exists and is collected.** Loki (`loki-0`, SingleBinary) and a 3-node Alloy
+  DaemonSet are running, and Alloy labels pod logs with `namespace`/`container`. LogQL over 24h,
+  `{container="oauth2-proxy"}`: crafty 12036 lines, jackett 11669, oauth2-proxy 34559, pihole
+  11520, qbittorrent 11520, radarr 11763, sonarr 11520. Filtered to `|= "No valid authentication
+  in request"`: **crafty 81, jackett 39, radarr 67**; to `|= "AuthSuccess"`: **crafty 1, jackett
+  4, radarr 1**. Those ratios (81:1, 10:1, 67:1) are the incident's signature and are in the
+  runbook as the thing to compare against.
+- **Why `kubectl logs` hid it.** The jackett sidecar's stdout is ~11.5k lines/day and almost all
+  of it is `GET "/ping" ... 200 2` from the kubelet probe, twice every 15s. A `--tail=40` shows
+  forty `/ping` lines and looks perfectly healthy. The runbook's commands all filter.
+- **Runbook registration, both halves.** `- oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md`
+  is in `runbooks/kustomization.yaml`'s `configMapGenerator.files`, and the page is listed under
+  *Identity* in `runbooks/README.md`. The page clears the L0 quality floor (the four operator
+  sections, >1500 bytes, mentions `dev.sh` and `--context k3s-prod`), and the alert's
+  `runbook_url` points at exactly that name.
+- **Gate:** run in the pinned toolbox from this worktree with the `ENGINE` shim. Exit code and
+  per-stage outcomes in the next section — read them there, not here.
+
+## What I did NOT verify
+
+- **The alert has never fired in Alertmanager.** Nothing is applied; the PrometheusRule exists
+  only in git on this branch. What is proven is that its expression returns the expected vector
+  against live prod data and that promtool agrees about its behaviour over synthetic series.
+  **Merging is what deploys it.**
+- **`OAuth2ProxyGateUnobserved` would NOT have fired *during* the 2026-09-30 jackett incident in
+  any useful sense** — it describes a standing condition that has been true since the sidecars
+  were deployed, so it would have been firing before, during and after. That is deliberate and it
+  is the honest claim: it is a **gap tracker**, not an incident detector. It would have meant that
+  on 2026-09-30 an operator asking "why did nothing alert?" had the answer already on the board
+  instead of discovering it during the investigation. **No alert on this platform could have
+  detected the incident itself, and none can today.**
+- **The alert that the brief actually asked for does not exist, and cannot be written yet.** The
+  sustained-redirect-ratio rule needs `oauth2_proxy_requests_total` from the *per-app* sidecars.
+  **Enabling that requires editing files owned by the two sibling branches** — the sidecar args in
+  `apps/wave-0/jackett.yaml` (Mira Vance) and `apps/wave-0/{qbittorrent,pihole}.yaml`,
+  `apps/wave-1/{radarr,sonarr}.yaml` (Theo Bright) — so it was **not attempted here**. The change
+  is three things per app, together: `--metrics-address=0.0.0.0:44180`, a `metrics` containerPort
+  plus Service port, and a ServiceMonitor. Once any one app has them, the rule belongs beside
+  `OAuth2ProxyGateUnobserved`, and that app's namespace drops out of it automatically.
+- **The Loki alerting path is inert and I did not fix it.** Loki *has* a ruler stanza and the
+  chart's `loki-sc-rules` sidecar is running with `LABEL=loki_rule`, `FOLDER=/rules` — so a
+  ConfigMap-shipped LogQL rule would be delivered. But `GET /loki/api/v1/rules` answers **"unable
+  to read rule dir /var/loki/rules/fake: no such file or directory"** (the ruler reads a different
+  directory from the one the sidecar writes), and the rendered config contains **no
+  `alertmanager_url`** — `grep -i 'alertmanager\|rule_path'` over the `loki` ConfigMap returns
+  nothing. So a LogQL alert on `|= "No valid authentication in request"` — which *would* have
+  caught this incident, and is the single highest-value follow-up here — **would evaluate and
+  notify nobody**. Wiring it means `rulerConfig` (alertmanager_url + a matching rules directory)
+  in `platform/monitoring/loki.yaml`, which is not a file this brief gave me. **I did not ship an
+  inert rule ConfigMap to look productive.**
+- **Nothing was reproduced in a browser.** The 302-to-200-with-HTML mechanism is read off the
+  sidecar access log and the response sizes (`/UI/Dashboard 200 44379`), not observed in devtools.
+- **Loki retention is 72h** (`platform/monitoring/loki.yaml`), so the 24h ratios above are a
+  snapshot; anything older than three days is already gone. This is stated in the runbook because
+  it bounds what an operator can still investigate.
+- **The gate's L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), so no runtime
+  requirement touched by this branch is certified. See the exit code below — **rc 3 is
+  UNDETERMINED, not green.**
+
+### Follow-ups, in order of value
+
+1. **Wire the Loki ruler to Alertmanager** (`platform/monitoring/loki.yaml`). This is the only
+   route to an alert that would have caught the actual 2026-09-30 failure, and the data is
+   already being collected. The rule is one expression:
+   `sum by (namespace) (rate({container="oauth2-proxy"} |= "No valid authentication in request" [15m]))
+   > 5 * sum by (namespace) (rate({container="oauth2-proxy"} |= "AuthSuccess" [15m]))`.
+2. **Give one gate sidecar a metrics endpoint** and watch its namespace drop out of
+   `OAuth2ProxyGateUnobserved`. That is the cheapest proof the alert is a tracker and not a nag,
+   and it unlocks the ratio rule in Prometheus.
+3. **Consider whether `--set-xauthrequest` apps should get `--api-routes`** so API paths answer
+   401 instead of 302. That is the fix that would make this failure *loud* rather than merely
+   observable, and it belongs with whoever owns the sidecar args.

infra/gitea/approval-policy.json

@@ -0,0 +1,28 @@
+{
+  "_intent": "The CI/CD pipeline is the approver. A human approval is required ONLY for changes that alter what the pipeline itself verifies, or the security posture it rests on -- because those are the changes CI cannot honestly gate: a PR that weakens the gate would be approved by the weakened gate.",
+  "_rule": "If a PR touches any sensitive_paths glob, it needs an APPROVED review from someone who is not its author. Everything else merges on a green gate alone.",
+  "_on_failure_to_evaluate": "BLOCK. Unlike gitea-settings, a skip here is a bypass, not a gap.",
+  "sensitive_paths": [
+    "ci/**",
+    ".gitea/**",
+    "conformance/**",
+    "clusters/**",
+    "bootstrap/**",
+    "infra/gitea/**",
+    "platform/kyverno/**",
+    "platform-config/kustomization.yaml",
+    ".sops.yaml",
+    "**/*.sops.yaml",
+    "**/rbac.yaml",
+    "CLAUDE.md"
+  ],
+  "_why_each": {
+    "ci/**, .gitea/**": "the pipeline itself -- the thing doing the approving",
+    "conformance/**": "the tests and the harness; changing them changes what green MEANS",
+    "clusters/**, bootstrap/**": "Flux wiring -- what reconciles, in what order, with what substitution",
+    "infra/gitea/**, platform-config/kustomization.yaml": "branch protection, the drift detector, and the list that decides which admission policies are actually applied",
+    "platform/kyverno/**": "the admission policy engine",
+    ".sops.yaml, **/*.sops.yaml, **/rbac.yaml": "encryption recipients, secrets, and cluster authority",
+    "CLAUDE.md": "the delivery contract agents follow"
+  }
+}

platform/monitoring/alerts.yaml

@@ -799,6 +799,123 @@ spec:
             summary: "Flux resource suspended: {{ $labels.kind }}/{{ $labels.name }}"
             description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has been suspended for 1h — reconciliation/drift-correction is off."
             runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-suspended"
+        # ===================================================================================
+        # DEPLOY NOTIFICATIONS (added 2026-09-29). The only rules in this repo whose PURPOSE is to
+        # send mail on the HEALTHY path: the operator asked to be told when Flux starts applying a
+        # git change and when it finishes. Read the two together — they share one expression and
+        # differ only in `for` and severity.
+        #
+        # WHY A RULE AND NOT A FLUX Alert/Provider. Neither Flux event can be filtered into a
+        # deploy signal. Measured on prod 2026-09-29: a Kustomization emits ReconciliationSucceeded
+        # on EVERY interval tick (101 events in one hour across 16 Kustomizations) with the message
+        # "Reconciliation finished in 324ms, next run in 10m0s" — byte-identical whether the apply
+        # changed anything or not; and `Progressing` is worse, because five Kustomizations re-apply
+        # the same object on every tick (event counts 2505/2504/2501/2495/2490), so anything keyed
+        # on it mails five times every ten minutes forever. The applied REVISION is the only signal
+        # that moves when and only when git moves. See the kube_flux_* block in
+        # kube-prometheus-stack.yaml for how it reaches the TSDB.
+        #
+        # THE EXPRESSION. `unless on (revision)` keeps a Kustomization only while its applied
+        # revision is absent from the set of revisions the GitRepository has fetched — i.e. while it
+        # is behind main. It empties itself as each Kustomization catches up, which is what makes
+        # the RESOLVED notification mean "the rollout completed" rather than "the alert timed out".
+        # Alertmanager's email-deploy receiver carries send_resolved: true for exactly that reason,
+        # and group_by: [alertname] collapses all 16 Kustomizations into one mail per rollout.
+        #
+        # TWO GUARDS, BOTH LOAD-BEARING, BOTH UNIT-TESTED (tests/flux-rollout.test.yaml):
+        #   * `and on () (count(...) > 0)` — if kube_flux_gitrepository_info is ABSENT (KSM rolled
+        #     without the ConfigMap, the flux RBAC lost, customResourceState reverted) then
+        #     `unless on (revision) <empty>` removes NOTHING and all 16 Kustomizations alert
+        #     forever, for no reason. That is the permanently-firing failure mode this repo already
+        #     shipped once (CNPG, 2026-09-12). The guard makes the missing-exporter case SILENT,
+        #     which is the correct direction for a notification — a broken exporter is what
+        #     FluxReconciliationFailure and the platform-health.monitoring-self rules are for.
+        #   * `unless on (namespace, name) (kube_flux_kustomization_suspended == 1)` — a suspended
+        #     Kustomization pins an old revision BY DESIGN, and would otherwise hold a rollout open
+        #     for as long as it stays suspended, so the resolved mail (the "it finished" half of the
+        #     feature) would never arrive. FluxResourceSuspended above is what reports that state.
+        #
+        # `revision!=""` on both sides: KSM omits a label whose path is absent, so a Kustomization
+        # that has never applied anything — or a GitRepository with no artifact yet — publishes the
+        # series with no `revision` label. Without the matcher those group together under the empty
+        # string and compare EQUAL, i.e. a never-applied Kustomization would read as converged.
+        #
+        # A THIRD THING THAT MUST NOT REGRESS — THE SEVERITY LABEL. These alerts are delivered
+        # because of the labels they do NOT carry as much as the ones they do. Do not "tidy" this
+        # pair by giving FluxRolloutInProgress a `severity`, and never `severity: info`: the chart's
+        # InfoInhibitor rule plus its `severity="info"` inhibit_rule suppress an info alert whenever
+        # it is the only thing firing in its namespace, which is every healthy deploy. Measured on
+        # prod 2026-09-30 — the alert fired correctly for 12 Kustomizations, resolved correctly, and
+        # was inhibited in between. The full reasoning is on the `notification: deploy` label below.
+        # ===================================================================================
+        - alert: FluxRolloutInProgress
+          expr: |
+            (
+              (
+                kube_flux_kustomization_info{revision!=""}
+                unless on (namespace, name) (kube_flux_kustomization_suspended == 1)
+              )
+              unless on (revision) kube_flux_gitrepository_info{name="flux-system", revision!=""}
+            )
+            and on () (count(kube_flux_gitrepository_info{name="flux-system", revision!=""}) > 0)
+          # NO `for:`, ON PURPOSE. Every other rule in this file waits, because every other rule
+          # reports a fault and a fault that clears in 30s is noise. This one reports an EVENT the
+          # operator asked to be told about, so the failure mode to avoid is the opposite: a `for:`
+          # of even 1m would silently swallow any rollout that converged faster than that (a manual
+          # `flux reconcile`, a change touching one Kustomization), and a deploy notification that
+          # only sometimes arrives is worse than no notification at all.
+          labels:
+            # ROUTED BY `notification`, AND DELIBERATELY CARRYING NO `severity` (fixed 2026-09-30).
+            # This alert originally used `severity: info`. That was WRONG, and wrong in the silent
+            # direction — measured on prod within minutes of the first real deploy.
+            # kube-prometheus-stack ships `general.rules` -> InfoInhibitor:
+            #     group by (namespace) (ALERTS{severity="info"} == 1)
+            #       unless on (namespace) group by (namespace)
+            #         (ALERTS{alertname!="InfoInhibitor", alertstate="firing", severity=~"warning|critical"} == 1)
+            # which fires for any namespace that has a firing info alert AND NOTHING ELSE WRONG in
+            # it; paired with the chart's inhibit_rule
+            #     source_matchers: [alertname="InfoInhibitor"]
+            #     target_matchers: [severity="info"]
+            #     equal: [namespace]
+            # the effect is that an info alert is suppressed EXACTLY WHEN it is the only thing
+            # happening in its namespace. For a deploy notification that is the only case that
+            # matters: on a healthy rollout of a healthy cluster, flux-system has no warning or
+            # critical alert, so InfoInhibitor fired and this alert never reached the receiver.
+            # Verified live: InfoInhibitor went active 2026-09-30T04:58:48Z, the same moment this
+            # alert began firing for 12 Kustomizations.
+            # Upstream's intent is explicit — `severity: info` means "only worth showing next to a
+            # real problem". A deploy notice is not that, so it must not claim that severity.
+            # `notification` is a label no inhibit_rule in the chart targets (all three match on
+            # severity values), and omitting `severity` also stops InfoInhibitor firing for
+            # flux-system at all. Route: platform/monitoring/kube-prometheus-stack.yaml matches
+            # `notification = "deploy"` -> the email-deploy receiver, subject "[FLUX] applying|applied".
+            notification: deploy
+          annotations:
+            summary: "Flux is applying a new revision — {{ $labels.name }} is behind main"
+            description: "Kustomization {{ $labels.namespace }}/{{ $labels.name }} has applied {{ $labels.revision }}, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
+        # The escalation: a rollout that STARTED and never finished. Same expression, so it cannot
+        # disagree with the notification above about what "behind main" means; 30m is comfortably
+        # past the slowest legitimate convergence (the longest Kustomization interval is 1h for
+        # cilium, but its dependents are 10m and a wave chain converges well inside 30m).
+        # severity: warning, so this one takes the normal email-warning path and reads as a fault.
+        - alert: FluxRolloutStuck
+          expr: |
+            (
+              (
+                kube_flux_kustomization_info{revision!=""}
+                unless on (namespace, name) (kube_flux_kustomization_suspended == 1)
+              )
+              unless on (revision) kube_flux_gitrepository_info{name="flux-system", revision!=""}
+            )
+            and on () (count(kube_flux_gitrepository_info{name="flux-system", revision!=""}) > 0)
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Flux rollout has not completed: {{ $labels.name }} stuck on an old revision"
+            description: "Kustomization {{ $labels.namespace }}/{{ $labels.name }} has been behind the flux-system GitRepository revision for 30m (applied {{ $labels.revision }}) — a git change started rolling out and has not converged. Check the Kustomization's Ready condition and its dependencies."
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
 
 # =====================================================================================
 # Certificates — cert-manager

platform/monitoring/kube-prometheus-stack.yaml

@@ -131,6 +131,44 @@ spec:
               # page-worthy — notify fast, remind often
               group_wait: 10s
               repeat_interval: 1h
+            # DEPLOY NOTICES, not health alerts (added 2026-09-29). FluxRolloutInProgress fires
+            # the moment the applied revision diverges from main and RESOLVES when every
+            # Kustomization has caught up — so the FIRED mail is "a deploy started" and the
+            # RESOLVED mail is "it finished". Both halves are the product, which is why
+            # send_resolved must stay true on the receiver below.
+            #
+            # A DEDICATED ROUTE, because the default receiver is `email-warning`: without it a
+            # routine deploy arrives subject-lined "[WARNING]" and becomes indistinguishable from a
+            # real platform fault, which is how alert mail stops being read at all.
+            #
+            # MATCHES `notification`, NOT `severity` (corrected 2026-09-30). This route first read
+            # `severity = "info"`, which routed correctly and then had the mail SUPPRESSED before it
+            # was sent: the chart's own InfoInhibitor rule fires for any namespace holding a firing
+            # info alert with no warning/critical beside it, and its inhibit_rule
+            # (source alertname="InfoInhibitor" -> target severity="info", equal [namespace]) drops
+            # exactly that alert. A healthy deploy in a healthy flux-system is precisely that case,
+            # so the notification was inhibited on every occasion it was wanted. Measured live on
+            # prod 2026-09-30: InfoInhibitor went active at 04:58:48Z as the alert began firing.
+            # Routing on a non-severity label sidesteps all three of the chart's inhibit_rules,
+            # which match on severity values only. See the label's own comment in alerts.yaml.
+            - matchers: ['notification = "deploy"']
+              receiver: email-deploy
+              # One mail per ROLLOUT, not per Kustomization: the alert fires once per lagging
+              # Kustomization (16 possible), and grouping on alertname alone collapses them into a
+              # single mail that lists all of them. The default group_by would have split them by
+              # namespace as well — harmless today (all 16 are in flux-system) and wrong the moment
+              # a Kustomization lives elsewhere.
+              group_by: ["alertname"]
+              group_wait: 30s
+              # 2m, not the 5m default: group_interval also bounds how long a group's RESOLVED
+              # notification waits, and holding "the deploy finished" four minutes behind the
+              # deploy finishing makes the completion half of this feature feel broken.
+              group_interval: 2m
+              # A deploy notice is a statement about one revision, so re-sending it on the 12h
+              # default would just repeat old news. 24h is long enough never to fire twice for one
+              # rollout, while still re-nagging if a rollout is genuinely stuck for a whole day
+              # (FluxRolloutStuck, severity warning, is the real escalation for that case).
+              repeat_interval: 24h
         receivers:
           - name: email-warning
             email_configs:
@@ -144,6 +182,19 @@ spec:
                 send_resolved: true
                 headers:
                   subject: '[CRITICAL] {{ .CommonLabels.alertname }} ({{ .CommonLabels.namespace }})'
+          # Flux deploy notices (the `severity = "info"` route above).
+          - name: email-deploy
+            email_configs:
+              - to: "[email protected]"    # alert inbox (operator: change if a different inbox is wanted)
+                # BOTH transitions are the deliverable here: firing = "Flux started applying a new
+                # revision", resolved = "every Kustomization reached it". Turning this off would
+                # delete the completion notification, not merely reduce noise.
+                send_resolved: true
+                headers:
+                  # `[FLUX] applying` / `[FLUX] applied` — readable in a notification preview
+                  # without opening the mail, and never the word WARNING for a healthy deploy.
+                  # .Status is "firing" or "resolved" in Alertmanager's notification template data.
+                  subject: '[FLUX] {{ if eq .Status "firing" }}applying{{ else }}applied{{ end }} — {{ .CommonLabels.alertname }}'
 
     # --- Grafana: dashboards (R-OBS-03), surfaced in the C2 UI (docs/12) behind Keycloak SSO ---
     grafana:
@@ -272,11 +323,33 @@ spec:
       #     `config` alone does not restart the pod: after editing it, `kubectl -n monitoring rollout
       #     restart deploy/kube-prometheus-stack-kube-state-metrics`. Enabling it the first time
       #     changes the pod spec (new arg + volume), so the initial rollout is automatic.
+      # FORCE A POD ROLL WHEN `customResourceState.config` BELOW CHANGES. The chart renders that
+      # config into a ConfigMap and mounts it, but the KSM Deployment carries NO checksum annotation
+      # over it — documented in the block above, and the reason the Velero metrics needed a manual
+      # `rollout restart` when they were first tuned. So a PR that edits ONLY `config` changes the
+      # ConfigMap, leaves the pod spec byte-identical, and KSM keeps running the OLD collector set:
+      # the new metrics never appear and every alert reading them is inert, while the PR is merged
+      # and green. That is the exact shape of defect this repo keeps getting bitten by, so it is
+      # closed declaratively rather than by remembering an operator step. BUMP THIS VALUE IN THE
+      # SAME COMMIT as any `customResourceState.config` edit and Flux rolls the pod for you.
+      # (Same mechanism as grafana.podAnnotations/oidc-rollout above.)
+      podAnnotations:
+        homelab.keithjackson.dev/crs-config-rollout: "1"
       rbac:
         extraRules:
           - apiGroups: ["velero.io"]
             resources: ["schedules"]
             verbs: ["get", "list", "watch"]
+          # Flux applied-revision metrics (the kube_flux_* block at the end of `config` below).
+          # MANDATORY for the same measured reason as velero.io above: without read access KSM
+          # publishes the custom-resource-state metrics as an EMPTY SET and logs a forbidden
+          # error, which would leave FluxRolloutInProgress/FluxRolloutStuck inert — silently.
+          - apiGroups: ["kustomize.toolkit.fluxcd.io"]
+            resources: ["kustomizations"]
+            verbs: ["get", "list", "watch"]
+          - apiGroups: ["source.toolkit.fluxcd.io"]
+            resources: ["gitrepositories"]
+            verbs: ["get", "list", "watch"]
       customResourceState:
         enabled: true
         config:
@@ -328,6 +401,89 @@ spec:
                         labelsFromPath:
                           cron: [spec, schedule]
                           phase: [status, phase]
+              # --- FLUX APPLIED REVISION AS METRICS (added 2026-09-29) --------------------------
+              # WHY. Nothing in the TSDB could answer "is the cluster running the commit that is on
+              # main?", so "Flux started applying a git change" and "Flux is idle" were
+              # indistinguishable and a deploy could start, stall halfway and never finish with
+              # nothing to say so. Measured on prod 2026-09-29, all three candidate signals fail:
+              #   * gotk_* (scraped by flux-podmonitor.yaml) carries NO revision label anywhere —
+              #     gotk_reconcile_condition 120 series, gotk_suspend_status 40, not one with a
+              #     revision. It can say a Kustomization is unhealthy, never which commit it holds.
+              #   * the EVENT stream cannot be filtered into a deploy signal: a Kustomization emits
+              #     ReconciliationSucceeded on EVERY interval tick — 101 events in one hour across
+              #     16 Kustomizations — with the message "Reconciliation finished in 324ms, next run
+              #     in 10m0s", byte-identical whether the apply changed anything or not.
+              #   * `Progressing` is worse, not better: five Kustomizations re-apply the same object
+              #     on every single tick (2505/2504/2501/2495/2490 on one event object each), so
+              #     anything keyed on it fires five times every ten minutes, forever.
+              # The applied REVISION is the one signal that moves only when git moves, and it lives
+              # in these two objects' status and nowhere else:
+              #   GitRepository/flux-system  .status.artifact.revision   = what main is (poll 1m)
+              #   Kustomization/*            .status.lastAppliedRevision = what is applied
+              # Exporting both makes "applied != desired" a PromQL question, which is what
+              # FluxRolloutInProgress / FluxRolloutStuck ask (platform/monitoring/alerts.yaml).
+              #
+              # NAMING: kube_flux_*, NOT gotk_* — the same rule as kube_velero_schedule_* above.
+              # These are kube-state-metrics series ABOUT Kubernetes objects and must never be
+              # mistaken for the gotk_* family the Flux controllers themselves export.
+              #
+              # VERIFIED ON PROD before writing this (2026-09-29): both CRDs serve `v1` as the
+              # STORAGE version (kustomizations.kustomize.toolkit.fluxcd.io,
+              # gitrepositories.source.toolkit.fluxcd.io) and both status paths below are populated
+              # — artifact.revision=main@sha1:b93594b7…, and all 16 Kustomizations reporting that
+              # same revision as lastAppliedRevision.
+              #
+              # Cost: 2 series per Kustomization (16 today) + 1 for the GitRepository = 33.
+              - groupVersionKind:
+                  group: kustomize.toolkit.fluxcd.io
+                  version: v1
+                  kind: Kustomization
+                metricNamePrefix: kube_flux_kustomization
+                labelsFromPath:
+                  name: [metadata, name]
+                  namespace: [metadata, namespace]
+                metrics:
+                  - name: info
+                    help: "Revision a Flux Kustomization has applied (status.lastAppliedRevision)."
+                    each:
+                      type: Info
+                      info:
+                        # KSM OMITS a label whose path is absent (the same behaviour noted for the
+                        # Velero `phase` label above), so a Kustomization that has never applied
+                        # anything publishes this series with NO `revision` label at all rather than
+                        # an empty one. Every rule that reads it therefore matches revision!="" —
+                        # otherwise those would collapse together under the empty string and a
+                        # never-applied Kustomization would compare equal to a converged one.
+                        labelsFromPath:
+                          revision: [status, lastAppliedRevision]
+                  - name: suspended
+                    help: "1 if the Flux Kustomization is suspended (spec.suspend)."
+                    each:
+                      type: Gauge
+                      gauge:
+                        # nilIsZero is MANDATORY, exactly as for the Velero `paused` gauge above:
+                        # measured 2026-09-29, NONE of the 16 Kustomizations sets spec.suspend at
+                        # all. Without it this metric would not exist, and the `unless` that stops a
+                        # suspended Kustomization holding a rollout notification open forever would
+                        # silently match nothing.
+                        path: [spec, suspend]
+                        nilIsZero: true
+              - groupVersionKind:
+                  group: source.toolkit.fluxcd.io
+                  version: v1
+                  kind: GitRepository
+                metricNamePrefix: kube_flux_gitrepository
+                labelsFromPath:
+                  name: [metadata, name]
+                  namespace: [metadata, namespace]
+                metrics:
+                  - name: info
+                    help: "Revision of the artifact a Flux GitRepository has fetched (what main is)."
+                    each:
+                      type: Info
+                      info:
+                        labelsFromPath:
+                          revision: [status, artifact, revision]
       # MEMORY RAISED 2026-09-13 after 91 OOMKills. Measured, not guessed: working set 111 MiB
       # with a 6h peak of 123.3 MiB against the old 128Mi limit — KSM was dying on every spike,
       # roughly every 11 minutes (exit 137). That matters beyond the restarts: THIRTEEN alert

platform/monitoring/tests/flux-rollout.test.yaml

@@ -0,0 +1,268 @@
+# promtool rule-unit tests for FluxRolloutInProgress / FluxRolloutStuck — the deploy-notification
+# pair in the platform-health.flux group (platform/monitoring/alerts.yaml).
+# Run with: platform/monitoring/tests/run.sh   (the merge gate runs the same harness via
+#           conformance/tests/l0-static/alert-rules-unit-tested)
+#
+# WHY THESE TESTS CARRY MORE WEIGHT THAN USUAL. This pair is a NOTIFICATION, so both failure
+# directions are silent in production and neither is visible on a healthy cluster:
+#   * fire when it should not -> a permanently-firing "deploy in progress" that mails on every
+#     repeat_interval and trains the operator to ignore Flux mail. Shipped once already in this
+#     repo (CNPG, 2026-09-12).
+#   * fail to fire -> no deploy notification at all, which is the exact complaint this pair was
+#     written to answer. Indistinguishable from "no deploys happened".
+# So every clause is pinned in both directions here, and the two GUARDS in the expression (the
+# missing-exporter guard and the suspended-Kustomization exclusion) each get a test whose ONLY
+# job is to fail if that clause is deleted.
+#
+# MODELLING A REVISION CHANGE. `revision` is a LABEL, so a Kustomization moving from revision A to
+# revision B is modelled as series A ending and series B beginning — not as one series changing
+# value. Prometheus' 5m lookback then means series A is still resolvable for 5m after its last
+# sample, so every eval_time below is placed at least 5m clear of a handoff. That is deliberate:
+# it keeps these tests about the RULE and not about staleness semantics.
+#
+# Sample interval 1m; the longest `for` under test is 30m.
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  # =============================================================================================
+  # 1. CONVERGED — the steady state this cluster is in almost all of the time.
+  #    Both rules MUST be silent. This is the "not permanently firing" half of the proof, and it
+  #    is the test that fails if `unless on (revision)` is mis-spelled: the left-hand side is
+  #    non-empty here, and only the revision match suppresses it.
+  # =============================================================================================
+  - interval: 1m
+    name: converged (every Kustomization on the GitRepository revision) - silent
+    input_series:
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:aaaaaaa",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x40'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x40'
+    alert_rule_test:
+      - eval_time: 35m
+        alertname: FluxRolloutInProgress
+        exp_alerts: []
+      - eval_time: 35m
+        alertname: FluxRolloutStuck
+        exp_alerts: []
+
+  # =============================================================================================
+  # 2. A COMMIT LANDS — the notification this whole change exists to produce.
+  #    GitRepository moves to revision bbbbbbb at t=11m. apps-wave-0 catches up at t=14m;
+  #    `platform` never does.
+  #      t=25m -> FluxRolloutInProgress fires for `platform` ONLY (apps-wave-0 has converged),
+  #               and FluxRolloutStuck is still quiet because its 30m has not elapsed.
+  #      t=50m -> both fire. ("started and never finished" is a real, separate condition.)
+  # =============================================================================================
+  - interval: 1m
+    name: new revision on main - InProgress fires for the lagging Kustomization only
+    input_series:
+      # The old artifact revision ends at t=10m, the new one starts at t=11m.
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:aaaaaaa",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '1x10'
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '_x10 1x70'
+      # `platform` stays on the old revision for the whole test.
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x80'
+      # apps-wave-0 catches up at t=14m.
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x8'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:bbbbbbb",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '_x13 1x67'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x80'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x80'
+    alert_rule_test:
+      # The deploy notification, ~14m after the commit was fetched. Exactly ONE alert: a rule that
+      # ignored the revision match would also list apps-wave-0, which has already converged.
+      - eval_time: 25m
+        alertname: FluxRolloutInProgress
+        exp_alerts:
+          - exp_labels:
+              notification: deploy
+              job: kube-state-metrics
+              namespace: flux-system
+              name: platform
+              revision: "main@sha1:aaaaaaa"
+              customresource_group: kustomize.toolkit.fluxcd.io
+              customresource_kind: Kustomization
+              customresource_version: v1
+            exp_annotations:
+              summary: "Flux is applying a new revision — platform is behind main"
+              description: "Kustomization flux-system/platform has applied main@sha1:aaaaaaa, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
+      # `for: 30m` on the escalation is real — not yet firing 14m in.
+      - eval_time: 25m
+        alertname: FluxRolloutStuck
+        exp_alerts: []
+      # ...and it does fire once the rollout has genuinely hung.
+      - eval_time: 50m
+        alertname: FluxRolloutStuck
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              job: kube-state-metrics
+              namespace: flux-system
+              name: platform
+              revision: "main@sha1:aaaaaaa"
+              customresource_group: kustomize.toolkit.fluxcd.io
+              customresource_kind: Kustomization
+              customresource_version: v1
+            exp_annotations:
+              summary: "Flux rollout has not completed: platform stuck on an old revision"
+              description: "Kustomization flux-system/platform has been behind the flux-system GitRepository revision for 30m (applied main@sha1:aaaaaaa) — a git change started rolling out and has not converged. Check the Kustomization's Ready condition and its dependencies."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
+
+  # =============================================================================================
+  # 3. THE ROLLOUT COMPLETES -> the alert clears, which is what sends the "applied" mail.
+  #    Without this test the completion half of the feature is unproven: an expression that fired
+  #    correctly but never emptied itself would pass test 2 and still never deliver a resolved
+  #    notification.
+  # =============================================================================================
+  - interval: 1m
+    name: rollout completes - InProgress clears (this is the resolved/completion notification)
+    input_series:
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:aaaaaaa",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '1x10'
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '_x10 1x70'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x20'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:bbbbbbb",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '_x23 1x57'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x80'
+    alert_rule_test:
+      # Mid-rollout: firing.
+      - eval_time: 20m
+        alertname: FluxRolloutInProgress
+        exp_alerts:
+          - exp_labels:
+              notification: deploy
+              job: kube-state-metrics
+              namespace: flux-system
+              name: platform
+              revision: "main@sha1:aaaaaaa"
+              customresource_group: kustomize.toolkit.fluxcd.io
+              customresource_kind: Kustomization
+              customresource_version: v1
+            exp_annotations:
+              summary: "Flux is applying a new revision — platform is behind main"
+              description: "Kustomization flux-system/platform has applied main@sha1:aaaaaaa, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
+      # Converged (5m clear of the old series' last sample at t=20m): silent, and it never had
+      # 30m of continuous firing, so the escalation must not have fired either.
+      - eval_time: 32m
+        alertname: FluxRolloutInProgress
+        exp_alerts: []
+      - eval_time: 32m
+        alertname: FluxRolloutStuck
+        exp_alerts: []
+
+  # =============================================================================================
+  # 4. GUARD 1 — kube_flux_gitrepository_info ABSENT (KSM rolled without the ConfigMap, the flux
+  #    RBAC was lost, customResourceState reverted). `unless on (revision) <empty set>` removes
+  #    NOTHING, so without the `and on () (count(...) > 0)` guard EVERY Kustomization alerts
+  #    forever. This test exists to go red the moment that guard is deleted.
+  #    Correct behaviour is SILENCE: a broken exporter is FluxReconciliationFailure's job and the
+  #    platform-health.monitoring-self rules', not a deploy notification's.
+  # =============================================================================================
+  - interval: 1m
+    name: GUARD - GitRepository metric absent - silent, NOT 16 alerts forever
+    input_series:
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x60'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x60'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x60'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x60'
+    promql_expr_test:
+      # Stated explicitly so the reason for the silence is unambiguous in the test output: the
+      # guard's own count is empty, not merely false.
+      - expr: 'count(kube_flux_gitrepository_info{name="flux-system", revision!=""})'
+        eval_time: 45m
+        exp_samples: []
+    alert_rule_test:
+      - eval_time: 45m
+        alertname: FluxRolloutInProgress
+        exp_alerts: []
+      - eval_time: 45m
+        alertname: FluxRolloutStuck
+        exp_alerts: []
+
+  # =============================================================================================
+  # 5. GUARD 2 — a SUSPENDED Kustomization pins an old revision by design. It must not hold the
+  #    rollout notification open (or the "applied" mail never arrives), while a non-suspended
+  #    Kustomization on the same old revision MUST still alert.
+  #    Both halves are asserted in one test on purpose: an `unless` written with the wrong labels
+  #    would suppress EVERYTHING and would pass a suppression-only test.
+  # =============================================================================================
+  - interval: 1m
+    name: GUARD - suspended Kustomization excluded, non-suspended one still alerts
+    input_series:
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '1x40'
+      # Suspended and behind -> excluded.
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      # Not suspended and behind -> must alert.
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="conformance",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="conformance",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x40'
+    alert_rule_test:
+      - eval_time: 20m
+        alertname: FluxRolloutInProgress
+        exp_alerts:
+          - exp_labels:
+              notification: deploy
+              job: kube-state-metrics
+              namespace: flux-system
+              name: conformance
+              revision: "main@sha1:aaaaaaa"
+              customresource_group: kustomize.toolkit.fluxcd.io
+              customresource_kind: Kustomization
+              customresource_version: v1
+            exp_annotations:
+              summary: "Flux is applying a new revision — conformance is behind main"
+              description: "Kustomization flux-system/conformance has applied main@sha1:aaaaaaa, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
+
+  # =============================================================================================
+  # 6. A Kustomization that has NEVER applied anything publishes no `revision` label at all (KSM
+  #    omits absent paths). `revision!=""` must drop it rather than let it group under the empty
+  #    string and compare equal to the GitRepository revision.
+  # =============================================================================================
+  - interval: 1m
+    name: never-applied Kustomization (no revision label) is not counted as converged or lagging
+    input_series:
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '1x40'
+      # No `revision` label at all — this is what KSM publishes for status.lastAppliedRevision="".
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="brand-new",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="brand-new",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x40'
+    alert_rule_test:
+      # Silent: FluxReconciliationFailure (Ready=False for 15m) is what covers a Kustomization
+      # that cannot apply at all. This pair only ever speaks about revisions it can compare.
+      - eval_time: 30m
+        alertname: FluxRolloutInProgress
+        exp_alerts: []
+      - eval_time: 30m
+        alertname: FluxRolloutStuck
+        exp_alerts: []

platform/oauth2-proxy/README.md

@@ -88,3 +88,39 @@ This proves the R-IDP-02 property (unauthenticated cannot reach the upstream) on
 3. Log in as a **non-platform-admins** user -> expect **403** (authenticated but not authorized).
 4. Confirm the session cookie is `.home.keithjackson.dev`-scoped (single sign-on across the other
    protected UIs).
+
+## Observability: what this component can and cannot see (measured 2026-09-30)
+
+`servicemonitor.yaml` scrapes **this namespace's** deployment and nothing else. On prod that is
+two targets, `job="oauth2-proxy"`, both `up=1`. The complete set of oauth2-proxy-specific series
+in the TSDB is:
+
+    count by (__name__) ({__name__=~"oauth2_proxy.*"})
+      oauth2_proxy_requests_in_flight => 2
+
+**The per-app gate sidecars are not scraped, and there is nothing to scrape.** Six apps run an
+`oauth2-proxy` container of their own — crafty, jackett, pihole, qbittorrent, radarr, sonarr —
+each with `--http-address=0.0.0.0:4180` and **no `--metrics-address`**, so oauth2-proxy's metrics
+listener never starts. They declare only a `http:4180` container port, no `metrics` port exists on
+their Services, and `kubectl get servicemonitors,podmonitors -A` selects none of them. Any alert
+written against `oauth2_proxy_requests_total{namespace="jackett"}` would be structurally inert.
+
+This matters because the gate's characteristic failure is invisible by construction: an
+unauthenticated **XHR** gets a `302` to Keycloak rather than a `401`, the browser follows it
+against a live SSO session with no prompt, and the call resolves **200 carrying an HTML page**
+where the app expected JSON. Nothing in the request path reports an error. On 2026-09-30 jackett
+spent a burst of requests in exactly that state, re-established a session, and lost it 26 seconds
+into a 5-minute session, while `jackett-down` sat at 1 replica available the whole time.
+
+`auth-visibility-alerts.yaml` therefore alerts on **the gap**, not on the incident:
+`OAuth2ProxyGateUnobserved` compares `kube_pod_container_info{container="oauth2-proxy"}` against
+`up{container="oauth2-proxy"}` — two already-scraped families — and names every namespace running
+a gate that nothing observes. It fires for six namespaces today and clears per-namespace as each
+one gains a scrape. `runbooks/oauth2-proxy-auth-failures.md` is the by-hand substitute meanwhile,
+and it reads the sidecar logs out of Loki (Alloy collects them; 72h retention).
+
+**To close the gap for an app**, three things are needed together, on that app's own manifest:
+`--metrics-address=0.0.0.0:44180` on the sidecar, a `metrics` containerPort plus the matching
+Service port, and a ServiceMonitor selecting it. Only then does the auth-churn rule — sustained
+`code="302"` as a share of `oauth2_proxy_requests_total`, held across a window longer than the
+session lifetime — become writable rather than decorative.

platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml

@@ -0,0 +1,117 @@
+---
+# promtool unit tests for OAuth2ProxyGateUnobserved (platform/oauth2-proxy/auth-visibility-alerts.yaml).
+#
+# WHY. The rule is an `unless` between two aggregations over DIFFERENT metric families, and the
+# failure modes of that shape are both silent: a label that does not survive `count by` makes the
+# right-hand side match nothing (the alert fires for the scraped namespace too, i.e. permanently),
+# and a right-hand side that over-matches makes it fire for nobody (i.e. inert). Neither is
+# visible by reading the expression, and neither is visible on a live cluster where the answer
+# happens to look plausible. So all four states are pinned: fires for an unscraped namespace,
+# stays quiet for a scraped one, honours `for: 1h`, and STOPS once a scrape appears.
+#
+# HOW TO RUN. The one repo-wide harness, by the companion convention:
+#   ./infra/devcontainer/dev.sh run bash -c 'platform/monitoring/tests/run.sh platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml'
+# promtool is pinned in infra/devcontainer/versions.env; never run it off the host PATH.
+#
+# THE SERIES BELOW ARE REAL SHAPES read off live prod Prometheus on 2026-09-30 before being
+# copied here — kube_pod_container_info as kube-state-metrics emits it (job="kube-state-metrics",
+# its own instance/pod labels, image/image_id/container_id) and `up` as the oauth2-proxy
+# ServiceMonitor emits it (endpoint="metrics", port 44180). The point of using the real shapes is
+# that `count by (namespace)` has to discard all of them; a test with tidy two-label series would
+# pass while the real expression collapsed differently.
+#
+# ASSERTIONS USE `ALERTS` RATHER THAN exp_alerts: the description annotation is long prose and
+# comparing it verbatim would couple this file to every wording change while proving nothing the
+# ALERTS series does not already prove.
+rule_files:
+  - auth-visibility-alerts.rules.yaml
+evaluation_interval: 1m
+
+tests:
+  # ---------------------------------------------------------------------------------------------
+  # 1. THE STATE PROD IS ACTUALLY IN (2026-09-30): jackett runs a gate sidecar with no scrape;
+  #    the shared oauth2-proxy namespace runs two that ARE scraped. Asserts the alert fires for
+  #    jackett and NOT for oauth2-proxy, and that `for: 1h` is honoured (pending at 59m).
+  - interval: 1m
+    name: an unscraped gate namespace fires after 1h; the scraped one never does
+    input_series:
+      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
+        values: '1x130'
+      # The shared instance: kube-state-metrics sees both replicas...
+      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://aa11bb22",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000001"}'
+        values: '1x130'
+      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://cc33dd44",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000002"}'
+        values: '1x130'
+      # ...and both replicas are genuinely scraped, so `up` carries container="oauth2-proxy" there.
+      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.189:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="oauth2-proxy"}'
+        values: '1x130'
+      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.2.47:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="oauth2-proxy"}'
+        values: '1x130'
+    promql_expr_test:
+      # Not yet firing at 59m — `for: 1h` has not elapsed. Guards against someone "simplifying"
+      # the hold away and turning every rollout into a page.
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
+        eval_time: 59m
+        exp_samples: []
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett"}'
+        eval_time: 59m
+        exp_samples:
+          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett",severity="warning"}'
+            value: 1
+      # Firing for jackett past the hold...
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
+        eval_time: 65m
+        exp_samples:
+          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
+            value: 1
+      # ...and never for the namespace that IS scraped. This is the assertion that catches an
+      # `unless` whose right-hand side stopped matching: without it, a broken rule that fires for
+      # every gate namespace would pass the test above.
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="oauth2-proxy"}'
+        eval_time: 65m
+        exp_samples: []
+      # The count must be 1 per namespace-with-a-gate, not 1 overall: a `count by (namespace)`
+      # that lost its `by` would collapse to a single unlabelled series.
+      - expr: 'count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by (namespace) (up{container="oauth2-proxy"})'
+        eval_time: 65m
+        exp_samples:
+          - labels: '{namespace="jackett"}'
+            value: 1
+
+  # ---------------------------------------------------------------------------------------------
+  # 2. THE FIX LANDING. jackett's sidecar gains --metrics-address and a ServiceMonitor 70m in, so
+  #    `up` starts carrying container="oauth2-proxy" for namespace="jackett" under a DIFFERENT job
+  #    name than the shared proxy's. The alert must stop. This is what makes the rule a gap
+  #    tracker rather than a permanent nag, and it pins that the match is on `container`, not on
+  #    a hardcoded job="oauth2-proxy" (which would keep firing forever after a correct fix).
+  - interval: 1m
+    name: the alert clears once the sidecar is actually scraped
+    input_series:
+      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
+        values: '1x130'
+      # Absent for the first 70 minutes, then present. `_x` is "no sample"; the scrape did not
+      # exist yet.
+      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.213:44180",job="jackett-auth",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="jackett-auth-metrics"}'
+        values: '_x70 1x60'
+    promql_expr_test:
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett"}'
+        eval_time: 65m
+        exp_samples:
+          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
+            value: 1
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="jackett"}'
+        eval_time: 80m
+        exp_samples: []
+
+  # ---------------------------------------------------------------------------------------------
+  # 3. NO GATE, NO ALERT. A namespace with no oauth2-proxy container must never appear, or the
+  #    rule would page for every app on the platform the day someone widened the matcher.
+  - interval: 1m
+    name: namespaces without a gate sidecar are never reported
+    input_series:
+      - series: 'kube_pod_container_info{container="plex",container_id="containerd://feed0001",endpoint="http",image="ghcr.io/example/plex:1",image_id="ghcr.io/example/plex@sha256:feed0001",image_spec="ghcr.io/example/plex:1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="plex",pod="plex-0",service="kube-prometheus-stack-kube-state-metrics",uid="2222bbbb-0000-0000-0000-000000000001"}'
+        values: '1x130'
+    promql_expr_test:
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved"}'
+        eval_time: 65m
+        exp_samples: []

platform/oauth2-proxy/auth-visibility-alerts.yaml

@@ -0,0 +1,113 @@
+---
+# oauth2-proxy AUTH-VISIBILITY alerting (R-OBS-04).
+#
+# ============================================================================================
+# WHY THIS FILE EXISTS — THE 2026-09-30 JACKETT FAILURE THAT NOTHING OBSERVED
+# ============================================================================================
+# Jackett's UI stopped working. Reloading never fixed it and the user was never re-prompted to
+# log in. The sidecar's own access log had the whole story:
+#
+#   [oauthproxy.go:1026] No valid authentication in request. Initiating login.
+#   GET "/api/v2.0/indexers/yts/config?_=..."          HTTP/1.1 ... 302 400
+#   GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
+#   ... ten of these at 15:30:53, then a fresh [AuthSuccess] at 15:31:06,
+#   then "No valid authentication in request" again 26 SECONDS LATER, inside the same session.
+#
+# NOTHING ANYWHERE ELSE RECORDED IT. oauth2-proxy answers an unauthenticated XHR with a 302 to
+# Keycloak rather than a 401; the browser follows it silently, the live SSO session satisfies it
+# without a prompt, and the XHR resolves 200 carrying a 44 KB HTML dashboard where the app's
+# JavaScript wanted JSON. From the browser, from the user, and from any uptime check, the service
+# is healthy. `jackett-down` (apps/wave-0/jackett.yaml) watches
+# kube_deployment_status_replicas_available, which was 1 throughout and would have been 1 if the
+# gate had rejected every request ever made.
+#
+# ============================================================================================
+# WHY THIS FILE DOES NOT CONTAIN THE OBVIOUS ALERT — MEASURED, prod, 2026-09-30
+# ============================================================================================
+# The alert you would want is "a gated app is serving a sustained rate of auth redirects to a
+# user who is already authenticated". It is NOT WRITABLE against this cluster's TSDB, and the
+# reason is this file's real subject:
+#
+#   THE PER-APP GATE SIDECARS EXPORT NO METRICS AT ALL.
+#
+#   $ promtool query instant :9090 'count by (__name__) ({__name__=~"oauth2_proxy.*"})'
+#     oauth2_proxy_requests_in_flight => 2          <- the SHARED proxy's two replicas. That is ALL.
+#
+#   Every sidecar (crafty, jackett, pihole, qbittorrent, radarr, sonarr) runs with
+#   `--http-address=0.0.0.0:4180` and NO `--metrics-address`, so oauth2-proxy's metrics listener
+#   is never started; the container declares only port `http:4180`, there is no `metrics` port on
+#   any of their Services, and there is no ServiceMonitor or PodMonitor for any of them anywhere
+#   in the cluster (measured: `kubectl get servicemonitors,podmonitors -A`). Six gates, zero
+#   telemetry. An alert written against `oauth2_proxy_requests_total{namespace="jackett"}` would
+#   have been PERMANENTLY INERT — present, evaluating, silent, and indistinguishable from health.
+#   This repo has shipped that mistake before (VeleroNoRecentBackup, the Longhorn rules), which is
+#   why `conformance/tests/l0-static/alert-metrics-have-scrape-targets` exists, and why the
+#   temptation was refused here rather than ratcheted into a register.
+#
+#   The SHARED proxy IS scraped (job="oauth2-proxy", 2 targets, up=1) but it is not in the data
+#   path for any sidecar-gated app — each app gates itself — and its current pods export no
+#   `oauth2_proxy_requests_total` at all, having proxied nothing since they started. An alert on
+#   the shared instance would not have fired on this incident either. Saying so is the point.
+#
+# So this file alerts on the CONDITION THAT MADE THE INCIDENT INVISIBLE, not on the incident.
+# That condition is measurable today, from two already-scraped metric families, and it is the
+# precondition for ever being able to alert on the incident itself. It self-clears the moment a
+# sidecar gains a metrics endpoint and a scrape — at which point the auth-churn rule becomes
+# writable and belongs here beside it. See runbooks/oauth2-proxy-auth-failures.md, which is the
+# by-hand equivalent an operator uses in the meantime.
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: oauth2-proxy-auth-visibility
+  namespace: oauth2-proxy
+  labels:
+    release: kube-prometheus-stack     # kube-prometheus-stack's ruleSelector
+    app.kubernetes.io/name: oauth2-proxy
+spec:
+  groups:
+    - name: oauth2-proxy.auth-visibility
+      rules:
+        - alert: OAuth2ProxyGateUnobserved
+          # A namespace runs an oauth2-proxy gate container, and Prometheus holds no scrape
+          # target for that container in that namespace.
+          #
+          # BOTH SIDES ARE ALREADY-SCRAPED FAMILIES, deliberately:
+          #   * kube_pod_container_info  — kube-state-metrics, the declared owner in
+          #     alert-metrics-have-scrape-targets. It sees the container whether or not the
+          #     container exports anything, which is exactly the property needed here.
+          #   * up                       — UNIVERSAL. `up` carries the `container` label for any
+          #     ServiceMonitor/PodMonitor target, so this works for a future per-app scrape
+          #     without knowing what job name it will pick.
+          # Nothing in this expression touches `oauth2_proxy_*`, so the rule cannot itself be
+          # inert for the reason it is reporting.
+          #
+          # MEASURED ON PROD 2026-09-30 — this expression returns six namespaces right now:
+          #   {namespace="crafty"} {namespace="jackett"} {namespace="pihole"}
+          #   {namespace="qbittorrent"} {namespace="radarr"} {namespace="sonarr"}
+          # and excludes namespace="oauth2-proxy", whose two replicas ARE scraped. It fires on
+          # the state the cluster is in; it is not a rule waiting for a hypothetical.
+          expr: |
+            count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
+              unless
+            count by (namespace) (up{container="oauth2-proxy"})
+          # 1h, not 5m: a rollout briefly has a running sidecar whose target has not been
+          # discovered yet, and this condition is a standing architectural gap, never an
+          # emergency. Long enough that no deploy can produce it.
+          for: 1h
+          labels:
+            severity: warning
+          annotations:
+            summary: "oauth2-proxy gate in {{ $labels.namespace }} exports no metrics"
+            description: >-
+              The {{ $labels.namespace }} namespace runs an oauth2-proxy gate container and
+              Prometheus has NO scrape target for it, so every authentication decision that gate
+              makes — every redirect to Keycloak, every 403 on a group check, every session that
+              silently fails to hold — is invisible to metrics and to alerting. This is not a
+              theoretical gap: on 2026-09-30 the jackett gate answered a burst of the app's own
+              XHR calls with 302s to Keycloak, re-established a session, and lost it again 26
+              seconds later, and the only record anywhere was a line in the sidecar's stdout.
+              No alert could have fired, because there is nothing to alert on. Until a scrape
+              exists the runbook's LogQL queries are the substitute; the fix is
+              `--metrics-address=0.0.0.0:44180` on the sidecar plus a `metrics` port and a
+              ServiceMonitor, after which the auth-churn rule becomes writable.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/oauth2-proxy-auth-failures"

platform/oauth2-proxy/kustomization.yaml

@@ -14,3 +14,7 @@ resources:
   - oauth2-proxy-credentials.sops.yaml
   - httproute.yaml       # forward-auth sign-in/callback ingress (was referenced in-place by platform-config)
   - servicemonitor.yaml  # Prometheus scrape (was referenced in-place by platform-config)
+  # Alerting on whether the gates are OBSERVABLE at all. Travels with the servicemonitor above
+  # for the same reason it does: monitoring.coreos.com CRDs are registered by the `platform`
+  # tier, which is Ready before this tier's platform-config dependency completes.
+  - auth-visibility-alerts.yaml

runbooks/README.md

@@ -51,6 +51,18 @@ rule groups still evaluate against nothing and **cannot fire**:
 | Flux (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` |
 | Hubble (`hubble_drop_total`) | `HubbleDeniedFlowSpike` |
 | etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all |
+| per-app oauth2-proxy sidecars (`oauth2_proxy_*`) — measured 2026-09-30 | no authentication alerting exists at all for crafty, jackett, pihole, qbittorrent, radarr, sonarr |
+
+The oauth2-proxy row is the newest and the least obvious, because the scrape is not merely missing —
+there is **nothing to scrape**. `count by (__name__) ({__name__=~"oauth2_proxy.*"})` on prod returns
+exactly one family, `oauth2_proxy_requests_in_flight`, from the two replicas of the **shared**
+forward-auth instance in the `oauth2-proxy` namespace. The six per-app gate sidecars run without
+`--metrics-address`, so their metrics listener never starts; they declare no `metrics` container port;
+and no ServiceMonitor or PodMonitor selects them. Any rule written against `oauth2_proxy_*` for those
+namespaces would be inert on arrival. `OAuth2ProxyGateUnobserved`
+(`platform/oauth2-proxy/auth-visibility-alerts.yaml`) alerts on that gap itself, using
+`kube_pod_container_info` and `up` — two families that *are* scraped — and clears when a gate gains a
+real scrape. See `oauth2-proxy-auth-failures`.
 
 Each affected runbook says so on its own page and gives the by-hand equivalent. See `targets-down`.
 
@@ -108,6 +120,14 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
 - **`keycloak-db`** — Keycloak's CloudNativePG database is unavailable.
 - **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
   every green signal missed.
+- **`oauth2-proxy-auth-failures`** — a gated app "is up but does not work". Linked from
+  `OAuth2ProxyGateUnobserved`. The gate answers an unauthenticated **XHR** with a 302 to Keycloak
+  rather than a 401, the browser follows it silently against a live SSO session, and the call
+  resolves **200 carrying HTML** where the app's JavaScript wanted JSON — so the user is never
+  re-prompted and every uptime check stays green. Tells the gate's refusal from the group check
+  from the app's own login, from the sidecar log and from Loki, with the 2026-09-30 jackett trace
+  as the worked example. **Read the caveat**: the per-app gate sidecars export no metrics at all,
+  so these logs (72h of them) are the only record this class of failure leaves.
 
 ### Operator procedures (no alert links here)
 - **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only

runbooks/flux-rollout.md

@@ -0,0 +1,118 @@
+# flux-rollout — a git change is rolling out, or has stopped halfway
+
+**Alerts:** `FluxRolloutInProgress` (info, no `for:`) and `FluxRolloutStuck` (warning, 30m)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.flux`
+**Metrics:** `kube_flux_kustomization_info` / `kube_flux_kustomization_suspended` /
+`kube_flux_gitrepository_info` — kube-state-metrics custom-resource-state, configured in
+`platform/monitoring/kube-prometheus-stack.yaml`
+
+## What this means here
+
+These two are the **deploy notification**, not a fault report. On this cluster merging a PR into
+`main` IS the deploy, and until this pair existed nothing told you a deploy had begun: the Flux event
+stream emits `ReconciliationSucceeded` on every 10-minute tick whether or not anything changed (101
+such events in one hour, measured 2026-09-29), so "applying a commit" and "idle" looked identical.
+
+The pair reads the one signal that moves only when git moves — the **applied revision**:
+
+- `GitRepository/flux-system` `.status.artifact.revision` — what `main` is (polled every 1m)
+- `Kustomization/*` `.status.lastAppliedRevision` — what each tier has actually applied
+
+**`FluxRolloutInProgress` fires** the moment any non-suspended Kustomization's applied revision is
+not the revision fetched from `main`, and **clears when every one of them has caught up**. So:
+
+| Mail | Subject | Means |
+|---|---|---|
+| firing | `[FLUX] applying — FluxRolloutInProgress` | a commit started rolling out |
+| resolved | `[FLUX] applied — FluxRolloutInProgress` | every Kustomization reached it; rollout complete |
+
+Both are expected, routine, and not a problem. The `severity: info` label routes them to the
+dedicated `email-deploy` Alertmanager receiver so they never arrive subject-lined `[WARNING]`.
+
+**`FluxRolloutStuck` is the one that matters.** Same expression, `for: 30m`, `severity: warning`. It
+means a deploy started and did not converge — the "applying" mail arrived and the "applied" mail
+never did. The 14-tier chain is dependency-ordered and revision-gated, so a single stalled tier holds
+every tier below it on "dependency not ready".
+
+## First commands
+
+`kubectl` does not work on the workstation — the toolchain lives only in the pinned container, and
+the context must always be named explicitly.
+
+Which tiers are behind, and on what:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c '
+  kubectl --context k3s-prod -n flux-system get gitrepository flux-system \
+    -o jsonpath="WANTED: {.status.artifact.revision}{\"\n\"}"
+  kubectl --context k3s-prod get kustomizations -A \
+    -o custom-columns="NAME:.metadata.name,APPLIED:.status.lastAppliedRevision,READY:.status.conditions[?(@.type==\"Ready\")].status"'
+```
+
+Any row whose `APPLIED` differs from `WANTED` is a tier this alert is naming. Then ask **why** that
+tier is not moving — its Ready condition carries the reason:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c '
+  kubectl --context k3s-prod -n flux-system get kustomization <name> \
+    -o jsonpath="{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{\"\n\"}{end}"'
+```
+
+Confirm the alert is reading live data rather than a stale exporter:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c '
+  POD=$(kubectl --context k3s-prod -n monitoring get pod -l app.kubernetes.io/name=prometheus -o name | head -1)
+  kubectl --context k3s-prod -n monitoring exec "$POD" -c prometheus -- \
+    /bin/promtool query instant http://localhost:9090 "count by (__name__) ({__name__=~\"kube_flux.*\"})"'
+```
+
+Three names must come back (`kube_flux_kustomization_info`, `kube_flux_kustomization_suspended`,
+`kube_flux_gitrepository_info`). If that query is **empty**, the alerts are inert — go to the last
+cause below.
+
+## Likely causes, most likely first
+
+1. **Nothing is wrong — the rollout is simply in flight.** Most Kustomizations reconcile on a 10m
+   interval and the chain is sequential, so several minutes between the "applying" and "applied"
+   mails is normal. `cilium` is on a 1h interval and can legitimately be the last to converge.
+2. **A gating tier is unhealthy, so everything below it is parked.** Look for `DependencyNotReady`
+   in the stuck tier's message and walk *up* the chain to the first tier that is not Ready; that one
+   is the real fault. Its own alert (`FluxReconciliationFailure`) and runbook `flux-reconcile` own it.
+3. **A Kustomization is suspended.** A suspended tier is excluded from this alert by design (so it
+   cannot hold the "applied" mail open forever), but anything that `dependsOn` it still stalls and
+   *will* show here. `FluxResourceSuspended` / runbook `flux-suspended` covers it.
+4. **A health check is timing out.** A `wait: true` tier only records `lastAppliedRevision` once its
+   objects pass health checks, so a workload that never goes Ready keeps the tier on the old
+   revision even though the manifests applied fine.
+5. **kube-state-metrics is not publishing the `kube_flux_*` family.** Then the rules are inert and
+   you would get *no* mail at all rather than a stuck one — so this is the cause when the complaint
+   is silence. The metrics come from `customResourceState` plus a read-only RBAC rule for
+   `kustomize.toolkit.fluxcd.io` / `source.toolkit.fluxcd.io`; without that RBAC, KSM emits the set
+   **empty** and logs a forbidden error. Check its logs:
+   ```bash
+   ./infra/devcontainer/dev.sh run bash -c '
+     kubectl --context k3s-prod -n monitoring logs deploy/kube-prometheus-stack-kube-state-metrics \
+       --tail=50 | grep -i "forbidden\|custom.resource" || echo "no RBAC complaints"'
+   ```
+   Note the chart puts **no checksum annotation** over that ConfigMap, so a change to the config
+   alone does not restart the pod — that is why `podAnnotations` carries a
+   `homelab.keithjackson.dev/crs-config-rollout` counter that must be bumped in the same commit.
+
+## When to escalate or stop
+
+- **`FluxRolloutInProgress` alone, cleared within ~15 minutes: do nothing.** That is the feature
+  working. Do not suspend, reconcile or apply anything in response to it.
+- **`FluxRolloutStuck` firing: fix the upstream fault, not this symptom.** This alert names the
+  tier that is behind; it never names the cause. Walk up to the first non-Ready tier and use
+  `flux-reconcile`.
+- **Never `kubectl apply`, `edit` or `patch` the change through by hand.** Flux reconciles `main`;
+  a hand-applied object is drift that the next reconcile reverts or, worse, adopts — and it makes
+  the applied-revision signal these alerts read a lie. `flux reconcile` (an operator action, never
+  an agent's) is the only nudge that is safe, and it only *hurries* the same pull.
+- **Escalate to a revert if a stuck rollout is degrading service.** Reverting the offending commit
+  on `main` through the normal PR flow is the rollback path; see runbook `update-rollback`.
+- **If both alerts are silent during a deploy you know happened,** the notification itself is broken,
+  not the deploy. That is cause 5 above, and it is worth fixing immediately — silence from these
+  rules is indistinguishable from "no deploys happened", which is the exact defect they were
+  written to close.

runbooks/kustomization.yaml

@@ -65,6 +65,7 @@ configMapGenerator:
       - crafty-admin-credential=crafty-admin-credential.md
       - denied-flows=denied-flows.md
       - flux-reconcile=flux-reconcile.md
+      - flux-rollout=flux-rollout.md
       - flux-suspended=flux-suspended.md
       - image-automation=image-automation.md
       - image-policy-stalled=image-policy-stalled.md
@@ -84,6 +85,10 @@ configMapGenerator:
       - node-io=node-io.md
       - node-memory=node-memory.md
       - node-memory-swap=node-memory-swap.md
+      # Linked from OAuth2ProxyGateUnobserved, and the diagnostic procedure for any gated app
+      # that "is up but does not work" — the gate answers an unauthenticated XHR with a 302,
+      # not a 401, so the failure arrives as an HTTP 200 carrying the wrong body.
+      - oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md
       # Operator procedure, not an alert page (no alert links to it — the
       # alert-runbooks-resolvable check treats that as advisory). It is served
       # because the plex.tv claim window is short and is not a thing to go looking

runbooks/oauth2-proxy-auth-failures.md

@@ -0,0 +1,171 @@
+# oauth2-proxy auth failures — telling the three layers apart
+
+Linked from `OAuth2ProxyGateUnobserved`, and the page to open whenever a gated app "is up but
+does not work": a UI that loads and then shows nothing, a page that never finishes, a list that
+stays empty, a save that silently does not save. The alert is about the *visibility* gap; this
+page is the diagnostic procedure, and it works whether or not the alert ever fired.
+
+## What this means here
+
+Every app behind Keycloak on this platform has **three** places a request can be refused, and
+from a browser they look identical — because two of the three answer with **HTTP 200**.
+
+| Layer | What refuses | What the browser sees |
+|---|---|---|
+| 1. The gate (oauth2-proxy sidecar) | no valid session cookie | **302 to Keycloak**, then usually **200** |
+| 2. The gate's group check | session valid, wrong group | 403, or a redirect loop |
+| 3. The app itself | the app's own login / API key | the app's own 401 or login page |
+
+**Layer 1 is the one that hides.** oauth2-proxy answers an unauthenticated request with a
+redirect, not a 401 — correct for a browser navigation, disastrous for an **XHR**. The app's
+JavaScript asks for JSON, the gate answers `302`, the browser follows it silently, the live
+Keycloak SSO session satisfies it with no prompt, and the XHR resolves **200 with an HTML
+dashboard in the body**. `fetch()` sees a success. The user sees an empty page. Reloading does
+nothing, and nobody is ever asked to log in, because from the browser's point of view nothing
+failed.
+
+The worked example, jackett on 2026-09-30, straight from the sidecar log:
+
+    [oauthproxy.go:1026] No valid authentication in request. Initiating login.
+    GET "/api/v2.0/indexers/yts/config?_=..."          HTTP/1.1 ... 302 400
+    GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
+    ... ten of these at 15:30:53
+    [AuthSuccess] Authenticated via OAuth2: Session{email:... created:2026-09-30 15:31:06
+                  expires:2026-09-30 15:36:06 refresh_token:true groups:[platform-admins]}
+    GET "/UI/Dashboard"        200 44379
+    GET "/api/v2.0/indexers?_=..." 200 1501229
+    ... and 26 seconds later, inside that same 5-minute session:
+    [oauthproxy.go:1026] No valid authentication in request. Initiating login.
+
+A session created at 15:31:06, valid until 15:36:06, not honoured at 15:31:32. That is the
+signature of the session being **re-established over and over instead of held** — a cookie that
+another proxy is overwriting, or a cookie that never comes back at all — not of Keycloak being
+down and not of the app being broken.
+
+**Read this before you trust any silence:** the per-app gate sidecars export **no metrics**. They
+run without `--metrics-address`, so oauth2-proxy's metrics listener never starts; they declare no
+`metrics` port; and no ServiceMonitor or PodMonitor selects them. Only the shared
+`oauth2-proxy` namespace instance is scraped, and it is not in the data path for any of these
+apps. So there is nothing to graph and nothing that could have alerted. The logs below are not a
+convenience — until that changes they are the **only** record that this class of failure happened.
+
+## First commands
+
+`kubectl` does not exist on the workstation; everything runs in the pinned container, and always
+with an explicit context.
+
+**1. Which namespaces have an unobserved gate** (this is what the alert measures):
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
+     -c prometheus -- /bin/promtool query instant http://localhost:9090 \
+     "count by (namespace) (kube_pod_container_info{container=\"oauth2-proxy\"}) unless count by (namespace) (up{container=\"oauth2-proxy\"})"'
+```
+
+**2. The sidecar log — but you must grep it.** The gate's stdout is ~11,500 lines a day per app
+and almost all of it is the kubelet health probe (`GET "/ping" ... 200`). A plain
+`kubectl logs --tail=100` shows you nothing but `/ping` and looks perfectly healthy. Filter:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n jackett logs deploy/jackett -c oauth2-proxy --since=2h \
+     | grep -v "/ping" | grep -E "No valid authentication|AuthSuccess|403|401|callback"'
+```
+
+Substitute the namespace: `crafty`, `jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`.
+
+**3. The same question across every gate at once, from Loki.** Alloy collects all of these, so
+you can ask about the whole fleet and about a window that has already scrolled out of
+`kubectl logs`. Port-forward Loki and query it:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring port-forward svc/loki 13100:3100 & sleep 5; \
+   curl -s -G --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"No valid authentication in request\" [24h]))" \
+     http://127.0.0.1:13100/loki/api/v1/query | python3 -m json.tool | grep -E "namespace|value" '
+```
+
+and the denominator, which is what makes the number mean something:
+
+```
+   ... --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"AuthSuccess\" [24h]))" ...
+```
+
+**How to read the ratio.** A healthy sign-in is roughly **one or two** `No valid authentication`
+lines per `AuthSuccess` — the first request of a session is unauthenticated by definition. A
+ratio in the tens means sessions are not surviving. Measured on prod 2026-09-30 (24h):
+
+| namespace | `No valid authentication` | `AuthSuccess` | ratio |
+|---|---|---|---|
+| crafty | 81 | 1 | 81:1 |
+| radarr | 67 | 1 | 67:1 |
+| jackett | 39 | 4 | ~10:1 |
+| pihole, qbittorrent, sonarr | 0 | 0 | (unused in the window) |
+
+All three of those are the failure, not the baseline.
+
+**4. Did the gate admit the user, or did the app refuse them?** Look for the `[AuthSuccess]` line
+and what follows it. If `[AuthSuccess]` appears and the very next thing is the *app's own* login
+page or the app's 401, the gate did its job and layer 3 is your problem — that is the Crafty
+shape, and `runbooks/crafty-admin-credential.md` covers it.
+
+**5. Cookie names, when sessions do not survive.** Two oauth2-proxy instances emitting the
+**same** cookie name on the **same** domain hold different cookie-secrets, so each overwrites the
+other's cookie and the loser logs exactly the `No valid authentication` line above. List what
+each gate is configured to emit (arguments only — never print a Secret's data):
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'for ns in crafty jackett pihole qbittorrent radarr sonarr; do printf "%-12s " "$ns"; \
+     kubectl --context k3s-prod -n $ns get deploy $ns \
+       -o jsonpath="{range .spec.template.spec.containers[?(@.name==\"oauth2-proxy\")]}{.args}{end}" \
+     | tr "," "\n" | grep -E "cookie-name|cookie-domain" | tr "\n" " "; echo; done'
+```
+
+A gate with no `--cookie-name` is using oauth2-proxy's default `_oauth2_proxy`. If two of them
+share that default **and** a zone-wide `--cookie-domain=.home.keithjackson.dev`, they are
+evicting each other, and the symptom is intermittent and depends on which app you visited last.
+
+## Likely causes, most likely first
+
+1. **Cookie collision between two gates on one domain.** Default cookie names plus a zone-wide
+   cookie domain. Signature: high `No valid authentication`-to-`AuthSuccess` ratio, sessions
+   dying seconds after being created, symptom depends on browsing order, and *nothing* wrong at
+   Keycloak. Fix is a unique `--cookie-name` and a host-scoped `--cookie-domain` per gate — see
+   `decisions/2026-09-29-crafty-oauth2-proxy-signin.md`, which found and fixed exactly this.
+2. **The cookie is too large to come back.** With `--pass-access-token` the session carries the
+   access token; a fat Keycloak token pushes the cookie past 4 KB and it gets split into
+   `_oauth2_proxy_0`, `_oauth2_proxy_1`… If a chunk is dropped the session is unreadable and the
+   gate logs the same line. Look for numbered cookie names in the log, and for a session that
+   never survives a single navigation.
+3. **The XHR path is not the navigation path.** The app's API is under a prefix the gate handles
+   differently, or the app issues cross-origin XHRs the cookie does not accompany. Signature:
+   `/UI/...` returns 200 and `/api/...` returns 302 in the *same second*, which is precisely the
+   jackett trace above.
+4. **Session lifetime shorter than the app's polling interval,** with no refresh. The jackett
+   session above was a 5-minute session. If `refresh_token:true` is present and sessions still
+   die early, the refresh is failing, not missing.
+5. **Group check, not authentication.** `--allowed-group` is a floor: a valid Keycloak session in
+   the wrong group is refused *after* a successful login, which reads as a loop. Look for 403 and
+   for the `groups:[...]` list on the `[AuthSuccess]` line.
+6. **The app, not the gate.** `[AuthSuccess]` followed by the app's own login page or 401.
+
+## When to escalate or stop
+
+- **Stop and hand over if the fix is a manifest change.** Everything above is read-only
+  diagnosis. Changing a `--cookie-name`, adding `--metrics-address`, or editing an alert goes
+  through a branch and a PR; `kubectl edit` on a gate is drift that the next Flux reconcile
+  reverts, and in the meantime you have an unreproducible cluster.
+- **Stop if you find yourself about to print a Secret.** The cookie-secret and client-secret live
+  in `oauth2-proxy-credentials` and the per-app equivalents. Compare *key names* and whether two
+  Secrets differ; never dump `.data` or `.stringData`. A password has been leaked into a
+  transcript here exactly this way.
+- **Escalate to the platform owner if Keycloak itself is the suspect** — check `KeycloakDown` and
+  `KeycloakDatabaseDown` first (`runbooks/keycloak-down.md`, `runbooks/keycloak-db.md`). In the
+  2026-09-30 incident Keycloak was healthy throughout and blaming it would have cost days.
+- **The honest limit of this page.** With no metrics from the sidecars there is no history beyond
+  Loki's **72h** retention (`platform/monitoring/loki.yaml`, `retention_period: 72h`). If the
+  report is older than three days, the evidence is gone and the only remaining move is to
+  reproduce it. That is the cost of the gap `OAuth2ProxyGateUnobserved` is tracking, and it is
+  the reason to close it rather than silence it.