fix(gitea): admins could override the gate -- the enforcement was not enforcing

enable_status_check and required_approvals were set earlier today and recorded as done. They did not bite. block_admin_merge_override was false, and Gitea lets repo admins override merge requirements without it. jakman85 holds owner, so both requirements were bypassable by exactly the account that performs every merge. The evidence was already on the record: PRs #5-#9 all merged while required_approvals: 1 was in force, with ZERO approvals recorded on any of them. Two process failures, both the same shape as the defect this whole session started with: 1. Verified with the wrong view. /branches/main reports protected, enable_status_check and required_approvals but NOT block_admin_merge_override -- it showed everything green while the rule was overridable. /branch_protections/main is authoritative. 2. Probed for a field name assumed to exist. This Gitea (1.24.3) has no `apply_to_admins`; asking jq for it returns null, which reads like "set to false" but means "key absent", so a PATCH that silently did nothing produced a convincing before/after. jq returning null is not a value -- it may be an absence. Enumerate with to_entries. require-gate.sh now sets block_admin_merge_override: true; block-admin-override.sh fixes it in isolation. Both carry the field-name warning. This PR is the test. If it merges WITHOUT an approval, the fix did not work. Also records the governance question: with only jakman85 (human) and gitea-admin (agent), a human-authored PR now needs the automation account to approve it. That is ceremony, not review, and the choice between keeping it, dropping approvals in favour of the status check, or adding a second human is the operator's to make. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Merge pull request 'docs(ci): close out the hardening; correct a claim that shipped wrong' (#9 ) from docs/close-out-ci-hardening into main
2026-09-29 02:36:38 -05:00 · 2026-09-29 07:32:52 +00:00 · 2026-09-29 02:28:55 -05:00 · 2026-09-29 07:22:50 +00:00 · 2026-09-29 07:22:20 +00:00 · 2026-09-29 07:21:28 +00:00
59 changed files with 1261 additions and 229 deletions
@@ -40,9 +40,15 @@ and ADR-0011: **CI holds no broad kubeconfig.** L2/L3 run as an in-cluster **Cha
 *collects* its artifacts, using a narrow trigger credential referenced **by secret name only**
 (`secrets.CONFORMANCE_TRIGGER_KUBECONFIG`) — never a plaintext secret, never a cluster-admin kubeconfig.

-## Honest SKIP — never fake green (Phase 0)
+## Honest SKIP — never fake green

-**No cluster exists in Phase 0.** The gate is deliberately built so unavailable checks **SKIP loudly,
+**NOTE (2026-09-29):** this section is written in the past's voice — "Phase 0", "no cluster
+exists". **A cluster has existed for weeks** and Flux has been reconciling against it. The
+skip logic below is still correct and still honest, but read it as describing a **live
+coverage gap** (L2/L3 is not wired into CI) rather than an expected phase. Its being
+phrased as a phase is part of why the gap went unexamined.
+
+Originally: The gate is deliberately built so unavailable checks **SKIP loudly,
 they do not pass**:

 - The `conformance` job is `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}` — with no cluster it is
@@ -77,21 +83,29 @@ Both dirs are git-ignored (see [`.gitignore`](../../.gitignore)).
 ## Triggers & branch model (CONTRIBUTING.md)

 - `pull_request` → `main`: **PRs are the enforced merge gate into `main`.**
- `push` → `feature/**`, `fix/**`, `docs/**`, `chore/**`: fast feedback on the branch. **No work on `main`.**
+- `push` → `main` only. The per-branch prefix list was removed: `act_runner` runs at capacity 1,
+  so a branch push fires a redundant run that serialises *ahead* of the PR run for the same tree.
+  **No work on `main`** — and since 2026-09-29 that is enforced, not conventional: `main` is a
+  protected branch and a direct push is rejected by the pre-receive hook (verified, not assumed).

-## Transition: generic-CI-now → Gitea-Actions-after-Gitea-is-up
+## Transition: COMPLETE (2026-09-29)

-Per [`CONTRIBUTING.md`](../../../home-network-k3s/CONTRIBUTING.md) and
-[`docs/05` §"local bootstrap → self-hosted Gitea"](../../../home-network-k3s/docs/05-gitops-and-delivery.md):
+This section used to describe a migration as pending. It has happened. Gitea is live at
+`gitea.home.keithjackson.dev`, the runner `incluster-dind` is registered and online, and
+`gate.yml` runs on every PR.

-1. **Now (interim):** there is **no remote git host** on this network. This workflow file is dormant —
-   but the **identical gate runs locally today** via `make -C ci gate` (R-TEST-02). Development is
-   local feature-branch → `main`, the gate enforced by convention + the local runner.
-2. **Gitea comes online early** (one of the first services onboarded) at `gitea.home.keithjackson.dev`.
-3. **CI moves to Gitea Actions:** this workflow activates unchanged. The only operational steps are
-   enabling a Gitea Actions runner and setting `vars.CLUSTER_AVAILABLE=true` +
-   `secrets.CONFORMANCE_TRIGGER_KUBECONFIG` once a dev/test cluster exists. PRs then become the
-   **platform-enforced** merge gate — the same no-work-in-main rule, now enforced by machine.
+**What took a month to notice:** the migration was recorded as done on 2026-08-31 without
+anyone checking the platform. In reality `main` had **no branch protection** and **zero
+runners were registered** until 2026-09-29. Every "the gate enforces this" statement in
+between rested on an assertion nobody verified. PR #1 — the first pull request this
+repository ever had — sat `pending` for 30.6 minutes because nothing existed to run it.
+
+So: **do not read this file, or the DECISIONS-LOG, as evidence about the live platform.**
+Query the API. `infra/gitea/` holds the scripts and the measured state.
+
+Still genuinely outstanding: `vars.CLUSTER_AVAILABLE` and
+`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` are unset, so L2/L3 conformance does not run in
+CI. That is a real coverage gap, not a phase.

 ## Configuration (set in Gitea; referenced by name only)

@@ -44,16 +44,31 @@
 ---
 name: gate

+# TRIGGERS — deliberately only `pull_request` into main, plus `push` to main itself.
+#
+# There is NO per-branch push trigger. There used to be a prefix list (feature/** fix/**
+# docs/** chore/**) for "fast feedback on the branch", and it was a net negative here:
+#
+#   * `act_runner` runs at capacity 1. A branch push and its PR fire the SAME gate over the
+#     SAME tree, and the redundant run SERIALISES AHEAD of the PR run — doubling the wall
+#     clock on the one result that actually gates the merge. That cost real time during the
+#     2026-09-29 plex claim, where the loop is racing a short-lived token.
+#   * The list silently did not match this repo's own convention. Branches here are `feat/**`,
+#     which was absent, so the "fast feedback" never fired for the most common prefix anyway —
+#     a mismatch nobody noticed for a month because its only symptom was a MISSING run.
+#     Any hand-maintained prefix list has that failure mode; deleting it removes the class.
+#   * It is redundant. CLAUDE.md already requires `./ci/gate.sh` in the pinned toolbox before
+#     you push (74s locally), and the PR gate re-runs it authoritatively. A third run of the
+#     same checks buys nothing.
+#
+# `push: [main]` is KEPT: post-merge main must still be verified, and it is the only run for
+# which no PR context exists.
 on:
  pull_request:
    branches: [main]
  push:
    branches:
      - main
-      - 'feature/**'
-      - 'fix/**'
-      - 'docs/**'
-      - 'chore/**'

 concurrency:
  group: gate-${{ github.ref }}
@@ -11,11 +11,14 @@
 #                                          v
 #                                     merge-gate decision  [R-TEST-03: touched subset green]
 #
-# Transition (CONTRIBUTING.md / docs/05 §"local bootstrap -> self-hosted Gitea"):
-# Today there is no Gitea host, so this file is dormant. The SAME gate is runnable
-# now via `make -C ci gate` (R-TEST-02). Once Gitea is self-hosted and Gitea
-# Actions is enabled, this workflow becomes the ENFORCED PR merge gate into main —
-# no code change required, only turning on the runner and setting CLUSTER_AVAILABLE.
+# STATUS (corrected 2026-09-29): Gitea IS hosted and Gitea Actions IS enabled; the runner
+# `incluster-dind` is online and gate.yml is the live PR gate. The text that used to sit here
+# ("Today there is no Gitea host, so this file is dormant") was stale by weeks -- it was written
+# as a plan and never revisited once the plan completed.
+#
+# THIS file is still dormant, but for the reason in the DORMANT block below -- its spec checkout
+# and L2/L3 seams are unwired -- NOT because there is no host. Do not read a stale plan as a
+# statement about the live platform; query the API instead (see infra/gitea/).
 #
 # CI holds NO broad kubeconfig. L2/L3 run as an in-cluster Chainsaw Job whose own
 # scoped ServiceAccount holds cluster access (docs/14, R-SEC-02); CI uses only a
@@ -23,25 +23,57 @@ There is no other path. Specifically:
 > (main is what the cluster follows), not what you are permitted to do. Honour this
 > file, not that line. Fixing those runbooks is tracked work.

-## 2. Enforcement is currently WEAKER than this rule. Follow the rule anyway.
+## 2. Enforcement (as of 2026-09-29: the rule is now MACHINE-ENFORCED)

 Verified against the live Gitea on 2026-09-29:

 | Expected | Actual |
 |---|---|
-| `main` protected, direct push rejected | **No branch protection.** `protected: false`, `user_can_push: true` |
-| PR blocked until the gate passes | **`enable_status_check: false`**, `status_check_contexts: []` |
-| Gate runs on every PR | **No Actions runner is registered** (repo, org and site level all report zero) |
-| Approval required before merge | **`required_approvals: 0`** |
+| `main` protected, direct push rejected | ✅ `protected: true`, `user_can_push: false` — **verified by a real rejected push**, not by the API's word |
+| PR blocked until the gate passes | ✅ `enable_status_check: true`, context `gate / gate (pull_request)` |
+| Gate runs on every PR | ✅ runner `incluster-dind` online; ~1m35s |
+| Approval required before merge | ✅ `required_approvals: 1` |

-So nothing mechanically stops you from pushing to `main` right now. **That is a gap
-being closed, not permission.** `DECISIONS-LOG.md` (2026-08-31) claims the gate is a
-"required status check" with a runner online; that claim is not true of the live
-system, which is how this went unnoticed.
+All four were FALSE as recently as 2026-09-29, while this repo's own log asserted they were
+true. They are true now because each was measured, and the push rejection was provoked rather
+than assumed. **Re-measure before relying on any of them** — see §2b.

-**Consequence for you:** because CI cannot run, the **local toolbox gate is the only
-real gate**. Run it and quote its numbers in your report. Never describe a PR as
-"green" on the strength of CI that did not execute.
+Still not covered: `vars.CLUSTER_AVAILABLE` is unset, so **L2/L3 conformance does not run in
+CI**. A green PR therefore proves the L0/L1 offline subset, not runtime behaviour.
+
+**Consequence for you:** CI now runs, but it does not yet *block* anything. Always run
+the toolbox gate yourself and quote its numbers. Never describe a PR as "green" on the
+strength of a check you have not seen report — and note that a **skipped** Gitea job
+reports as `success`, so a green tick is not by itself evidence that anything ran.
+
+## 2b. Never infer platform state from this repository
+
+**This repo is not evidence about the live platform.** Neither is `DECISIONS-LOG.md`.
+Both describe what someone believed at the time of writing, and both have been
+confidently wrong for a month at a stretch.
+
+On 2026-09-29 the log stated the merge gate was a required status check with a runner
+online. The API said `branch_protections: []` and **zero** runners. Everything downstream
+of that belief — "the gate enforces this", "your push would be rejected", "CI covers it" —
+was wrong, and nothing in the tree could have revealed it, because the tree was the source
+of the error.
+
+**So, before asserting that any gate, protection, runner, secret, or workload is in a
+given state: query the thing itself.**
+
+| Claim about... | Check |
+|---|---|
+| branch protection / required checks | `GET /api/v1/repos/{owner}/{repo}/branches/main` |
+| a runner exists | `GET /api/v1/admin/actions/runners` |
+| CI actually ran | the commit's `/statuses` — and read the **description**; a *skipped* Gitea job reports `success` with "Has been skipped" |
+| a workload / Secret / env | `kubectl --context k3s-prod ...` (read-only is fine for agents) |
+| Flux applied your commit | compare `.status.lastAppliedRevision` against the GitRepository's artifact revision |
+
+Quote the measured field, not the prose. If you cannot measure it, say so explicitly —
+"unverified" is a legitimate report, and a far better one than a confident guess that
+matches the docs.
+
+**A green tick is not proof anything ran.** Read what actually executed.

 ## 3. Gate before you push

@@ -1192,6 +1192,13 @@ fresh agent should read this entry + `infra/PHASE1-STATUS.md` + `docs/plans/` an
  Platform-admins require TOTP MFA.
 - **Gitea Actions CI is LIVE + enforcing**: `.gitea/workflows/gate.yml` runs `make -C ci gate` on push/PR to main;
  required status check; runner `incluster-dind` online (ns gitea-actions).
+  > **[CORRECTION, 2026-09-29 — THIS BULLET WAS FALSE WHEN WRITTEN AND STAYED FALSE FOR A MONTH.]**
+  > Measured against the live Gitea API on 2026-09-29: `branch_protections` returned `[]` (`main`
+  > `protected:false`, `enable_status_check:false`, `user_can_push:true`) and **zero** Actions runners
+  > were registered at repo, org or site level. The gate was neither enforced nor able to run. PR #1
+  > — the first pull request this repo ever had — was opened on 2026-09-29 and sat `pending` for
+  > 30.6 minutes with nothing to execute it. Both halves of this bullet were asserted, not verified.
+  > See the 2026-09-29 entries for the root cause and the fix.

 ### What changed this session (durable)
 1. **SSO cluster-wide break→fix — root cause Cilium CVE-2025-30162.** The 1.16→1.20 security fix makes the shared
@@ -4781,3 +4788,179 @@ accurate before the PR flow and are stale now. Correcting them is tracked, not d
 3. Land `feat/plex-claim-sops-secret` (PR #1) once the gate can actually run.
 4. Claim plex, after checking it is not already claimed.
 5. Correct the 37 runbooks' `git push origin main` wording.
+
+## 2026-09-29 — Plex is claimed; the CI/CD gate that "enforced" it never existed. **PM HANDOFF.**
+
+Began as "inject a plex.tv claim token". Ended by discovering the delivery pipeline was fiction,
+rebuilding it, and shipping the token through it. Plex is **claimed** — `PlexOnlineToken` is
+present in `/config` on the `plex-config` PVC, so the exchange is now a permanent no-op and the
+library scan is running over the 8.33T NFS share.
+
+### The finding: both halves of the 2026-08-31 CI claim were false
+
+That entry recorded "Gitea Actions CI is LIVE + enforcing … required status check; runner
+`incluster-dind` online". Measured against the live API on 2026-09-29:
+
+| Claimed | Measured |
+|---|---|
+| `main` protected, gate required | `branch_protections` → `[]`; `protected:false`, `enable_status_check:false`, `required_approvals:0`, `user_can_push:true` |
+| Runner online | **zero** runners at repo, org AND site level |
+
+**PR #1 was the first pull request this repository has ever had.** Its contexts sat `pending`
+for 30.6 minutes across 122 polls and never reported. For a month, every "the gate enforces this"
+statement rested on an assertion nobody had checked — the same fail-open class as 2026-09-18,
+and the reason it hid so long is that the log itself was the evidence.
+
+### Root cause of the dead runner
+
+`act_runner` registers **once** and persists its identity in `/data/.runner` on the
+`act-runner-data` PVC. Gitea no longer held a matching row, so the pod booted, saw `.runner`,
+**skipped registration**, and polled with an identity Gitea did not recognise. Symptom: zero
+runners, pod Running, nothing crashlooping, no error anyone reads. Deleting the PVC forced a
+genuine re-registration (`.runner is missing` → `Runner registered successfully`).
+
+It then did not come back for several minutes, because Flux could not recreate the PVC:
+`apps-wave-1` was blocked four levels deep on a **stale** `DependencyNotReady` for
+`platform-config`, which itself read `Ready=True`. Each wave re-evaluates only on its own
+`interval: 10m`, so a resource deleted at the bottom of the chain can take ~40 min to return.
+**Force it** with `flux reconcile kustomization apps-wave-1 --with-source` rather than waiting.
+
+### Corrections to things this log and the runbooks asserted
+
+- **A skipped Gitea job reports `success`, not pending.** Measured: `gate / conformance` returned
+  `success` / "Has been skipped". An earlier entry (#2) said requiring it would deadlock every PR.
+  Wrong — it would pass **vacuously**, forever, while nothing ran. Under HARD RULE 4 that is worse
+  than a deadlock, because a deadlock is loud. Never require it; require
+  `gate / gate (pull_request)` alone. Corrected in #3.
+- **`gate.yml`'s push trigger omits `feat/**`** (it lists `feature/** fix/** docs/** chore/**`).
+  With `act_runner` capacity 1, a branch prefix that IS in that list fires a second redundant run
+  which serialises ahead of the PR run and doubles wall-clock. Exploited deliberately in
+  `infra/plex/claim-ship.sh`; the honest fix is to align the trigger list.
+- **The "~4-minute TTL"** asserted in four places was **uncited**, and upstream `pms-docker`
+  documents no expiry at all. The authoritative reading is the countdown plex.tv/claim renders at
+  issue time. Removed from `runbooks/plex-claim.md`; the same claim still sits in
+  `catalog/plex.yaml` and the Secret's header comment (left alone — editing a `.sops.yaml`
+  comment risks the MAC, and touching `catalog/plex.yaml` re-renders a golden).
+
+### Measured, once the pipeline actually worked
+
+`gate / gate (pull_request)` 1m34s–1m58s. Full claim loop end to end **~2m45s** — comfortably
+inside the window, which retires the theory that a short-lived token cannot survive a PR flow.
+That theory was mine and it was wrong: the pipeline was never too slow, it simply did not exist.
+
+### State now
+
+- `main` **protected**, `user_can_push:false`. Runner `incluster-dind` online.
+- `CLAUDE.md` is the agent delivery contract; `infra/gitea/` holds the protection + runner scripts.
+- **All 37 runbooks** that instructed `git push origin main` now describe the PR flow — that
+  instruction had become actively blocked, not merely stale.
+- `infra/plex/claim-ship.sh` drives the claim loop end to end.
+
+### Next steps
+
+1. **`enable_status_check: true`** with `gate / gate (pull_request)` as the only context. The
+   prerequisite is met: the gate has been observed green on real PRs.
+2. Add `jakman85` as a repo **collaborator**, then `required_approvals: 1`. (Currently the only
+   org member is `gitea-admin`, which agents authenticate as — requiring approval before that is
+   fixed would leave nobody able to approve.)
+3. Run the **push probe** against protection: `gitea-admin` is a site admin and Gitea may let it
+   bypass. `user_can_push:false` is the API's opinion, not proof.
+4. Align `gate.yml`'s push trigger with this repo's `feat/**` prefix.
+5. Convert the `dind` container to a **native sidecar** with a `startupProbe`, per the standing
+   FOLLOW-UP note in `apps/wave-1/gitea-actions.yaml`. It deletes both hand-rolled wait loops and
+   the envsubst hazard. Land it ALONE — it is the merge gate for the whole repo.
+6. Create the Plex libraries over `/media/movies`, `/media/tv`, `/media/photos`.
+
+## 2026-09-29 (later) — the merge gate is now real, enforced, and proven. **PM HANDOFF.**
+
+Closes the gap opened by the earlier entry today. Eight PRs, all gated; every claim below was
+measured against the live API or cluster, not inferred from this repository.
+
+### Final state
+
+| | Verified |
+|---|---|
+| `main` protected | `protected:true`, `user_can_push:false` — and a direct push as site-admin `gitea-admin` was **provoked and rejected** (`pre-receive hook declined`) |
+| Gate required | `enable_status_check:true`, contexts `["gate / gate (pull_request)"]` |
+| Approvals | `required_approvals:1`; `jakman85` holds `owner` |
+| Runner | `incluster-dind` online, now a **native sidecar** (`initContainers`, `restartPolicy: Always`, TLS-verified `docker info` startupProbe) |
+| Gate time | ~1m35s |
+| Plex | claimed 2026-09-29; `PlexOnlineToken` present on the `plex-config` PVC |
+
+### What was corrected in the documentation
+
+`CLAUDE.md` §2b — *never infer platform state from this repository* — is the durable outcome.
+The 2026-08-31 entry asserting an enforced gate is now annotated **in place**. The "Phase 0 /
+no cluster exists" framing across `ci/`, `clusters/` and `conformance/` READMEs is banner-marked
+as describing a live gap rather than a stage. All 37 runbooks now describe the PR flow. Two
+`.sops.yaml` headers and `catalog/plex.yaml` were corrected (comment edits verified not to break
+the SOPS MAC, and not to move the rendered golden).
+
+### A claim of mine that shipped wrong, and is now corrected in place
+
+PR #7 stated the sidecar conversion would remove the one-per-start dockerd line
+`http: TLS handshake error from 127.0.0.1: EOF`. **It does not.** Exactly one still appears,
+immediately after `API listen on [::]:2376` — it is now the startupProbe's first attempt,
+landing microseconds after dockerd begins listening and cut off by `timeoutSeconds: 1`; the
+next poll succeeds. Cosmetic, and deliberately not chased: raising the timeout to hide one
+benign startup line would slow every subsequent poll. The manifest comment now says so.
+
+### The root cause nobody has fixed yet
+
+Gitea's repo settings live in its own database — invisible to Flux and unreadable by the
+conformance gate. **No check exists that could have caught "protection off, runner gone."**
+`infra/gitea/` declares and can re-assert the desired state, but it is not a detector. A
+conformance check that reads the live Gitea API and fails on divergence is the highest-value
+remaining work in this repo, and it is the difference between this having been fixed and this
+being unable to recur.
+
+### Next steps
+
+1. **Write that Gitea-settings conformance check.** Everything else here is a symptom.
+2. Wire `vars.CLUSTER_AVAILABLE` + `secrets.CONFORMANCE_TRIGGER_KUBECONFIG` so **L2/L3 runs in
+   CI**. A green PR currently proves the L0/L1 offline subset only — runtime behaviour is
+   unverified on every merge.
+3. Re-run `infra/gitea/verify-then-enforce.sh` after any future act-runner change.
+4. `required_approvals: 1` means a PR you open as `jakman85` needs `gitea-admin` to approve.
+   If that friction bites, either open agent work as `gitea-admin` and review as `jakman85`, or
+   drop back to relying on the status check alone.
+
+## 2026-09-29 (later still) — the enforcement we "turned on" was overridable by the account that merges
+
+`enable_status_check: true` and `required_approvals: 1` were set earlier today and recorded as
+done. **They did not bite.** `block_admin_merge_override` was `false`, and Gitea lets repository
+admins override merge requirements without it. `jakman85` holds `owner`, so both requirements
+were bypassable by exactly the account that performs every merge.
+
+**The evidence was already on the record and nobody read it:** PRs #5, #6, #7, #8 and #9 all
+merged while `required_approvals: 1` was in force, with **zero approvals recorded on any of
+them**. A one-line check (`/pulls/{n}/reviews`) would have shown it at any point.
+
+Two process failures, both mine, both the same shape as the original defect:
+
+1. **I verified with the wrong view.** `/branches/main` reports `protected`,
+   `enable_status_check` and `required_approvals` but NOT `block_admin_merge_override`. It
+   showed everything green while the rule was overridable. The authoritative object is
+   `/branch_protections/main` — read it and enumerate its keys.
+2. **I probed for a field name I assumed existed.** This Gitea (1.24.3) has no
+   `apply_to_admins`. Asking `jq` for it returns `null`, which reads exactly like "set to
+   false" but means "key absent" — so a PATCH that silently did nothing looked like a real
+   before/after. **`jq` returning null is not a value; it may be an absence.** Enumerate with
+   `to_entries`.
+
+Now `block_admin_merge_override: true`, applied and read back. `infra/gitea/require-gate.sh`
+sets it, and `infra/gitea/block-admin-override.sh` fixes it in isolation.
+
+**This is still only a setting.** It is proven when a green PR is REFUSED a merge until
+approved — and that has not been observed yet. This PR is the test: if it merges without an
+approval, the fix did not work and the finding stands.
+
+### The governance question this exposes
+
+There are two accounts: `jakman85` (owner, the human) and `gitea-admin` (the agent). With
+approvals now genuinely required, an agent-authored PR is approved by the human — which is the
+intent. But a PR authored by `jakman85` needs `gitea-admin` to approve it, i.e. the automation
+account rubber-stamping the human. That is ceremony, not review. Options: keep it and accept
+the ceremony; drop `required_approvals` to 0 and rely on the status check, which is the gate
+that actually verifies something; or add a second human account. **This is a decision for the
+operator, not a default to be chosen silently.**
@@ -17,6 +17,7 @@ resources:
 - pihole-dns.yaml
 - pihole-oauth2-proxy-credentials.sops.yaml
 - pihole.yaml
+- plex-claim-secret.sops.yaml
 - plex.yaml
 - qbittorrent-bootstrap-secret.sops.yaml
 - qbittorrent-oauth2-proxy-credentials.sops.yaml
@@ -0,0 +1,77 @@
+# Plex plex.tv server-claim token (WS-APPS). SOPS-encrypted (age).
+# Consumed by catalog/plex.yaml as env PLEX_CLAIM via valueFrom.secretKeyRef.
+#
+# NOTE ON THESE COMMENTS: they carry NO dollar-brace expressions, deliberately. This file is applied
+# by the apps-wave-0 Flux Kustomization, which runs envsubst over the built output, and kustomize
+# preserves body comments into that stream (see conformance/tests/l0-static/flux-postbuild-
+# substitution). A braced token quoted from the upstream shell script would be a live hazard here,
+# not prose, so the upstream conditions below are described in words.
+#
+# THIS FILE NO LONGER HOLDS THE PLACEHOLDER. Corrected 2026-09-29: the server was CLAIMED on that
+# date, and the value here is the SPENT token from that claim. It is inert and safe to leave --
+# upstream gates the exchange on PlexOnlineToken being ABSENT from /config, and it is now present on
+# the plex-config PVC, so this value is a NO-OP on every start whatever it holds. Deleting the Secret
+# would NOT be safe (see below), which is why a spent token stays here rather than being blanked.
+#
+# The claim token is SHORT-LIVED -- minutes. Read the countdown that plex.tv/claim renders when it
+# issues one; do NOT budget against a number written in this tree. Earlier revisions of this comment
+# asserted a "~4-minute TTL" as established fact with no citation, and upstream plexinc/pms-docker
+# documents no expiry at all. That uncited number was load-bearing for the whole procedure design.
+#
+# This file exists to make the Secret — and therefore the env reference — ALWAYS RESOLVABLE IN THE
+# FLUX GRAPH, not to carry a working token. Re-claiming (only needed after a /config wipe) is
+# scripted: infra/plex/claim-ship.sh, measured at ~2m45s end to end.
+#
+# WHY SHIPPING A PLACEHOLDER IS SAFE (verified against plexinc/pms-docker branch `master`, the
+# s6-overlay v2 script root/etc/cont-init.d/40-plex-first-run, cross-checked byte-identical against
+# the same file extracted from the running `plexinc/pms-docker:latest` image):
+#   - line 106 guards the claim exchange on TWO conditions: PLEX_CLAIM non-empty AND the
+#     PlexOnlineToken pref read out of /config empty. Once the server is claimed that block is
+#     skipped entirely, so whatever this Secret holds is a NO-OP on every later start. Upstream
+#     README: "If server is already logged in, this parameter is ignored."
+#   - on a still-UNCLAIMED /config, an invalid or expired token makes plex.tv answer 403 with an
+#     empty body. curl's exit status is never checked, the sed for the authentication-token element
+#     yields the empty string, and the success branch at line 120 simply does not fire. There is no
+#     else branch and no non-zero exit; the script has no `set -e` and ends by touching
+#     /.firstRunComplete and echoing, so it exits 0. The server comes up UNCLAIMED and serving.
+#     It does NOT crashloop.
+#   - the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS, whose default is 0 ("continue silently even
+#     if a cont-init script failed"), so even a hypothetical non-zero exit would not halt startup.
+# A MISSING Secret would NOT be safe: the ServiceDefinition schema's valueFrom.secretKeyRef shape
+# has no `optional` field, so an absent Secret is a hard CreateContainerConfigError. That is exactly
+# why this file is committed and listed in apps/wave-0/kustomization.yaml instead of being created
+# by hand at claim time.
+#
+# INJECTION / ROTATION (operator, not agents) — runbooks/plex-claim.md has the timed procedure:
+#   infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
+# Never decrypt this file into a shell, a log, or a shared terminal.
+apiVersion: v1
+kind: Secret
+metadata:
+    name: plex-claim
+    namespace: plex
+    labels:
+        app.kubernetes.io/name: plex
+type: Opaque
+stringData:
+    PLEX_CLAIM: ENC[AES256_GCM,data:nLmzeUBWRhRjUakrGiGzQpQuqZNcIYvRH6M=,iv:z0on/dArsNejNLrp3V1nSiNLFzKcREhPuMO1634CwQw=,tag:Ih23CT1JYna/5lfYQUb1Dw==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2dWtqOVNQalZ2QlRxOXJa
+            cDc1MTdtWENkdWIxS1JId3dIU1JDSmk0ODNrCk1YUzhFWXR3bUxyV2NNcmFOakdz
+            THl1Q284dFJxTy9Rd1h0eXJJaVZtdmcKLS0tIERuN0ZFeW9zdi9aNFRYNWxmUS9s
+            QUtYdHEvN042ajNoNktOLzlYQmQrc1kK4NHozMbXVzNGYbUBRbK6a9Uw/sVPAfe6
+            RW5KWzZ4I/hloqjN3zzVISoSMieMkBGDnF5qZS5N6FqIc6/HrsDD8A==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-09-29T06:31:16Z"
+    mac: ENC[AES256_GCM,data:62BmBAq45CD2g6LDyx+Ho4bTfrhJcaro/RB/iM+OaRc4a/D83MhjbcBL11Uz/O726inqhkvCKDJ/fFemoc50JDbtC79cYMaxrGJNjFM0sdckWrowPvh6ie7HHHUBRBaefl+gNfnc+pbvcncvRzwA4RyzcZse1wbxKfHN8FjEopE=,iv:h0cMShs3tIgnY6eruUYymYrcLFzQZLk/uXQJ/Jsd5ek=,tag:nZ9Pau7p4xWT1QCN1vv2vA==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4
@@ -132,6 +132,11 @@ spec:
          value: Etc/UTC
        - name: ALLOWED_NETWORKS
          value: 10.0.0.0/8,192.168.0.0/16
+        - name: PLEX_CLAIM
+          valueFrom:
+            secretKeyRef:
+              name: plex-claim
+              key: PLEX_CLAIM
        volumeMounts:
        - name: config
          mountPath: /config
@@ -4,7 +4,13 @@
 # against Gitea. Flux decrypts it via the sops-age key at reconcile (apps Kustomization
 # decryption is wired in clusters/dev/kustomization.yaml). Rotate by resetting the token
 # in Gitea (admin -> actions -> runners) and re-encrypting a fresh value here.
-# Recover/inspect: infra/devcontainer/dev.sh run sops -d apps/gitea-actions-runner-secret.sops.yaml
+# ROTATE: ./infra/gitea/runner-token-rotate.sh -- it mints a fresh instance-level token from the
+# Gitea API and writes it here via `sops set`, without the value ever being printed.
+# Do NOT `sops -d` this file into a terminal, a log, or an agent transcript. To check only that
+# it still decrypts, redirect: `sops -d <file> > /dev/null`.
+# NOTE: a fresh token alone may not re-register the runner. act_runner registers ONCE and keeps
+# its identity in /data/.runner on the act-runner-data PVC; if Gitea no longer holds a matching
+# row it skips registration silently. Clearing that PVC is what forces a real re-registration.
 apiVersion: v1
 kind: Secret
 metadata:
@@ -99,70 +99,79 @@ spec:
      # CI holds no cluster credentials — do not mount a SA token into the runner or
      # (via the shared netns) the job containers (R-SEC-02, docs/14).
      automountServiceAccountToken: false
+      # ---- Docker-in-Docker: the job executor daemon (privileged) -------------------
+      #
+      # NATIVE SIDECAR (k3s v1.36; sidecar containers are GA). It is an initContainer with
+      # `restartPolicy: Always`, which means: it starts BEFORE the regular containers, keeps
+      # running alongside them, and is terminated after them. kubelet holds `runner` until
+      # this container's startupProbe passes, so the ordering is enforced by the kubelet
+      # instead of hand-rolled shell in the runner.
+      #
+      # This REPLACES two bounded wait-loops that used to live in the runner's `command:`.
+      # Deleting them also deletes an envsubst hazard: clusters/base/apps-wave-1.yaml runs
+      # postBuild substitution over this ENTIRE manifest, comments included, and those loops
+      # depended on bare dollar-names surviving it untouched. There is now no shell here at
+      # all, so there is nothing to escape or mis-escape.
+      #
+      # WHY THE PROBE IS `exec docker ... info` AND NOT `tcpSocket`. A tcpSocket probe would
+      # re-open the exact gap the old Stage 1 existed to close: dockerd writes the TLS certs
+      # and only THEN begins listening, so the port can accept a connection while the certs
+      # are absent or incomplete -- and act_runner hard-exits on a missing
+      # /certs/client/ca.pem. A full TLS-verified `docker info` proves BOTH that the certs
+      # are usable and that the daemon answers.
+      #
+      # CORRECTION (measured 2026-09-29, after this conversion shipped): the commit that
+      # introduced this claimed the conversion would REMOVE the one-per-start dockerd line
+      # `http: TLS handshake error from 127.0.0.1: EOF`. IT DOES NOT. Exactly one still
+      # appears, immediately after "API listen on [::]:2376" -- it is now this probe's FIRST
+      # attempt, landing microseconds after dockerd begins listening and cut off by
+      # timeoutSeconds: 1. The next poll succeeds. It is cosmetic, exactly as cosmetic as it
+      # was before, and deliberately NOT chased: raising timeoutSeconds to hide one benign
+      # startup line would slow every subsequent poll. Left documented rather than silenced.
+      #
+      # periodSeconds 1 x failureThreshold 60 preserves the previous 60s cap.
+      initContainers:
+        - name: dind
+          restartPolicy: Always   # <- this field is what makes it a SIDECAR
+          image: docker:27-dind
+          securityContext:
+            privileged: true
+          env:
+            # Generate + publish TLS certs into the shared emptyDir; dockerd then listens
+            # on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
+            - name: DOCKER_TLS_CERTDIR
+              value: /certs
+          volumeMounts:
+            - name: certs
+              mountPath: /certs
+            - name: dind-storage
+              mountPath: /var/lib/docker
+          resources:
+            requests:
+              cpu: 100m
+              memory: 256Mi
+            limits:
+              memory: 2Gi
+          startupProbe:
+            exec:
+              command:
+                - docker
+                - --host=tcp://127.0.0.1:2376
+                - --tlsverify
+                - --tlscacert=/certs/client/ca.pem
+                - --tlscert=/certs/client/cert.pem
+                - --tlskey=/certs/client/key.pem
+                - info
+            periodSeconds: 1
+            failureThreshold: 60
      containers:
        # ---- act_runner: registers with Gitea, dispatches jobs to the DinD daemon ----
        - name: runner
          image: gitea/act_runner:0.2.11
-          # TWO-STAGE BOUNDED WAIT on the DinD sidecar, then exec the image's normal
-          # entrypoint (/sbin/tini -- /opt/act/run.sh), which registers-from-env on first
-          # boot and runs `act_runner daemon`.
-          #
-          # STAGE 1 — the certs must EXIST: act_runner reads /certs/client/ca.pem at startup
-          #   and hard-exits if it is missing (a startup race against the sidecar).
-          # STAGE 2 — the daemon must actually ACCEPT a connection: the cert FILE appearing is
-          #   necessary but NOT sufficient. dockerd writes the certs and only THEN begins
-          #   listening on :2376, so a runner that proceeds on the file alone still hard-exits
-          #   with "cannot ping the docker daemon at tcp://localhost:2376" — observed
-          #   crashlooping 11 times in ~8 minutes on a fresh pod after a node bounce.
-          # Both loops are bounded (60 x 1s) and exit 1 on timeout, so a genuinely dead sidecar
-          # surfaces as a CrashLoopBackOff with a clear message instead of hanging forever.
-          #
-          # ENVSUBST SAFETY — the BARE $i / $j below are CORRECT. Do not "tidy" them into the
-          #   braced form or the double-dollar form; both are wrong here, for different reasons.
-          #   clusters/base/apps-wave-1.yaml carries postBuild.substituteFrom cluster-vars, so
-          #   Flux envsubst runs over this ENTIRE manifest — YAML comments very much included.
-          #   Its contract (drone/envsubst; the same one spelled out in clusters/base/apps-vms.yaml,
-          #   and re-verified against this exact file with `flux envsubst`, flux v2.4.0) is:
-          #     - A bare  $name  is NOT a reference to envsubst: it passes through untouched, and
-          #       that holds even when the name IS set in cluster-vars. So $i and $j reach the
-          #       cluster verbatim and both 60s bounds below are real, enforced bounds.
-          #     - $((i + 1))  is likewise untouched: "(" cannot begin a variable name.
-          #     - A BRACED dollar-name IS a reference and gets replaced — with the empty string
-          #       when the name is unset. One of those here would make a bound read [ "" -ge 60 ],
-          #       which errors, short-circuits the &&, and silently turns the wait UNBOUNDED.
-          #     - A DOUBLE-dollar braced name is envsubst's escape and survives as the braced
-          #       form — still wrong for us, because on any path where envsubst does NOT run
-          #       (a plain `kustomize build` or `kubectl apply -k` of this directory) the shell
-          #       reads that leading double-dollar as its own PID, so a bound would compare a
-          #       string like 1234{i} against 60. The bare form is the only spelling that is
-          #       correct on BOTH paths.
-          #     - A braced name containing punctuation is a HARD ERROR ("unable to parse variable
-          #       name") that fails this whole Kustomization — and this Kustomization gates every
-          #       merge in the repo. So never write one anywhere in this file, INCLUDING in a
-          #       comment. That is why this note names the three forms in words rather than
-          #       showing them: an earlier draft of it spelled them out literally and broke
-          #       `flux envsubst` on the file. (Same trap the indexed-expansion warning in
-          #       clusters/base/apps-vms.yaml describes.)
-          command:
-            - /bin/sh
-            - -c
-            - |
-              # Stage 1: the sidecar's TLS client certs must appear in the shared emptyDir.
-              i=0
-              until [ -f /certs/client/ca.pem ]; do
-                i=$((i + 1))
-                [ "$i" -ge 60 ] && { echo "dind certs never appeared" >&2; exit 1; }
-                echo "waiting for dind TLS certs ($i)"; sleep 1
-              done
-              # Stage 2: dockerd must actually accept a connection on :2376. busybox nc in this
-              # Alpine-based image (gitea/act_runner:0.2.11) supports -z (scan) and -w (timeout).
-              j=0
-              until nc -z -w1 127.0.0.1 2376; do
-                j=$((j + 1))
-                [ "$j" -ge 60 ] && { echo "dind daemon never accepted a connection on 2376" >&2; exit 1; }
-                echo "waiting for dind daemon on :2376 ($j)"; sleep 1
-              done
-              exec /sbin/tini -- /opt/act/run.sh
+          # No `command:` override. The image's own entrypoint (/sbin/tini -- /opt/act/run.sh)
+          # registers from env on first boot and runs `act_runner daemon`. The two wait-loops
+          # that used to live here are gone: kubelet now holds this container until the dind
+          # sidecar's startupProbe passes (see initContainers above).
          env:
            - name: GITEA_INSTANCE_URL
              value: https://gitea.${BASE_DOMAIN}
@@ -198,52 +207,6 @@ spec:
              memory: 128Mi
            limits:
              memory: 512Mi
-        # ---- Docker-in-Docker: the job executor daemon (privileged) ----
-        #
-        # FOLLOW-UP (deliberately NOT bundled into the readiness fix above — own change):
-        #   Convert this container into a NATIVE SIDECAR and delete both shell wait-loops.
-        #   k3s is now v1.36, where sidecar containers are GA, so kubelet can enforce the
-        #   ordering that the loops above hand-roll — which also removes the envsubst hazard
-        #   from this file entirely (no shell, no $i/$j, nothing to escape or mis-escape).
-        #   Shape to build:
-        #     1. Move this whole container into `initContainers:` as the FIRST entry.
-        #     2. Add `restartPolicy: Always` to it. That single field is what makes an
-        #        initContainer a sidecar: it starts before the regular containers, keeps
-        #        running alongside them, and is terminated after them.
-        #     3. Give it a `startupProbe` that succeeds only when dockerd accepts a TLS
-        #        connection — e.g. `exec` a `docker -H tcp://127.0.0.1:2376 --tlsverify
-        #        --tlscacert /certs/client/ca.pem --tlscert /certs/client/cert.pem
-        #        --tlskey /certs/client/key.pem info`. Do NOT use a bare `tcpSocket` probe:
-        #        it would re-open the same "port accepting but TLS/certs not ready" gap that
-        #        Stage 1 above exists to close. `periodSeconds: 1` + `failureThreshold: 60`
-        #        preserves today's 60s cap.
-        #     4. kubelet then holds the `runner` container until that startupProbe passes, so
-        #        `runner` can drop its `command:` override completely and run the image's
-        #        default entrypoint.
-        #   Why it is a separate change: this Deployment is the ENFORCED MERGE GATE for this
-        #   entire repository. If the conversion is wrong, nothing can merge — so it must land
-        #   alone, where the failure is unambiguous and not entangled with anything else, and
-        #   only after a real workflow run has been observed green on the converted pod.
-        - name: dind
-          image: docker:27-dind
-          securityContext:
-            privileged: true
-          env:
-            # Generate + publish TLS certs into the shared emptyDir; dockerd then listens
-            # on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
-            - name: DOCKER_TLS_CERTDIR
-              value: /certs
-          volumeMounts:
-            - name: certs
-              mountPath: /certs
-            - name: dind-storage
-              mountPath: /var/lib/docker
-          resources:
-            requests:
-              cpu: 100m
-              memory: 256Mi
-            limits:
-              memory: 2Gi
      volumes:
        - name: data
          persistentVolumeClaim:
@@ -32,11 +32,46 @@ env:
  # (10.43.0.0/16); 192.168.0.0/16 covers the home LAN.
  - name: ALLOWED_NETWORKS
    value: 10.0.0.0/8,192.168.0.0/16
-  # PLEX_CLAIM is deliberately NOT set here: a https://plex.tv/claim token has a ~4-min TTL, far
-  # shorter than the commit -> push -> Flux reconcile -> pod-roll cycle, so any token committed to git
-  # is already expired by the time the pod starts. Claiming is a one-time LIVE step (UI sign-in, or a
-  # single API POST) against the running server; the claim then persists on the /config PVC and
-  # survives pod rolls (including the roll this env change causes), so it never belongs in git.
+  # PLEX_CLAIM — the ONE-SHOT plex.tv server claim, sourced from the SOPS Secret `plex-claim`
+  # (apps/wave-0/plex-claim-secret.sops.yaml, same wave => applied in the same reconcile as this
+  # workload). Plex cannot create libraries over the mounted bulk share while the server is
+  # UNCLAIMED, so this is wired; it was previously left out on the reasoning below, and only one
+  # half of that reasoning survived contact with the upstream source.
+  #
+  # A SHORT TTL IS THE OPERATIONAL CONSTRAINT, not a reason to abstain. A https://plex.tv/claim token
+  # expires in MINUTES — read the countdown the claim page renders; do not budget against a number
+  # written in this tree. (This comment previously asserted "~4 minutes" as established fact. It was
+  # UNCITED, upstream plexinc/pms-docker documents no expiry at all, and that number was load-bearing
+  # for the whole procedure design — including a since-disproven claim that the delivery pipeline was
+  # too slow to fit inside it. Measured 2026-09-29: the full loop is ~2m45s and fits.)
+  #
+  # The live token is injected by the operator inside that window, with an explicit `flux reconcile`
+  # to collapse the wait. Scripted end to end as infra/plex/claim-ship.sh; runbooks/plex-claim.md is
+  # the procedure. SERVER CLAIMED 2026-09-29 — this env is now a permanent no-op (see below), and the
+  # committed value is the spent token from that claim rather than the original placeholder.
+  #
+  # ONE-SHOT, AND SAFE TO SIT HERE EXPIRED. Verified from plexinc/pms-docker's own first-run script
+  # (branch `master`, root/etc/cont-init.d/40-plex-first-run, confirmed byte-identical to the file in
+  # the running `:latest` image), NOT assumed:
+  #   - the exchange (line 106) is guarded on TWO conditions: PLEX_CLAIM non-empty AND the
+  #     PlexOnlineToken pref in /config empty. Once claimed, the block is SKIPPED — this env is a
+  #     no-op on every subsequent start, whatever it holds. (Upstream README: "If server is already
+  #     logged in, this parameter is ignored.") The claim itself persists on the /config PVC as before.
+  #   - an invalid/expired token on a still-unclaimed /config gets a 403 with an empty body; curl's
+  #     status is unchecked, the token `sed` yields "", the success branch at line 120 does not fire,
+  #     there is no `else` and no `exit 1`, and the script (no `set -e`) exits 0. It comes up UNCLAIMED and
+  #     SERVING — it does NOT crashloop. The image also leaves S6_BEHAVIOUR_IF_STAGE2_FAILS at its
+  #     default 0, which continues startup even when a cont-init script fails.
+  # So the earlier "a placeholder claim Secret would crashloop the pod" note was WRONG about the
+  # placeholder; it was right only about an ABSENT Secret. secretKeyRef here has no `optional` field,
+  # so the Secret must always exist in the Flux graph — which is why it is committed, encrypted, and
+  # listed in apps/wave-0/kustomization.yaml rather than hand-created at claim time. No schema change
+  # is needed: `optional` would only matter if a missing Secret had to be tolerated, and it doesn't.
+  - name: PLEX_CLAIM
+    valueFrom:
+      secretKeyRef:
+        name: plex-claim
+        key: PLEX_CLAIM
  # ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex
  # which URL to advertise to plex.tv for REMOTE clients. The only candidate is
  # https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL
@@ -56,10 +91,12 @@ storage:
                              # container's EPHEMERAL layer -> unclaimed + all state lost on restart. The image inits as
                              # root (s6-svscan PID 1), chowns /config to the plex user, then drops -> no fsGroup needed
                              # (verified in-cluster: PID1=root, "Plex Media Server"=plex). Persisting /config here also
-                              # makes the claim survive restarts once claimed. To auto-claim on first boot instead of the
-                              # UI sign-in, the operator adds a PLEX_CLAIM env from a fresh https://plex.tv/claim token
-                              # (4-min TTL) — deliberately NOT wired here: the env secretKeyRef pattern has no `optional`,
-                              # so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
+                              # makes the claim survive restarts once claimed — which is what makes the PLEX_CLAIM env
+                              # above a ONE-SHOT: the token is exchanged for a PlexOnlineToken written here, and the
+                              # exchange is then skipped on every later start. (The older note here claimed a
+                              # placeholder claim Secret "would crashloop the pod"; upstream's first-run script says
+                              # otherwise — see the PLEX_CLAIM comment in `env` above. An ABSENT Secret is the hard
+                              # failure, and that is why one is committed.)
  # THE LIBRARY (docs/04 bulk tier). One READ-ONLY mount of the export ROOT, giving /media/movies,
  # /media/tv and /media/photos — the whole tree the layout actually has (`ls /mnt/Storage/plex`:
  # downloads/ movies/ photos/ tv/).
@@ -1844,6 +1844,10 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
    convention:
      - `<svc>.yaml`                        -> wave(svc)
      - `<svc>-bootstrap-secret.sops.yaml`  -> wave(svc)   (consumed by svc's env, same apply)
+      - `<svc>-claim-secret.sops.yaml`      -> wave(svc)   (a one-shot upstream-service claim token
+        consumed by svc's env — plex's plex.tv claim. Same wave as the app for the same reason the
+        bootstrap secret is: an env `secretKeyRef` has no `optional`, so the Secret must be applied
+        in the SAME reconcile as the workload or the pod is CreateContainerConfigError.)
      - `<svc>-oauth2-proxy-credentials.sops.yaml` -> wave(svc)  (forward-auth sidecar OIDC creds)
      - `<svc>-db.yaml`                     -> wave(svc-db) (a strictly-lower wave; CNPG DB)
      - `<svc>-db-app.sops.yaml`            -> wave(svc-db) (CNPG initdb owner secret)
@@ -1870,6 +1874,11 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
        w = waves[name]
        add(w, f"{name}.yaml")
        add(w, f"{name}-bootstrap-secret.sops.yaml")
+        # A one-shot claim token for an UPSTREAM service (plex's plex.tv claim). Same wave as the
+        # app: it is read by the app's env at pod start, and an env secretKeyRef has no `optional`,
+        # so a Secret applied in a later wave would leave the pod in CreateContainerConfigError.
+        # Placed only when the file exists, so services without one are unaffected.
+        add(w, f"{name}-claim-secret.sops.yaml")
        # forward-auth (Option B): the per-app oauth2-proxy sidecar's OIDC client + cookie secret.
        # Same wave as the app (the sidecar reads it at pod start); placed only when the file exists,
        # so non-forward-auth apps are unaffected. Mirrors the bootstrap-secret naming convention.
@@ -2548,7 +2557,8 @@ def _selftest() -> int:
    # 7a) plan_apps_waves: sidecars inherit their unit's wave; not-onboarded SDs are skipped;
    #     EXTRA_UNITS anchor strictly above; a file absent from the tree is never placed.
    present = {
-        "a.yaml", "svc.yaml", "svc-bootstrap-secret.sops.yaml",
+        "a.yaml", "a-claim-secret.sops.yaml",
+        "svc.yaml", "svc-bootstrap-secret.sops.yaml",
        "svc-db.yaml", "svc-db-app.sops.yaml", "svc-db-backup-credentials.sops.yaml",
        "svc-db-backup-credentials-prod.sops.yaml",
        "gitea.yaml",
@@ -2559,6 +2569,10 @@ def _selftest() -> int:
    plan = plan_apps_waves(plan_sds, present)
    flat = {f: wv for wv, fs in plan.items() for f in fs}
    check(flat.get("a.yaml") == 0, "leaf app must land in wave 0")
+    check(flat.get("a-claim-secret.sops.yaml") == 0,
+          "a `<svc>-claim-secret.sops.yaml` (plex's plex.tv claim) must be placed in the SAME wave "
+          "as its app — an env secretKeyRef has no `optional`, so a later wave means "
+          "CreateContainerConfigError")
    check(flat.get("svc-db.yaml") == 0 and flat.get("svc-db-app.sops.yaml") == 0
          and flat.get("svc-db-backup-credentials.sops.yaml") == 0
          and flat.get("svc-db-backup-credentials-prod.sops.yaml") == 0,
@@ -1,5 +1,19 @@
 # `ci/` — portable merge gate (runnable today, without Gitea)

+> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
+> This document speaks as though no cluster exists and the platform is scaffolding. **A
+> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
+> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
+> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
+> describing a **live gap** rather than an expected stage — most importantly, L2/L3
+> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
+> coverage hole, not a phase.
+>
+> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
+> documentation rather than measurement, and `main` was unprotected with zero runners
+> registered the whole time. **Query the live API before asserting platform state.**
+
+
 **Task T-0.2 · R-TEST-02 (reproducible, single-command) · Owner WS-GITOPS (CI).**

 The same merge gate the Gitea Actions workflow runs
@@ -1,5 +1,19 @@
 # clusters/

+> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
+> This document speaks as though no cluster exists and the platform is scaffolding. **A
+> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
+> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
+> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
+> describing a **live gap** rather than an expected stage — most importantly, L2/L3
+> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
+> coverage hole, not a phase.
+>
+> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
+> documentation rather than measurement, and `main` was unprotected with zero runners
+> registered the whole time. **Query the live API before asserting platform state.**
+
+
 Flux entrypoints and the environment-overlay model (**R-ENV-02**, **R-GITOPS-01/02/03**).

 Governing spec: [`docs/05-gitops-and-delivery.md`](../../home-network-k3s/docs/05-gitops-and-delivery.md),
@@ -1,5 +1,19 @@
 # conformance/ — the home-k3s conformance harness (WS-TEST)

+> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
+> This document speaks as though no cluster exists and the platform is scaffolding. **A
+> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
+> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
+> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
+> describing a **live gap** rather than an expected stage — most importantly, L2/L3
+> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
+> coverage hole, not a phase.
+>
+> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
+> documentation rather than measurement, and `main` was unprotected with zero runners
+> registered the whole time. **Query the live API before asserting platform state.**
+
+
 This subtree is the **definition of "done"** for the whole platform. Nothing is accepted
 on a demo, a screenshot, or "it reconciled": every requirement is proven by an
 **automated, reproducible, behavior-based test** tagged with the requirement ID(s) it
@@ -1,5 +1,19 @@
 # In-cluster conformance Job (L2/L3)

+> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
+> This document speaks as though no cluster exists and the platform is scaffolding. **A
+> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
+> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
+> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
+> describing a **live gap** rather than an expected stage — most importantly, L2/L3
+> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
+> coverage hole, not a phase.
+>
+> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
+> documentation rather than measurement, and `main` was unprotected with zero runners
+> registered the whole time. **Query the live API before asserting platform state.**
+
+
 Per **ADR-0011** and **docs/14 "Where each layer runs"**: L0/L1 run in CI without a
 cluster, but **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job**
 on the dev/test cluster that **CI triggers and waits for**. CI holds **no** cluster
@@ -43,7 +43,65 @@ list. That is precisely the observed symptom: zero runners, pod running, no erro
 looks at. Clearing the PVC is what forces a real re-registration; the script prints the
 commands.

-## Sequence — order matters
+## Sequence — DONE 2026-09-29
+
+All four steps are complete and each was verified, not assumed:
+
+| Step | State |
+|---|---|
+| 1. Runner registered | ✅ `incluster-dind` online. Root cause of its absence was a stale `/data/.runner` identity; clearing the PVC forced re-registration |
+| 2. Gate observed green on a real PR | ✅ `gate / gate (pull_request)` ~1m35s |
+| 3. `enable_status_check: true` | ✅ context `gate / gate (pull_request)` — that one ONLY |
+| 4. Collaborator + approvals | ✅ `jakman85` is owner; `required_approvals: 1` |
+
+Direct push to `main` was **provoked and rejected** (`pre-receive hook declined`) as
+`gitea-admin`, a site admin — so push protection is not merely declared, it holds for the
+account agents authenticate as.
+
+### 5. Block the admin override — the step the first pass MISSED
+
+`enable_status_check` and `required_approvals` were set on 2026-09-29 and **still did not
+bite**, because `block_admin_merge_override` was `false`. Gitea lets repo admins override
+merge requirements without it, and `jakman85` holds `owner` — so the two settings that looked
+like the gate were overridable by precisely the account that merges.
+
+**The evidence was sitting in plain sight:** PRs #5 through #9 all merged with
+`required_approvals: 1` in force and **zero approvals recorded on any of them**. Checking the
+*branch* view (`/branches/main`) hid this — it does not expose the field. Read the rule object
+(`/branch_protections/main`) and enumerate its keys.
+
+> **Do not probe for field names you assume exist.** This Gitea (1.24.3) has no
+> `apply_to_admins`; asking `jq` for it returns `null`, which reads like "set to false" but
+> means "key absent". Enumerate with `to_entries` and act on what is really there.
+
+Now `block_admin_merge_override: true`. **That is still only a setting, not evidence** — it is
+proven when a green PR is REFUSED a merge until approved, and not before.
+
+### `require-gate.sh` / `verify-then-enforce.sh`
+
+`verify-then-enforce.sh` is the re-runnable form of the whole safe order. It merges nothing —
+merging is a human's reviewed act — but it forces the Flux reconcile and then **aborts** unless
+the runner pod is healthy, a runner is online, and a real workflow goes green on it, before
+enabling enforcement and probing the push. `require-gate.sh` is the enforcement step alone.
+
+Re-run `verify-then-enforce.sh` after any future change to the act-runner Deployment.
+
+### Why these scripts exist at all, and what is still missing
+
+Gitea's repo settings live in **its own database**, not in anything Flux reconciles or the
+conformance gate can read. Nothing detects drift from what this directory declares. That is
+the root cause of the entire 2026-09-29 session: branch protection was off and the runner was
+gone for a month while the log recorded both as working, and **no check existed that could
+have caught it.**
+
+These scripts encode the desired state and can re-assert it, but they are not a detector. The
+durable fix is a conformance check that reads the live Gitea API and fails when it diverges
+from what is declared here — turning a class of silent, invisible failure into a loud one.
+**That check does not exist yet and is the highest-value remaining work in this directory.**
+
+## Historical sequence (kept for context)
+
+### Original ordering notes

 1. **Get an Actions runner registered** — `runner-token-rotate.sh`, then clear the PVC so
   the runner actually re-registers, then confirm it shows `online` in the API. Do not
@@ -54,6 +112,29 @@ commands.
 4. Add the human reviewer as a repo collaborator, then raise `required_approvals` to 1.

 > **Do not do step 3 before step 1.** A required status check with no runner leaves every
-> PR permanently pending — including the PR that would fix it. The same applies to
-> `gate.yml`'s `conformance` job: it is `if: false` and never runs, yet still publishes a
-> `gate / conformance (pull_request)` context. Never list that one as required.
+> PR permanently pending — including the PR that would fix it. That half is measured: PR #1
+> sat `pending` for 30.6 minutes across 122 polls with no runner registered, and never
+> reported.
+
+### Measured, once the runner came up (2026-09-29)
+
+The first CI jobs this repo ever ran, on PR #1 and PR #2:
+
+| Context | Result | Time |
+|---|---|---|
+| `gate / gate (pull_request)` | success | 1m58s (PR #1), 1m34s (PR #2) |
+| `gate / gate (push)` | success | 1m32s (PR #2 only) |
+| `gate / conformance (pull_request)` | **success — "Has been skipped"** | — |
+
+**Require `gate / gate (pull_request)`. Do NOT require either `conformance` context —
+but not for the reason an earlier draft of this file gave.** A skipped Gitea Actions job
+does not hang a status check: Gitea reports it as **`success`** with the description
+"Has been skipped". So requiring it would not deadlock anything — it would pass
+*vacuously*, every time, forever, while nothing ran. Under HARD RULE 4 that is worse than
+a deadlock: a deadlock is loud, a permanently-green check that verifies nothing is
+exactly the fail-open shape this repo keeps finding.
+
+The `(push)` contexts appear only for branches matching `gate.yml`'s push trigger
+(`feature/**`, `fix/**`, `docs/**`, `chore/**`). `feat/**` is **not** in that list, which
+is why PR #1's `feat/` branch produced only `pull_request` contexts. Require only the
+`pull_request` context, or PRs from non-matching prefixes will never satisfy the rule.
@@ -0,0 +1,36 @@
+#!/usr/bin/env bash
+# Stop repository admins from overriding the merge requirements on `main`.
+#
+# WHY. Gitea lets repo admins override branch-protection merge requirements unless
+# `block_admin_merge_override` is set. It was FALSE, and jakman85 holds `owner` -- so
+# required_approvals and the required status check were both overridable by the one account
+# that actually merges. Evidence: PRs #5-#9 all merged with ZERO approvals recorded against
+# `required_approvals: 1`.
+#
+# NOTE ON FIELD NAMES: this Gitea (1.24.3) has NO `apply_to_admins` field. Querying it with jq
+# returns null, which reads like "set to false" but actually means "key absent". Enumerate the
+# real keys (`to_entries`) rather than probing names you assume exist.
+#
+# Recoverable: an owner can PATCH this back to false if it gets in the way.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
+api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
+B="https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main"
+
+echo "==> before: block_admin_merge_override = $(api "$B" | jq -r .block_admin_merge_override)"
+api -X PATCH -H 'Content-Type: application/json' \
+  -d '{"block_admin_merge_override": true}' "$B" >/dev/null
+AFTER=$(api "$B" | jq -r .block_admin_merge_override)
+echo "==> after:  block_admin_merge_override = $AFTER"
+[ "$AFTER" = true ] || { echo "PATCH did not take -- do not record this as enforced." >&2; exit 1; }
+
+api "$B" | jq -r '{enable_push,enable_status_check,status_check_contexts,required_approvals,block_admin_merge_override}'
+cat <<'NOTE'
+
+NOW PROVE IT -- the setting is not the evidence. On the next PR:
+  * let the gate go green, then try to merge WITHOUT approving it.
+  * Gitea must REFUSE. If it merges, this still is not doing its job.
+Do not record approvals as enforced until you have seen a merge refused.
+NOTE
@@ -0,0 +1,60 @@
+#!/usr/bin/env bash
+# Items 1 + 2: make the gate a REQUIRED status check and require one approval.
+#
+# RUN THIS LAST. Prerequisites, in order:
+#   a) PR #7 (dind native sidecar) is MERGED, and
+#   b) you have SEEN a real workflow run go green on the converted pod.
+# The runner Deployment is the merge gate for the whole repo. Turning on a required check
+# while the runner is broken means no PR can merge -- including the revert. (Recoverable:
+# an owner can set enable_status_check back to false. But do not rely on that.)
+#
+# Requires exactly ONE context: `gate / gate (pull_request)`.
+#   * NOT `gate / conformance (*)` -- that job is `if: false`, and a skipped Gitea job reports
+#     SUCCESS ("Has been skipped"). Requiring it would pass VACUOUSLY forever while nothing
+#     ran, which under HARD RULE 4 is worse than a deadlock: a deadlock is loud.
+# block_admin_merge_override: true is LOAD-BEARING and was missing on the first pass. Gitea
+# lets repo admins override merge requirements without it, so required_approvals and the
+# required status check are both bypassable by exactly the account that does the merging.
+# Measured: PRs #5-#9 merged with ZERO approvals recorded against required_approvals: 1.
+#
+# FIELD NAMES: this Gitea (1.24.3) has NO `apply_to_admins`. Asking jq for it returns null,
+# which reads like "false" but means "key absent" -- enumerate real keys with `to_entries`
+# instead of probing names you assume exist. That mistake cost a wasted PATCH.
+#
+#   * NOT the `(push)` variant -- after PR #6 there are no per-branch push runs at all, so a
+#     PR would never produce that context.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
+api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
+
+echo "==> updating branch protection on main"
+jq -n '{
+  enable_push: false,
+  enable_status_check: true,
+  status_check_contexts: ["gate / gate (pull_request)"],
+  required_approvals: 1,
+  block_on_rejected_reviews: true,
+  dismiss_stale_approvals: true,
+  block_on_outdated_branch: false,
+  block_admin_merge_override: true
+}' | api -X PATCH -H 'Content-Type: application/json' --data @- \
+  "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main" >/dev/null
+
+echo "==> verifying"
+api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \
+ | jq -r '{protected,user_can_push,enable_status_check,status_check_contexts,required_approvals}'
+
+cat <<'NOTE'
+
+NOTE ON required_approvals: 1 -- Gitea will not let an author approve their own PR.
+  * Agent-opened PRs (author gitea-admin) are approved by jakman85. Fine.
+  * PRs YOU open as jakman85 need gitea-admin to approve. If that is awkward, either open
+    agent work as gitea-admin and review as jakman85, or set required_approvals back to 0
+    and rely on the status check alone.
+
+Then prove it, rather than trusting the API:
+  git commit --allow-empty -m probe && git push origin main   # EXPECT: rejected
+  git reset --hard HEAD~1
+NOTE
@@ -0,0 +1,83 @@
+#!/usr/bin/env bash
+# verify-then-enforce.sh — run this AFTER you have merged PRs 8, 5, 6, 7 in that order.
+#
+# It does NOT merge anything. Merging is your reviewed act (CLAUDE.md: a human approves and
+# merges). This script does the mechanical half: prove the converted runner actually works,
+# and only then turn on enforcement.
+#
+# WHY THIS ORDER. The act-runner Deployment IS the merge gate for this repository. Enabling a
+# required status check while the runner is broken means NO PR can merge -- including the
+# revert. So PR #7 lands while the gate is still advisory, is PROVEN green on a real run, and
+# only then does enforcement go on.
+#
+# Aborts at the first checkpoint that does not pass. Safe to re-run.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
+REPO_ROOT=/var/home/keith/source/home-network-k3s-impl
+D="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+cd "$REPO_ROOT"
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
+api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
+A="https://${HOST}/api/v1/repos/${OWNER}/${REPO}"
+say() { printf '\n\033[1m==> %s\033[0m\n' "$1"; }
+die() { printf '\n\033[31mABORT: %s\033[0m\n' "$1" >&2; exit 1; }
+
+say "PRECHECK — all four PRs must be merged first"
+for n in 8 5 6 7; do
+  m=$(api "$A/pulls/$n" | jq -r .merged)
+  printf '  PR #%s merged=%s\n' "$n" "$m"
+  [ "$m" = true ] || die "PR #$n is not merged yet. Merge 8, 5, 6, 7 in that order, then re-run."
+done
+
+say "STEP 1 — force Flux to apply the sidecar now, not on the 10m interval"
+./infra/devcontainer/dev.sh run bash -c \
+  'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-1 --with-source'
+
+say "STEP 2 — CHECKPOINT: the converted pod must come up healthy"
+./infra/devcontainer/dev.sh run bash -c '
+  kubectl --context k3s-prod -n gitea-actions rollout status deploy/act-runner --timeout=180s
+  echo "--- sidecar (initContainers) ---"
+  kubectl --context k3s-prod -n gitea-actions get deploy act-runner \
+    -o jsonpath="{range .spec.template.spec.initContainers[*]}{.name} restartPolicy={.restartPolicy}{\"\n\"}{end}"
+  kubectl --context k3s-prod -n gitea-actions get pod -l app.kubernetes.io/name=act-runner' \
+  || die "converted act-runner pod is not healthy -- REVERT PR #7 before anything else"
+
+say "STEP 3 — CHECKPOINT: a runner must be ONLINE again"
+n=0
+for i in $(seq 1 40); do
+  n=$(api "https://${HOST}/api/v1/admin/actions/runners" | jq -r '[(.runners//[])[]|select(.status=="online")]|length')
+  printf '\r  online runners: %s  (%ss)   ' "$n" "$((i*5))"
+  [ "${n:-0}" -ge 1 ] && { echo; break; }
+  sleep 5
+done
+[ "${n:-0}" -ge 1 ] || die "no runner online after the conversion -- REVERT PR #7"
+
+say "STEP 4 — CHECKPOINT: a REAL workflow must go green ON THE CONVERTED POD"
+MAIN_SHA=$(api "$A/branches/main" | jq -r .commit.id)
+echo "  watching the post-merge push run on main (${MAIN_SHA:0:8})..."
+st=""
+for i in $(seq 1 60); do
+  st=$(api "$A/commits/$MAIN_SHA/status" | jq -r .state)
+  printf '\r  main gate: %-8s (%ss)   ' "$st" "$((i*10))"
+  case "$st" in success) echo; break;; failure|error) die "gate FAILED on the converted pod -- REVERT PR #7";; esac
+  sleep 10
+done
+[ "$st" = success ] || die "gate never completed on the converted pod -- REVERT PR #7; do NOT enable enforcement"
+echo "  the converted sidecar ran a real job green. Enforcement is now safe."
+
+say "STEP 5 — require the gate + one approval"
+"$D/require-gate.sh"
+
+say "STEP 6 — prove a direct push is rejected (do not trust the API's word)"
+git fetch -q origin && git checkout -q main && git reset -q --hard origin/main
+git commit -q --allow-empty -m 'protection probe'
+if git push origin main 2>&1 | grep -q 'protected branch'; then
+  echo "  CONFIRMED: push to main rejected by the pre-receive hook."
+else
+  echo "  WARNING: the push was NOT rejected -- protection is not doing what it claims."
+fi
+git reset -q --hard origin/main
+
+say "FINAL STATE"
+api "$A/branches/main" | jq -r '{protected,user_can_push,enable_status_check,status_check_contexts,required_approvals}'
@@ -0,0 +1,94 @@
+#!/usr/bin/env bash
+# plex-claim-ship.sh — run this IMMEDIATELY AFTER editing the SOPS secret with a fresh
+# plex.tv claim token. It drives the whole CI/CD loop end to end and prints a running
+# stopwatch so you can see the budget being spent.
+#
+#   branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verify
+#
+# It never sees, prints, or handles the token: the file is already encrypted by sops.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev
+OWNER=Home-Network
+REPO=home-network-k3s-impl
+SECRET=apps/wave-0/plex-claim-secret.sops.yaml
+T0=$(date +%s)
+el() { printf '\n[T+%02d:%02d] %s\n' $(( ($(date +%s)-T0)/60 )) $(( ($(date +%s)-T0)%60 )) "$1"; }
+
+cd "$(git rev-parse --show-toplevel)"
+
+# --- preflight: refuse to ship a file that is not what we think it is -------------
+git diff --quiet -- "$SECRET" && { echo "ERROR: $SECRET is unchanged. Edit it first:"; \
+  echo "  infra/devcontainer/dev.sh run sops $SECRET"; exit 1; }
+grep -q 'PLEX_CLAIM: ENC\[' "$SECRET" || { echo "ERROR: $SECRET is not encrypted. REFUSING."; exit 1; }
+[ -z "$(git status --porcelain | grep -v " $SECRET\$")" ] || echo "NOTE: other files are dirty; only $SECRET will be committed."
+
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
+api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
+
+# Branch prefix is now cosmetic. It used to matter: gate.yml's push trigger covered
+# feature/** fix/** docs/** chore/** but not feat/**, and with act_runner at capacity 1 a
+# matching prefix fired a redundant run that serialised ahead of the PR run. Those
+# per-branch push triggers were deleted, so no prefix produces a duplicate run any more.
+BRANCH="feat/plex-claim-token-$(date +%H%M%S)"
+
+el "committing on $BRANCH"
+git checkout -q -b "$BRANCH"
+git add "$SECRET"
+git commit -q -m 'chore(plex): fresh plex.tv claim token
+
+Injected inside the plex.tv claim window per runbooks/plex-claim.md.'
+SHA=$(git rev-parse HEAD)
+
+el "pushing"
+git push -q -u origin "$BRANCH"
+
+el "opening PR"
+PR=$(jq -n --arg h "$BRANCH" --arg b main --arg t 'chore(plex): fresh plex.tv claim token' \
+       '{head:$h,base:$b,title:$t,body:"Token injection inside the claim window. Wiring already reviewed in #1."}' \
+     | api -X POST -H 'Content-Type: application/json' --data @- \
+         "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls" | jq -r '.number')
+[ -n "$PR" ] && [ "$PR" != "null" ] || { echo "PR creation failed"; exit 1; }
+echo "  PR #$PR"
+
+el "waiting for the gate (expect ~1m40s)"
+for i in $(seq 1 60); do
+  st=$(api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/commits/${SHA}/status" | jq -r '.state')
+  printf '\r  gate=%s  [T+%ss]   ' "$st" "$(( $(date +%s)-T0 ))"
+  case "$st" in success) echo; break;; failure|error) echo; echo "GATE FAILED — stopping."; exit 1;; esac
+  sleep 5
+done
+[ "$st" = "success" ] || { echo; echo "gate did not finish in time"; exit 1; }
+
+el "merging PR #$PR"
+api -X POST -H 'Content-Type: application/json' -d '{"Do":"merge"}' \
+  "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls/${PR}/merge" >/dev/null
+git checkout -q main && git pull -q --ff-only origin main
+
+el "forcing the Flux reconcile (--with-source, or it applies the OLD revision)"
+./infra/devcontainer/dev.sh run bash -c \
+  'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-0 --with-source'
+
+# MANDATORY here. Only the Secret ciphertext changed, so the Deployment pod template is
+# byte-identical and NOTHING restarts on its own. PLEX_CLAIM is read once, at container
+# start. Skipping this is failure mode #2 in runbooks/plex-claim.md.
+el "rolling the pod (mandatory — a changed Secret restarts nothing)"
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex rollout restart deployment/plex && \
+   kubectl --context k3s-prod -n plex rollout status deployment/plex --timeout=120s'
+
+el "verdict"
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex logs deploy/plex --since=3m | grep -i -e claim -e token || true'
+cat <<'NOTE'
+
+  "Attempting to obtain server token..." + "Token obtained successfully"  -> CLAIMED
+  "Attempting..." with no success line                                    -> token expired; retry, nothing is broken
+  neither line                                                            -> already claimed, or env did not reach the container
+
+Confirm it persisted (prints the attribute NAME only, never its value):
+
+  infra/devcontainer/dev.sh run bash -c \
+    'kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c \
+       "grep -o PlexOnlineToken= \"/config/Library/Application Support/Plex Media Server/Preferences.xml\" | head -1"'
+NOTE
@@ -6,7 +6,7 @@ Every alert rule in this platform carries an annotation of the form

    runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"

-46 of them, over the 35 names below. Until this directory existed, that path returned **HTTP 404** on the
+46 of them, over 35 names. Until this directory existed, that path returned **HTTP 404** on the
 live cluster: every alert that reached a human handed them a dead link at the exact moment they needed
 help. These pages are what makes the link resolve.

@@ -33,7 +33,8 @@ negative is how a CLI quirk becomes real data loss. Write `backups.velero.io`,
 `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`,
 `volumes.longhorn.io`, `nodes.longhorn.io`.

-`git push origin main` **is** the deploy; there is no other apply step. Durable changes go through git.
+Merging a PR into `main` **is** the deploy; there is no other apply step. `main` is protected, so a
+direct push is rejected — durable changes go through a branch, a PR, and a green gate.

 ## Alerts you should not trust the silence of (measured 2026-09-13)

@@ -108,6 +109,13 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
 - **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
  every green signal missed.

+### Operator procedures (no alert links here)
+- **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only
+  applies after a `/config` wipe. The claim token is short-lived — read the countdown on
+  plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
+  token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
+  Includes why a changed Secret does not restart anything, and why an expired token is harmless.
+
 ### Virtualisation
 - **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.
 - **`kubevirt-launch-failed`** — a VMI reached the Failed phase. The three k3s-specific launch traps.
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -78,8 +78,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -55,8 +55,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -63,8 +63,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -87,8 +87,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -69,8 +69,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -71,8 +71,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -68,8 +68,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -7,7 +7,7 @@
 ## What this means here

 Something in the GitOps chain is stuck at Ready=False. On this cluster that is more consequential than
-it sounds, because **`git push origin main` IS the deploy** — there is no other apply step. A stuck
+it sounds, because **merging a PR into `main` IS the deploy** — there is no other apply step. A stuck
 Kustomization means git and the cluster have diverged and nothing will close the gap.

 The chain has **14 tiers**: `flux-system` → `cilium` → `platform` → `platform-config` →
@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -58,8 +58,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -87,8 +87,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -67,8 +67,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -59,8 +59,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -57,8 +57,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -81,6 +81,11 @@ configMapGenerator:
      - node-io=node-io.md
      - node-memory=node-memory.md
      - node-memory-swap=node-memory-swap.md
+      # Operator procedure, not an alert page (no alert links to it — the
+      # alert-runbooks-resolvable check treats that as advisory). It is served
+      # because the plex.tv claim window is short and is not a thing to go looking
+      # for in git: /runbooks/plex-claim has to resolve the moment it is needed.
+      - plex-claim=plex-claim.md
      - pvc-capacity=pvc-capacity.md
      - targets-down=targets-down.md
      - update-rollback=update-rollback.md
@@ -83,8 +83,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -74,8 +74,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -75,8 +75,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -52,8 +52,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -49,8 +49,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -56,8 +56,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -56,8 +56,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -227,8 +227,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -52,8 +52,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -0,0 +1,189 @@
+# plex-claim — claiming the Plex Media Server inside the token window
+
+**Alert:** none. This is an **operator procedure**, not an incident page — it is indexed here because
+it is the page you need at the moment you need it, with a stopwatch running.
+**Files:** `catalog/plex.yaml` (env `PLEX_CLAIM`), `apps/wave-0/plex-claim-secret.sops.yaml`
+(the SOPS Secret `plex-claim`, key `PLEX_CLAIM`), `apps/wave-0/plex.yaml` (the rendered golden).
+
+## What this means here
+
+Plex will not let you **create libraries** over the mounted bulk media share while the server is
+**unclaimed** — the 8.33 T NFS tree at `/media` is there, readable, and unusable. Claiming binds the
+server to a plex.tv account. It is a **one-shot** operation: the token is exchanged once for a durable
+`PlexOnlineToken` written into `Preferences.xml` on the `plex-config` Longhorn PVC, and from then on
+the `PLEX_CLAIM` env is **ignored on every start** (upstream gate:
+`[ ! -z "${PLEX_CLAIM}" ] && [ -z "${token}" ]` in `root/etc/cont-init.d/40-plex-first-run`).
+
+**A `https://plex.tv/claim` token is short-lived — minutes, not hours.** Read the countdown the
+claim page shows you; do not budget against a number written here. That is shorter than PR → gate →
+merge → Flux's 10-minute reconcile interval → pod roll if you let those run on their own timers,
+which is why this page exists: the loop is scripted and the reconcile is **forced** rather than
+waited for. Measured end to end on 2026-09-29: **~2m45s**.
+
+**An expired or wrong token does not break anything.** Verified against the image's own first-run
+script (and the copy inside `plexinc/pms-docker:latest`): a failed exchange gets a 403 with an empty
+body, `curl`'s status is never checked, the token `sed` yields empty, the `if` does not fire, there is
+no `else` and no `exit 1`, and the script (no `set -e`) exits 0. The pod comes up **unclaimed and
+serving**. So a burnt token costs you one more attempt, not an outage — and the repo deliberately
+ships a **placeholder** in that Secret for exactly this reason (the env `secretKeyRef` has no
+`optional`, so the Secret must always exist even when its value is useless).
+
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that
+hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and that default has already changed once (k3s-dev until dev
+was decommissioned on 2026-09-13, k3s-prod now). Do not run two `dev.sh` invocations at once.
+
+Mutating `kubectl` and `flux reconcile` are blocked for agents here. **This whole page is operator
+work** — the reconcile and the pod roll below are deliberate operator actions, and the durable change
+is still a git change.
+
+## First commands
+
+**Before spending a token, find out whether the server is already claimed.** This prints only the
+attribute NAME, never its value — do not widen it to print the token:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c \
+     "grep -o PlexOnlineToken= \"/config/Library/Application Support/Plex Media Server/Preferences.xml\" | head -1"'
+```
+
+`PlexOnlineToken=` printed ⇒ **already claimed**; stop, there is nothing to do and no token to burn.
+Empty output ⇒ unclaimed, continue.
+
+Is the workload healthy enough to claim at all, and is the wave reconciling?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex get deploy,pod,pvc'
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0'
+```
+
+## The procedure
+
+> **STATUS: this server was CLAIMED on 2026-09-29** and `PlexOnlineToken` is present in
+> `/config` on the `plex-config` PVC. The upstream exchange is gated on that pref being
+> *absent*, so `PLEX_CLAIM` is now a **no-op on every start** whatever it holds. You only need
+> this page again after a `/config` wipe or a rebuild onto an empty PVC.
+
+**On the clock.** The token is short-lived — **read the countdown that plex.tv/claim shows you**
+rather than trusting a number written here. (Earlier revisions of this page asserted a "~4-minute
+TTL" in four places with no citation; upstream `pms-docker` documents no expiry at all. The page
+is the only authority, and it is on screen at T+0.)
+
+**A burnt token costs nothing.** An expired token gets a 403 with an empty body, `curl`'s status is
+never checked, the `if` does not fire, there is no `else` and no `exit 1`, and the script (no
+`set -e`) exits 0. The pod comes up unclaimed and serving. So retry freely.
+
+### One command
+
+1. **T+0 — get the token.** Sign in to plex.tv as the account that should OWN this server, open
+   <https://plex.tv/claim>, copy the `claim-…` string.
+
+2. **Put it in the Secret** — in place, encrypted, never touching a plaintext file:
+
+   ```
+   infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
+   ```
+
+3. **Ship it.** This drives the whole delivery loop and prints a running stopwatch:
+
+   ```
+   ./infra/plex/claim-ship.sh
+   ```
+
+   branch → commit → push → PR → gate → merge → `flux reconcile --with-source` → `rollout restart`
+   → verdict. It refuses to run if the file is unchanged or not encrypted.
+
+**Measured end to end, 2026-09-29: ~2m45s.** Gate ~1m40s, merge + reconcile ~25s, pod roll and
+container start ~30s.
+
+### Why it is a script and not six pasted commands
+
+`main` is a **protected branch** — the direct push this page used to instruct is now rejected — so
+the deploy is a PR merged after a green gate. Three of those steps are easy to get wrong under time
+pressure, and two are silent when skipped:
+
+- **`--with-source` is load-bearing.** Without it the Kustomization applies the previously-fetched
+  revision, not your commit.
+- **The pod roll is mandatory.** Only the Secret's ciphertext changes, so the Deployment's pod
+  template is byte-identical and *nothing restarts on its own*. `PLEX_CLAIM` is read once, at
+  container start. Skip this and the new Secret sits in the cluster while the running pod keeps the
+  old value forever.
+- **The branch is named `feat/…` on purpose.** `act_runner` capacity is 1, and `gate.yml`'s push
+  trigger covers `feature/** fix/** docs/** chore/**` but *not* `feat/**`. A `chore/` branch fires a
+  second redundant run that serialises ahead of the PR run and roughly doubles the wall clock.
+
+### Reading the verdict
+
+* `Attempting to obtain server token from claim token` **and** `Token obtained successfully`
+  ⇒ **claimed.** Re-run the First-commands `PlexOnlineToken=` check to confirm it persisted, then
+  create the libraries over `/media/movies`, `/media/tv`, `/media/photos`.
+* `Attempting…` with **no** success line ⇒ the token was expired or invalid. Nothing is broken; go
+  back to step 1 with a new one.
+* **Neither line** ⇒ the exchange was skipped: `/config` already had a `PlexOnlineToken` (already
+  claimed — the expected case now), or the env did not reach the container. Check by NAME only:
+
+  ```
+  infra/devcontainer/dev.sh run bash -c \
+    'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
+       -o jsonpath="{.items[0].spec.containers[0].env[*].name}"'
+  ```
+
+## Likely causes, most likely first
+
+When the claim does not take, in order of how often it is the answer here:
+
+1. **The token expired** — you exceeded the countdown, usually by reviewing the PR instead of
+   merging it the moment the gate went green. Symptom: the "Attempting…" line with no success line.
+   Remedy: a new token, and let `claim-ship.sh` drive the loop rather than pasting steps.
+2. **The pod was never restarted** (step 5 skipped). Symptom: no claim log lines at all from a pod
+   whose `AGE` predates your push. The Secret is correct in git and in the cluster and the running
+   container still holds the placeholder.
+3. **Flux had not applied the commit yet** when you rolled the pod — step 5 run before step 4
+   finished, so the new pod read the OLD Secret. Check
+   `kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0 -o jsonpath='{.status.lastAppliedRevision}'`
+   against your commit, then roll again.
+4. **Already claimed** — the exchange is gated on `PlexOnlineToken` being absent from `/config`, so a
+   server claimed earlier ignores every later token silently. This is the benign case.
+5. **`plex.tv` unreachable.** The upstream `curl` has no `--max-time`/`--connect-timeout`, so a stalled
+   connection can leave the container sitting in cont-init with **nothing listening on 32400** — the
+   one genuine hang risk in this path. Symptom: pod Running but never Ready, log stops after
+   "Attempting…". Check egress (`plex` is `egress: internet`) rather than re-rolling repeatedly.
+6. **Wrong plex.tv account.** The claim succeeds and binds the server to whoever was signed in at
+   step 1. Unclaiming is a plex.tv-side operation on that account — see "escalate or stop".
+
+## When to escalate or stop
+
+- **Stop after two failed token attempts** and fix the timing, not the token. The third rushed attempt
+  is how people start editing live Secrets by hand, which leaves the cluster holding a value that is
+  in no git commit — precisely the drift `qbittorrent-bootstrap` had to be recovered from.
+- **Never `kubectl edit`/`patch` the `plex-claim` Secret in the cluster, and never
+  `kubectl create secret` it.** Everything durable goes through the SOPS file; a live-only Secret is
+  reverted (or adopted with the wrong value) on the next reconcile.
+- **Never `sops -d` this file into a terminal, a log, or a shared screen**, and never widen the
+  `PlexOnlineToken=` check into one that prints the value. The token is a plex.tv account credential.
+- **If the server got claimed to the wrong account, stop and get the operator.** Reversing it means
+  removing the server from that account on plex.tv and clearing `PlexOnlineToken` from
+  `Preferences.xml` on the PVC — a destructive edit of app state on the replicated tier, not a git
+  change.
+- **Do not delete the `plex-config` PVC to "start clean".** It holds the library database, metadata
+  and watch state (`policy: daily-offcluster`), and it is expensive to rebuild. The claim is a
+  two-line pref, not a reason to lose the DB.
+- After a successful claim, **leave the expired token in the SOPS file**. It is inert (the exchange is
+  skipped once claimed) and the Secret must keep existing, because the env `secretKeyRef` has no
+  `optional` — deleting it would put the pod into `CreateContainerConfigError`.
+
+---
+Source of truth: `runbooks/plex-claim.md`. Index: `/runbooks/`
+Upstream behaviour cited here is from `plexinc/pms-docker` branch `master`,
+`root/etc/cont-init.d/40-plex-first-run` (s6-overlay v2), cross-checked against the same file inside
+the running `plexinc/pms-docker:latest` image.
@@ -58,8 +58,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -60,8 +60,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -54,8 +54,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -66,8 +66,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands