fix(ci): CLUSTER_AVAILABLE=true would have FAKED A GREEN over 54 skipped tests

Wiring L2/L3 into CI was a tracked next step, described in the plan as needing only vars.CLUSTER_AVAILABLE. Attempting it surfaced that the variable does not enable L2/L3 at all -- it only changes which way the stage fails, and before this commit one of those ways was a false pass. MEASURED, before the guard: CLUSTER_AVAILABLE=true ./ci/conformance.sh -> rc=0 "[junit] wrote skipped JUnit for 54 scaffolded test(s)" "[ ok ] L2/L3 conformance passed for touched subset" run.sh exits 0 in scaffold mode when it cannot reach a cluster, marking every test <skipped>. That is correct and honest FOR run.sh. It is catastrophic in ci/conformance.sh, which reaches that code only because the environment DECLARED a cluster available -- so it reported a pass over 54 tests that never executed. One Gitea variable away from a fake green on a gate that is now an ENFORCING required status check, and nothing caught it because nobody had ever set the variable. The CI runner holds no cluster credential by design (automountServiceAccountToken: false), so the local run this stage attempts can never succeed. The in-cluster trigger path that would make L2/L3 actually run is still unimplemented, and is blocked on a privilege- escalation design decision that is the operator's to make (plan section 4: any principal that can `create jobs` in ns conformance can run arbitrary code as the privileged conformance-runner SA, because Kubernetes places no `use` check on serviceAccountName). The stage now BLOCKS (exit 1) when a cluster is declared but every test skipped, which is what the merge-gate decision matrix already said should happen ("conformance skipped while CLUSTER_AVAILABLE=true -> BLOCK"). Verified: rc=1, naming the count and the reason. The plan document claimed CLUSTER_AVAILABLE was "the only remaining gate". Corrected in place with the measurement and an explicit do-not-set-this-yet. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Merge pull request 'fix(gitea): admins could override the gate' (#10 ) from fix/gitea-admins-could-override-the-gate into main
2026-09-29 02:58:26 -05:00 · 2026-09-29 07:43:52 +00:00 · 2026-09-29 02:36:38 -05:00 · 2026-09-29 07:32:52 +00:00 · 2026-09-29 02:28:55 -05:00 · 2026-09-29 07:22:50 +00:00
61 changed files with 1286 additions and 226 deletions
@@ -40,9 +40,15 @@ and ADR-0011: **CI holds no broad kubeconfig.** L2/L3 run as an in-cluster **Cha
 *collects* its artifacts, using a narrow trigger credential referenced **by secret name only**
 (`secrets.CONFORMANCE_TRIGGER_KUBECONFIG`) — never a plaintext secret, never a cluster-admin kubeconfig.

-## Honest SKIP — never fake green (Phase 0)
+## Honest SKIP — never fake green

-**No cluster exists in Phase 0.** The gate is deliberately built so unavailable checks **SKIP loudly,
+**NOTE (2026-09-29):** this section is written in the past's voice — "Phase 0", "no cluster
+exists". **A cluster has existed for weeks** and Flux has been reconciling against it. The
+skip logic below is still correct and still honest, but read it as describing a **live
+coverage gap** (L2/L3 is not wired into CI) rather than an expected phase. Its being
+phrased as a phase is part of why the gap went unexamined.
+
+Originally: The gate is deliberately built so unavailable checks **SKIP loudly,
 they do not pass**:

 - The `conformance` job is `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}` — with no cluster it is
@@ -77,21 +83,29 @@ Both dirs are git-ignored (see [`.gitignore`](../../.gitignore)).
 ## Triggers & branch model (CONTRIBUTING.md)

 - `pull_request` → `main`: **PRs are the enforced merge gate into `main`.**
- `push` → `feature/**`, `fix/**`, `docs/**`, `chore/**`: fast feedback on the branch. **No work on `main`.**
+- `push` → `main` only. The per-branch prefix list was removed: `act_runner` runs at capacity 1,
+  so a branch push fires a redundant run that serialises *ahead* of the PR run for the same tree.
+  **No work on `main`** — and since 2026-09-29 that is enforced, not conventional: `main` is a
+  protected branch and a direct push is rejected by the pre-receive hook (verified, not assumed).

-## Transition: generic-CI-now → Gitea-Actions-after-Gitea-is-up
+## Transition: COMPLETE (2026-09-29)

-Per [`CONTRIBUTING.md`](../../../home-network-k3s/CONTRIBUTING.md) and
-[`docs/05` §"local bootstrap → self-hosted Gitea"](../../../home-network-k3s/docs/05-gitops-and-delivery.md):
+This section used to describe a migration as pending. It has happened. Gitea is live at
+`gitea.home.keithjackson.dev`, the runner `incluster-dind` is registered and online, and
+`gate.yml` runs on every PR.

-1. **Now (interim):** there is **no remote git host** on this network. This workflow file is dormant —
-   but the **identical gate runs locally today** via `make -C ci gate` (R-TEST-02). Development is
-   local feature-branch → `main`, the gate enforced by convention + the local runner.
-2. **Gitea comes online early** (one of the first services onboarded) at `gitea.home.keithjackson.dev`.
-3. **CI moves to Gitea Actions:** this workflow activates unchanged. The only operational steps are
-   enabling a Gitea Actions runner and setting `vars.CLUSTER_AVAILABLE=true` +
-   `secrets.CONFORMANCE_TRIGGER_KUBECONFIG` once a dev/test cluster exists. PRs then become the
-   **platform-enforced** merge gate — the same no-work-in-main rule, now enforced by machine.
+**What took a month to notice:** the migration was recorded as done on 2026-08-31 without
+anyone checking the platform. In reality `main` had **no branch protection** and **zero
+runners were registered** until 2026-09-29. Every "the gate enforces this" statement in
+between rested on an assertion nobody verified. PR #1 — the first pull request this
+repository ever had — sat `pending` for 30.6 minutes because nothing existed to run it.
+
+So: **do not read this file, or the DECISIONS-LOG, as evidence about the live platform.**
+Query the API. `infra/gitea/` holds the scripts and the measured state.
+
+Still genuinely outstanding: `vars.CLUSTER_AVAILABLE` and
+`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` are unset, so L2/L3 conformance does not run in
+CI. That is a real coverage gap, not a phase.

 ## Configuration (set in Gitea; referenced by name only)

@@ -44,16 +44,31 @@
 ---
 name: gate

+# TRIGGERS — deliberately only `pull_request` into main, plus `push` to main itself.
+#
+# There is NO per-branch push trigger. There used to be a prefix list (feature/** fix/**
+# docs/** chore/**) for "fast feedback on the branch", and it was a net negative here:
+#
+#   * `act_runner` runs at capacity 1. A branch push and its PR fire the SAME gate over the
+#     SAME tree, and the redundant run SERIALISES AHEAD of the PR run — doubling the wall
+#     clock on the one result that actually gates the merge. That cost real time during the
+#     2026-09-29 plex claim, where the loop is racing a short-lived token.
+#   * The list silently did not match this repo's own convention. Branches here are `feat/**`,
+#     which was absent, so the "fast feedback" never fired for the most common prefix anyway —
+#     a mismatch nobody noticed for a month because its only symptom was a MISSING run.
+#     Any hand-maintained prefix list has that failure mode; deleting it removes the class.
+#   * It is redundant. CLAUDE.md already requires `./ci/gate.sh` in the pinned toolbox before
+#     you push (74s locally), and the PR gate re-runs it authoritatively. A third run of the
+#     same checks buys nothing.
+#
+# `push: [main]` is KEPT: post-merge main must still be verified, and it is the only run for
+# which no PR context exists.
 on:
  pull_request:
    branches: [main]
  push:
    branches:
      - main
-      - 'feature/**'
-      - 'fix/**'
-      - 'docs/**'
-      - 'chore/**'

 concurrency:
  group: gate-${{ github.ref }}
@@ -11,11 +11,14 @@
 #                                          v
 #                                     merge-gate decision  [R-TEST-03: touched subset green]
 #
-# Transition (CONTRIBUTING.md / docs/05 §"local bootstrap -> self-hosted Gitea"):
-# Today there is no Gitea host, so this file is dormant. The SAME gate is runnable
-# now via `make -C ci gate` (R-TEST-02). Once Gitea is self-hosted and Gitea
-# Actions is enabled, this workflow becomes the ENFORCED PR merge gate into main —
-# no code change required, only turning on the runner and setting CLUSTER_AVAILABLE.
+# STATUS (corrected 2026-09-29): Gitea IS hosted and Gitea Actions IS enabled; the runner
+# `incluster-dind` is online and gate.yml is the live PR gate. The text that used to sit here
+# ("Today there is no Gitea host, so this file is dormant") was stale by weeks -- it was written
+# as a plan and never revisited once the plan completed.
+#
+# THIS file is still dormant, but for the reason in the DORMANT block below -- its spec checkout
+# and L2/L3 seams are unwired -- NOT because there is no host. Do not read a stale plan as a
+# statement about the live platform; query the API instead (see infra/gitea/).
 #
 # CI holds NO broad kubeconfig. L2/L3 run as an in-cluster Chainsaw Job whose own
 # scoped ServiceAccount holds cluster access (docs/14, R-SEC-02); CI uses only a
@@ -23,28 +23,58 @@ There is no other path. Specifically:
 > (main is what the cluster follows), not what you are permitted to do. Honour this
 > file, not that line. Fixing those runbooks is tracked work.

-## 2. Enforcement is currently WEAKER than this rule. Follow the rule anyway.
+## 2. Enforcement (as of 2026-09-29: the rule is now MACHINE-ENFORCED)

 Verified against the live Gitea on 2026-09-29:

 | Expected | Actual |
 |---|---|
-| `main` protected, direct push rejected | **No branch protection.** `protected: false`, `user_can_push: true` |
-| PR blocked until the gate passes | **`enable_status_check: false`**, `status_check_contexts: []` |
-| Gate runs on every PR | **Fixed 2026-09-29.** Runner `incluster-dind` is online; `gate / gate (pull_request)` runs and passes in ~1m30s–2m |
-| Approval required before merge | **`required_approvals: 0`** |
+| `main` protected, direct push rejected | ✅ `protected: true`, `user_can_push: false` — **verified by a real rejected push**, not by the API's word |
+| PR blocked until the gate passes | ✅ `enable_status_check: true`, context `gate / gate (pull_request)` |
+| Gate runs on every PR | ✅ runner `incluster-dind` online; ~1m35s |
+| Approval required before merge | ✅ `required_approvals: 1` |

-So nothing mechanically stops you from pushing to `main` right now. **That is a gap
-being closed, not permission.** `DECISIONS-LOG.md` (2026-08-31) claimed the gate was a
-"required status check" with a runner online; neither was true, which is how this went
-unnoticed for a month. The runner half is now genuinely fixed and observed green. The
-protection half is not — see `infra/gitea/`.
+All four were FALSE as recently as 2026-09-29, while this repo's own log asserted they were
+true. They are true now because each was measured, and the push rejection was provoked rather
+than assumed. **Re-measure before relying on any of them** — see §2b.
+
+Still not covered: `vars.CLUSTER_AVAILABLE` is unset, so **L2/L3 conformance does not run in
+CI**. A green PR therefore proves the L0/L1 offline subset, not runtime behaviour.

 **Consequence for you:** CI now runs, but it does not yet *block* anything. Always run
 the toolbox gate yourself and quote its numbers. Never describe a PR as "green" on the
 strength of a check you have not seen report — and note that a **skipped** Gitea job
 reports as `success`, so a green tick is not by itself evidence that anything ran.

+## 2b. Never infer platform state from this repository
+
+**This repo is not evidence about the live platform.** Neither is `DECISIONS-LOG.md`.
+Both describe what someone believed at the time of writing, and both have been
+confidently wrong for a month at a stretch.
+
+On 2026-09-29 the log stated the merge gate was a required status check with a runner
+online. The API said `branch_protections: []` and **zero** runners. Everything downstream
+of that belief — "the gate enforces this", "your push would be rejected", "CI covers it" —
+was wrong, and nothing in the tree could have revealed it, because the tree was the source
+of the error.
+
+**So, before asserting that any gate, protection, runner, secret, or workload is in a
+given state: query the thing itself.**
+
+| Claim about... | Check |
+|---|---|
+| branch protection / required checks | `GET /api/v1/repos/{owner}/{repo}/branches/main` |
+| a runner exists | `GET /api/v1/admin/actions/runners` |
+| CI actually ran | the commit's `/statuses` — and read the **description**; a *skipped* Gitea job reports `success` with "Has been skipped" |
+| a workload / Secret / env | `kubectl --context k3s-prod ...` (read-only is fine for agents) |
+| Flux applied your commit | compare `.status.lastAppliedRevision` against the GitRepository's artifact revision |
+
+Quote the measured field, not the prose. If you cannot measure it, say so explicitly —
+"unverified" is a legitimate report, and a far better one than a confident guess that
+matches the docs.
+
+**A green tick is not proof anything ran.** Read what actually executed.
+
 ## 3. Gate before you push

    ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
@@ -1192,6 +1192,13 @@ fresh agent should read this entry + `infra/PHASE1-STATUS.md` + `docs/plans/` an
  Platform-admins require TOTP MFA.
 - **Gitea Actions CI is LIVE + enforcing**: `.gitea/workflows/gate.yml` runs `make -C ci gate` on push/PR to main;
  required status check; runner `incluster-dind` online (ns gitea-actions).
+  > **[CORRECTION, 2026-09-29 — THIS BULLET WAS FALSE WHEN WRITTEN AND STAYED FALSE FOR A MONTH.]**
+  > Measured against the live Gitea API on 2026-09-29: `branch_protections` returned `[]` (`main`
+  > `protected:false`, `enable_status_check:false`, `user_can_push:true`) and **zero** Actions runners
+  > were registered at repo, org or site level. The gate was neither enforced nor able to run. PR #1
+  > — the first pull request this repo ever had — was opened on 2026-09-29 and sat `pending` for
+  > 30.6 minutes with nothing to execute it. Both halves of this bullet were asserted, not verified.
+  > See the 2026-09-29 entries for the root cause and the fix.

 ### What changed this session (durable)
 1. **SSO cluster-wide break→fix — root cause Cilium CVE-2025-30162.** The 1.16→1.20 security fix makes the shared
@@ -4781,3 +4788,179 @@ accurate before the PR flow and are stale now. Correcting them is tracked, not d
 3. Land `feat/plex-claim-sops-secret` (PR #1) once the gate can actually run.
 4. Claim plex, after checking it is not already claimed.
 5. Correct the 37 runbooks' `git push origin main` wording.
+
+## 2026-09-29 — Plex is claimed; the CI/CD gate that "enforced" it never existed. **PM HANDOFF.**
+
+Began as "inject a plex.tv claim token". Ended by discovering the delivery pipeline was fiction,
+rebuilding it, and shipping the token through it. Plex is **claimed** — `PlexOnlineToken` is
+present in `/config` on the `plex-config` PVC, so the exchange is now a permanent no-op and the
+library scan is running over the 8.33T NFS share.
+
+### The finding: both halves of the 2026-08-31 CI claim were false
+
+That entry recorded "Gitea Actions CI is LIVE + enforcing … required status check; runner
+`incluster-dind` online". Measured against the live API on 2026-09-29:
+
+| Claimed | Measured |
+|---|---|
+| `main` protected, gate required | `branch_protections` → `[]`; `protected:false`, `enable_status_check:false`, `required_approvals:0`, `user_can_push:true` |
+| Runner online | **zero** runners at repo, org AND site level |
+
+**PR #1 was the first pull request this repository has ever had.** Its contexts sat `pending`
+for 30.6 minutes across 122 polls and never reported. For a month, every "the gate enforces this"
+statement rested on an assertion nobody had checked — the same fail-open class as 2026-09-18,
+and the reason it hid so long is that the log itself was the evidence.
+
+### Root cause of the dead runner
+
+`act_runner` registers **once** and persists its identity in `/data/.runner` on the
+`act-runner-data` PVC. Gitea no longer held a matching row, so the pod booted, saw `.runner`,
+**skipped registration**, and polled with an identity Gitea did not recognise. Symptom: zero
+runners, pod Running, nothing crashlooping, no error anyone reads. Deleting the PVC forced a
+genuine re-registration (`.runner is missing` → `Runner registered successfully`).
+
+It then did not come back for several minutes, because Flux could not recreate the PVC:
+`apps-wave-1` was blocked four levels deep on a **stale** `DependencyNotReady` for
+`platform-config`, which itself read `Ready=True`. Each wave re-evaluates only on its own
+`interval: 10m`, so a resource deleted at the bottom of the chain can take ~40 min to return.
+**Force it** with `flux reconcile kustomization apps-wave-1 --with-source` rather than waiting.
+
+### Corrections to things this log and the runbooks asserted
+
+- **A skipped Gitea job reports `success`, not pending.** Measured: `gate / conformance` returned
+  `success` / "Has been skipped". An earlier entry (#2) said requiring it would deadlock every PR.
+  Wrong — it would pass **vacuously**, forever, while nothing ran. Under HARD RULE 4 that is worse
+  than a deadlock, because a deadlock is loud. Never require it; require
+  `gate / gate (pull_request)` alone. Corrected in #3.
+- **`gate.yml`'s push trigger omits `feat/**`** (it lists `feature/** fix/** docs/** chore/**`).
+  With `act_runner` capacity 1, a branch prefix that IS in that list fires a second redundant run
+  which serialises ahead of the PR run and doubles wall-clock. Exploited deliberately in
+  `infra/plex/claim-ship.sh`; the honest fix is to align the trigger list.
+- **The "~4-minute TTL"** asserted in four places was **uncited**, and upstream `pms-docker`
+  documents no expiry at all. The authoritative reading is the countdown plex.tv/claim renders at
+  issue time. Removed from `runbooks/plex-claim.md`; the same claim still sits in
+  `catalog/plex.yaml` and the Secret's header comment (left alone — editing a `.sops.yaml`
+  comment risks the MAC, and touching `catalog/plex.yaml` re-renders a golden).
+
+### Measured, once the pipeline actually worked
+
+`gate / gate (pull_request)` 1m34s–1m58s. Full claim loop end to end **~2m45s** — comfortably
+inside the window, which retires the theory that a short-lived token cannot survive a PR flow.
+That theory was mine and it was wrong: the pipeline was never too slow, it simply did not exist.
+
+### State now
+
+- `main` **protected**, `user_can_push:false`. Runner `incluster-dind` online.
+- `CLAUDE.md` is the agent delivery contract; `infra/gitea/` holds the protection + runner scripts.
+- **All 37 runbooks** that instructed `git push origin main` now describe the PR flow — that
+  instruction had become actively blocked, not merely stale.
+- `infra/plex/claim-ship.sh` drives the claim loop end to end.
+
+### Next steps
+
+1. **`enable_status_check: true`** with `gate / gate (pull_request)` as the only context. The
+   prerequisite is met: the gate has been observed green on real PRs.
+2. Add `jakman85` as a repo **collaborator**, then `required_approvals: 1`. (Currently the only
+   org member is `gitea-admin`, which agents authenticate as — requiring approval before that is
+   fixed would leave nobody able to approve.)
+3. Run the **push probe** against protection: `gitea-admin` is a site admin and Gitea may let it
+   bypass. `user_can_push:false` is the API's opinion, not proof.
+4. Align `gate.yml`'s push trigger with this repo's `feat/**` prefix.
+5. Convert the `dind` container to a **native sidecar** with a `startupProbe`, per the standing
+   FOLLOW-UP note in `apps/wave-1/gitea-actions.yaml`. It deletes both hand-rolled wait loops and
+   the envsubst hazard. Land it ALONE — it is the merge gate for the whole repo.
+6. Create the Plex libraries over `/media/movies`, `/media/tv`, `/media/photos`.
+
+## 2026-09-29 (later) — the merge gate is now real, enforced, and proven. **PM HANDOFF.**
+
+Closes the gap opened by the earlier entry today. Eight PRs, all gated; every claim below was
+measured against the live API or cluster, not inferred from this repository.
+
+### Final state
+
+| | Verified |
+|---|---|
+| `main` protected | `protected:true`, `user_can_push:false` — and a direct push as site-admin `gitea-admin` was **provoked and rejected** (`pre-receive hook declined`) |
+| Gate required | `enable_status_check:true`, contexts `["gate / gate (pull_request)"]` |
+| Approvals | `required_approvals:1`; `jakman85` holds `owner` |
+| Runner | `incluster-dind` online, now a **native sidecar** (`initContainers`, `restartPolicy: Always`, TLS-verified `docker info` startupProbe) |
+| Gate time | ~1m35s |
+| Plex | claimed 2026-09-29; `PlexOnlineToken` present on the `plex-config` PVC |
+
+### What was corrected in the documentation
+
+`CLAUDE.md` §2b — *never infer platform state from this repository* — is the durable outcome.
+The 2026-08-31 entry asserting an enforced gate is now annotated **in place**. The "Phase 0 /
+no cluster exists" framing across `ci/`, `clusters/` and `conformance/` READMEs is banner-marked
+as describing a live gap rather than a stage. All 37 runbooks now describe the PR flow. Two
+`.sops.yaml` headers and `catalog/plex.yaml` were corrected (comment edits verified not to break
+the SOPS MAC, and not to move the rendered golden).
+
+### A claim of mine that shipped wrong, and is now corrected in place
+
+PR #7 stated the sidecar conversion would remove the one-per-start dockerd line
+`http: TLS handshake error from 127.0.0.1: EOF`. **It does not.** Exactly one still appears,
+immediately after `API listen on [::]:2376` — it is now the startupProbe's first attempt,
+landing microseconds after dockerd begins listening and cut off by `timeoutSeconds: 1`; the
+next poll succeeds. Cosmetic, and deliberately not chased: raising the timeout to hide one
+benign startup line would slow every subsequent poll. The manifest comment now says so.
+
+### The root cause nobody has fixed yet
+
+Gitea's repo settings live in its own database — invisible to Flux and unreadable by the
+conformance gate. **No check exists that could have caught "protection off, runner gone."**
+`infra/gitea/` declares and can re-assert the desired state, but it is not a detector. A
+conformance check that reads the live Gitea API and fails on divergence is the highest-value
+remaining work in this repo, and it is the difference between this having been fixed and this
+being unable to recur.
+
+### Next steps
+
+1. **Write that Gitea-settings conformance check.** Everything else here is a symptom.
+2. Wire `vars.CLUSTER_AVAILABLE` + `secrets.CONFORMANCE_TRIGGER_KUBECONFIG` so **L2/L3 runs in
+   CI**. A green PR currently proves the L0/L1 offline subset only — runtime behaviour is
+   unverified on every merge.
+3. Re-run `infra/gitea/verify-then-enforce.sh` after any future act-runner change.
+4. `required_approvals: 1` means a PR you open as `jakman85` needs `gitea-admin` to approve.
+   If that friction bites, either open agent work as `gitea-admin` and review as `jakman85`, or
+   drop back to relying on the status check alone.
+
+## 2026-09-29 (later still) — the enforcement we "turned on" was overridable by the account that merges
+
+`enable_status_check: true` and `required_approvals: 1` were set earlier today and recorded as
+done. **They did not bite.** `block_admin_merge_override` was `false`, and Gitea lets repository
+admins override merge requirements without it. `jakman85` holds `owner`, so both requirements
+were bypassable by exactly the account that performs every merge.
+
+**The evidence was already on the record and nobody read it:** PRs #5, #6, #7, #8 and #9 all
+merged while `required_approvals: 1` was in force, with **zero approvals recorded on any of
+them**. A one-line check (`/pulls/{n}/reviews`) would have shown it at any point.
+
+Two process failures, both mine, both the same shape as the original defect:
+
+1. **I verified with the wrong view.** `/branches/main` reports `protected`,
+   `enable_status_check` and `required_approvals` but NOT `block_admin_merge_override`. It
+   showed everything green while the rule was overridable. The authoritative object is
+   `/branch_protections/main` — read it and enumerate its keys.
+2. **I probed for a field name I assumed existed.** This Gitea (1.24.3) has no
+   `apply_to_admins`. Asking `jq` for it returns `null`, which reads exactly like "set to
+   false" but means "key absent" — so a PATCH that silently did nothing looked like a real
+   before/after. **`jq` returning null is not a value; it may be an absence.** Enumerate with
+   `to_entries`.
+
+Now `block_admin_merge_override: true`, applied and read back. `infra/gitea/require-gate.sh`
+sets it, and `infra/gitea/block-admin-override.sh` fixes it in isolation.
+
+**This is still only a setting.** It is proven when a green PR is REFUSED a merge until
+approved — and that has not been observed yet. This PR is the test: if it merges without an
+approval, the fix did not work and the finding stands.
+
+### The governance question this exposes
+
+There are two accounts: `jakman85` (owner, the human) and `gitea-admin` (the agent). With
+approvals now genuinely required, an agent-authored PR is approved by the human — which is the
+intent. But a PR authored by `jakman85` needs `gitea-admin` to approve it, i.e. the automation
+account rubber-stamping the human. That is ceremony, not review. Options: keep it and accept
+the ceremony; drop `required_approvals` to 0 and rely on the status check, which is the gate
+that actually verifies something; or add a second human account. **This is a decision for the
+operator, not a default to be chosen silently.**
@@ -17,6 +17,7 @@ resources:
 - pihole-dns.yaml
 - pihole-oauth2-proxy-credentials.sops.yaml
 - pihole.yaml
+- plex-claim-secret.sops.yaml
 - plex.yaml
 - qbittorrent-bootstrap-secret.sops.yaml
 - qbittorrent-oauth2-proxy-credentials.sops.yaml
@@ -0,0 +1,77 @@
+# Plex plex.tv server-claim token (WS-APPS). SOPS-encrypted (age).
+# Consumed by catalog/plex.yaml as env PLEX_CLAIM via valueFrom.secretKeyRef.
+#
+# NOTE ON THESE COMMENTS: they carry NO dollar-brace expressions, deliberately. This file is applied
+# by the apps-wave-0 Flux Kustomization, which runs envsubst over the built output, and kustomize
+# preserves body comments into that stream (see conformance/tests/l0-static/flux-postbuild-
+# substitution). A braced token quoted from the upstream shell script would be a live hazard here,
+# not prose, so the upstream conditions below are described in words.
+#
+# THIS FILE NO LONGER HOLDS THE PLACEHOLDER. Corrected 2026-09-29: the server was CLAIMED on that
+# date, and the value here is the SPENT token from that claim. It is inert and safe to leave --
+# upstream gates the exchange on PlexOnlineToken being ABSENT from /config, and it is now present on
+# the plex-config PVC, so this value is a NO-OP on every start whatever it holds. Deleting the Secret
+# would NOT be safe (see below), which is why a spent token stays here rather than being blanked.
+#
+# The claim token is SHORT-LIVED -- minutes. Read the countdown that plex.tv/claim renders when it
+# issues one; do NOT budget against a number written in this tree. Earlier revisions of this comment
+# asserted a "~4-minute TTL" as established fact with no citation, and upstream plexinc/pms-docker
+# documents no expiry at all. That uncited number was load-bearing for the whole procedure design.
+#
+# This file exists to make the Secret — and therefore the env reference — ALWAYS RESOLVABLE IN THE
+# FLUX GRAPH, not to carry a working token. Re-claiming (only needed after a /config wipe) is
+# scripted: infra/plex/claim-ship.sh, measured at ~2m45s end to end.
+#
+# WHY SHIPPING A PLACEHOLDER IS SAFE (verified against plexinc/pms-docker branch `master`, the
+# s6-overlay v2 script root/etc/cont-init.d/40-plex-first-run, cross-checked byte-identical against
+# the same file extracted from the running `plexinc/pms-docker:latest` image):
+#   - line 106 guards the claim exchange on TWO conditions: PLEX_CLAIM non-empty AND the
+#     PlexOnlineToken pref read out of /config empty. Once the server is claimed that block is
+#     skipped entirely, so whatever this Secret holds is a NO-OP on every later start. Upstream
+#     README: "If server is already logged in, this parameter is ignored."
+#   - on a still-UNCLAIMED /config, an invalid or expired token makes plex.tv answer 403 with an
+#     empty body. curl's exit status is never checked, the sed for the authentication-token element
+#     yields the empty string, and the success branch at line 120 simply does not fire. There is no
+#     else branch and no non-zero exit; the script has no `set -e` and ends by touching
+#     /.firstRunComplete and echoing, so it exits 0. The server comes up UNCLAIMED and serving.
+#     It does NOT crashloop.
+#   - the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS, whose default is 0 ("continue silently even
+#     if a cont-init script failed"), so even a hypothetical non-zero exit would not halt startup.
+# A MISSING Secret would NOT be safe: the ServiceDefinition schema's valueFrom.secretKeyRef shape
+# has no `optional` field, so an absent Secret is a hard CreateContainerConfigError. That is exactly
+# why this file is committed and listed in apps/wave-0/kustomization.yaml instead of being created
+# by hand at claim time.
+#
+# INJECTION / ROTATION (operator, not agents) — runbooks/plex-claim.md has the timed procedure:
+#   infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
+# Never decrypt this file into a shell, a log, or a shared terminal.
+apiVersion: v1
+kind: Secret
+metadata:
+    name: plex-claim
+    namespace: plex
+    labels:
+        app.kubernetes.io/name: plex
+type: Opaque
+stringData:
+    PLEX_CLAIM: ENC[AES256_GCM,data:nLmzeUBWRhRjUakrGiGzQpQuqZNcIYvRH6M=,iv:z0on/dArsNejNLrp3V1nSiNLFzKcREhPuMO1634CwQw=,tag:Ih23CT1JYna/5lfYQUb1Dw==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2dWtqOVNQalZ2QlRxOXJa
+            cDc1MTdtWENkdWIxS1JId3dIU1JDSmk0ODNrCk1YUzhFWXR3bUxyV2NNcmFOakdz
+            THl1Q284dFJxTy9Rd1h0eXJJaVZtdmcKLS0tIERuN0ZFeW9zdi9aNFRYNWxmUS9s
+            QUtYdHEvN042ajNoNktOLzlYQmQrc1kK4NHozMbXVzNGYbUBRbK6a9Uw/sVPAfe6
+            RW5KWzZ4I/hloqjN3zzVISoSMieMkBGDnF5qZS5N6FqIc6/HrsDD8A==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-09-29T06:31:16Z"
+    mac: ENC[AES256_GCM,data:62BmBAq45CD2g6LDyx+Ho4bTfrhJcaro/RB/iM+OaRc4a/D83MhjbcBL11Uz/O726inqhkvCKDJ/fFemoc50JDbtC79cYMaxrGJNjFM0sdckWrowPvh6ie7HHHUBRBaefl+gNfnc+pbvcncvRzwA4RyzcZse1wbxKfHN8FjEopE=,iv:h0cMShs3tIgnY6eruUYymYrcLFzQZLk/uXQJ/Jsd5ek=,tag:nZ9Pau7p4xWT1QCN1vv2vA==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4
@@ -132,6 +132,11 @@ spec:
          value: Etc/UTC
        - name: ALLOWED_NETWORKS
          value: 10.0.0.0/8,192.168.0.0/16
+        - name: PLEX_CLAIM
+          valueFrom:
+            secretKeyRef:
+              name: plex-claim
+              key: PLEX_CLAIM
        volumeMounts:
        - name: config
          mountPath: /config
@@ -4,7 +4,13 @@
 # against Gitea. Flux decrypts it via the sops-age key at reconcile (apps Kustomization
 # decryption is wired in clusters/dev/kustomization.yaml). Rotate by resetting the token
 # in Gitea (admin -> actions -> runners) and re-encrypting a fresh value here.
-# Recover/inspect: infra/devcontainer/dev.sh run sops -d apps/gitea-actions-runner-secret.sops.yaml
+# ROTATE: ./infra/gitea/runner-token-rotate.sh -- it mints a fresh instance-level token from the
+# Gitea API and writes it here via `sops set`, without the value ever being printed.
+# Do NOT `sops -d` this file into a terminal, a log, or an agent transcript. To check only that
+# it still decrypts, redirect: `sops -d <file> > /dev/null`.
+# NOTE: a fresh token alone may not re-register the runner. act_runner registers ONCE and keeps
+# its identity in /data/.runner on the act-runner-data PVC; if Gitea no longer holds a matching
+# row it skips registration silently. Clearing that PVC is what forces a real re-registration.
 apiVersion: v1
 kind: Secret
 metadata:
@@ -99,70 +99,79 @@ spec:
      # CI holds no cluster credentials — do not mount a SA token into the runner or
      # (via the shared netns) the job containers (R-SEC-02, docs/14).
      automountServiceAccountToken: false
+      # ---- Docker-in-Docker: the job executor daemon (privileged) -------------------
+      #
+      # NATIVE SIDECAR (k3s v1.36; sidecar containers are GA). It is an initContainer with
+      # `restartPolicy: Always`, which means: it starts BEFORE the regular containers, keeps
+      # running alongside them, and is terminated after them. kubelet holds `runner` until
+      # this container's startupProbe passes, so the ordering is enforced by the kubelet
+      # instead of hand-rolled shell in the runner.
+      #
+      # This REPLACES two bounded wait-loops that used to live in the runner's `command:`.
+      # Deleting them also deletes an envsubst hazard: clusters/base/apps-wave-1.yaml runs
+      # postBuild substitution over this ENTIRE manifest, comments included, and those loops
+      # depended on bare dollar-names surviving it untouched. There is now no shell here at
+      # all, so there is nothing to escape or mis-escape.
+      #
+      # WHY THE PROBE IS `exec docker ... info` AND NOT `tcpSocket`. A tcpSocket probe would
+      # re-open the exact gap the old Stage 1 existed to close: dockerd writes the TLS certs
+      # and only THEN begins listening, so the port can accept a connection while the certs
+      # are absent or incomplete -- and act_runner hard-exits on a missing
+      # /certs/client/ca.pem. A full TLS-verified `docker info` proves BOTH that the certs
+      # are usable and that the daemon answers.
+      #
+      # CORRECTION (measured 2026-09-29, after this conversion shipped): the commit that
+      # introduced this claimed the conversion would REMOVE the one-per-start dockerd line
+      # `http: TLS handshake error from 127.0.0.1: EOF`. IT DOES NOT. Exactly one still
+      # appears, immediately after "API listen on [::]:2376" -- it is now this probe's FIRST
+      # attempt, landing microseconds after dockerd begins listening and cut off by
+      # timeoutSeconds: 1. The next poll succeeds. It is cosmetic, exactly as cosmetic as it
+      # was before, and deliberately NOT chased: raising timeoutSeconds to hide one benign
+      # startup line would slow every subsequent poll. Left documented rather than silenced.
+      #
+      # periodSeconds 1 x failureThreshold 60 preserves the previous 60s cap.
+      initContainers:
+        - name: dind
+          restartPolicy: Always   # <- this field is what makes it a SIDECAR
+          image: docker:27-dind
+          securityContext:
+            privileged: true
+          env:
+            # Generate + publish TLS certs into the shared emptyDir; dockerd then listens
+            # on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
+            - name: DOCKER_TLS_CERTDIR
+              value: /certs
+          volumeMounts:
+            - name: certs
+              mountPath: /certs
+            - name: dind-storage
+              mountPath: /var/lib/docker
+          resources:
+            requests:
+              cpu: 100m
+              memory: 256Mi
+            limits:
+              memory: 2Gi
+          startupProbe:
+            exec:
+              command:
+                - docker
+                - --host=tcp://127.0.0.1:2376
+                - --tlsverify
+                - --tlscacert=/certs/client/ca.pem
+                - --tlscert=/certs/client/cert.pem
+                - --tlskey=/certs/client/key.pem
+                - info
+            periodSeconds: 1
+            failureThreshold: 60
      containers:
        # ---- act_runner: registers with Gitea, dispatches jobs to the DinD daemon ----
        - name: runner
          image: gitea/act_runner:0.2.11
-          # TWO-STAGE BOUNDED WAIT on the DinD sidecar, then exec the image's normal
-          # entrypoint (/sbin/tini -- /opt/act/run.sh), which registers-from-env on first
-          # boot and runs `act_runner daemon`.
-          #
-          # STAGE 1 — the certs must EXIST: act_runner reads /certs/client/ca.pem at startup
-          #   and hard-exits if it is missing (a startup race against the sidecar).
-          # STAGE 2 — the daemon must actually ACCEPT a connection: the cert FILE appearing is
-          #   necessary but NOT sufficient. dockerd writes the certs and only THEN begins
-          #   listening on :2376, so a runner that proceeds on the file alone still hard-exits
-          #   with "cannot ping the docker daemon at tcp://localhost:2376" — observed
-          #   crashlooping 11 times in ~8 minutes on a fresh pod after a node bounce.
-          # Both loops are bounded (60 x 1s) and exit 1 on timeout, so a genuinely dead sidecar
-          # surfaces as a CrashLoopBackOff with a clear message instead of hanging forever.
-          #
-          # ENVSUBST SAFETY — the BARE $i / $j below are CORRECT. Do not "tidy" them into the
-          #   braced form or the double-dollar form; both are wrong here, for different reasons.
-          #   clusters/base/apps-wave-1.yaml carries postBuild.substituteFrom cluster-vars, so
-          #   Flux envsubst runs over this ENTIRE manifest — YAML comments very much included.
-          #   Its contract (drone/envsubst; the same one spelled out in clusters/base/apps-vms.yaml,
-          #   and re-verified against this exact file with `flux envsubst`, flux v2.4.0) is:
-          #     - A bare  $name  is NOT a reference to envsubst: it passes through untouched, and
-          #       that holds even when the name IS set in cluster-vars. So $i and $j reach the
-          #       cluster verbatim and both 60s bounds below are real, enforced bounds.
-          #     - $((i + 1))  is likewise untouched: "(" cannot begin a variable name.
-          #     - A BRACED dollar-name IS a reference and gets replaced — with the empty string
-          #       when the name is unset. One of those here would make a bound read [ "" -ge 60 ],
-          #       which errors, short-circuits the &&, and silently turns the wait UNBOUNDED.
-          #     - A DOUBLE-dollar braced name is envsubst's escape and survives as the braced
-          #       form — still wrong for us, because on any path where envsubst does NOT run
-          #       (a plain `kustomize build` or `kubectl apply -k` of this directory) the shell
-          #       reads that leading double-dollar as its own PID, so a bound would compare a
-          #       string like 1234{i} against 60. The bare form is the only spelling that is
-          #       correct on BOTH paths.
-          #     - A braced name containing punctuation is a HARD ERROR ("unable to parse variable
-          #       name") that fails this whole Kustomization — and this Kustomization gates every
-          #       merge in the repo. So never write one anywhere in this file, INCLUDING in a
-          #       comment. That is why this note names the three forms in words rather than
-          #       showing them: an earlier draft of it spelled them out literally and broke
-          #       `flux envsubst` on the file. (Same trap the indexed-expansion warning in
-          #       clusters/base/apps-vms.yaml describes.)
-          command:
-            - /bin/sh
-            - -c
-            - |
-              # Stage 1: the sidecar's TLS client certs must appear in the shared emptyDir.
-              i=0
-              until [ -f /certs/client/ca.pem ]; do
-                i=$((i + 1))
-                [ "$i" -ge 60 ] && { echo "dind certs never appeared" >&2; exit 1; }
-                echo "waiting for dind TLS certs ($i)"; sleep 1
-              done
-              # Stage 2: dockerd must actually accept a connection on :2376. busybox nc in this
-              # Alpine-based image (gitea/act_runner:0.2.11) supports -z (scan) and -w (timeout).
-              j=0
-              until nc -z -w1 127.0.0.1 2376; do
-                j=$((j + 1))
-                [ "$j" -ge 60 ] && { echo "dind daemon never accepted a connection on 2376" >&2; exit 1; }
-                echo "waiting for dind daemon on :2376 ($j)"; sleep 1
-              done
-              exec /sbin/tini -- /opt/act/run.sh
+          # No `command:` override. The image's own entrypoint (/sbin/tini -- /opt/act/run.sh)
+          # registers from env on first boot and runs `act_runner daemon`. The two wait-loops
+          # that used to live here are gone: kubelet now holds this container until the dind
+          # sidecar's startupProbe passes (see initContainers above).
          env:
            - name: GITEA_INSTANCE_URL
              value: https://gitea.${BASE_DOMAIN}
@@ -198,52 +207,6 @@ spec:
              memory: 128Mi
            limits:
              memory: 512Mi
-        # ---- Docker-in-Docker: the job executor daemon (privileged) ----
-        #
-        # FOLLOW-UP (deliberately NOT bundled into the readiness fix above — own change):
-        #   Convert this container into a NATIVE SIDECAR and delete both shell wait-loops.
-        #   k3s is now v1.36, where sidecar containers are GA, so kubelet can enforce the
-        #   ordering that the loops above hand-roll — which also removes the envsubst hazard
-        #   from this file entirely (no shell, no $i/$j, nothing to escape or mis-escape).
-        #   Shape to build:
-        #     1. Move this whole container into `initContainers:` as the FIRST entry.
-        #     2. Add `restartPolicy: Always` to it. That single field is what makes an
-        #        initContainer a sidecar: it starts before the regular containers, keeps
-        #        running alongside them, and is terminated after them.
-        #     3. Give it a `startupProbe` that succeeds only when dockerd accepts a TLS
-        #        connection — e.g. `exec` a `docker -H tcp://127.0.0.1:2376 --tlsverify
-        #        --tlscacert /certs/client/ca.pem --tlscert /certs/client/cert.pem
-        #        --tlskey /certs/client/key.pem info`. Do NOT use a bare `tcpSocket` probe:
-        #        it would re-open the same "port accepting but TLS/certs not ready" gap that
-        #        Stage 1 above exists to close. `periodSeconds: 1` + `failureThreshold: 60`
-        #        preserves today's 60s cap.
-        #     4. kubelet then holds the `runner` container until that startupProbe passes, so
-        #        `runner` can drop its `command:` override completely and run the image's
-        #        default entrypoint.
-        #   Why it is a separate change: this Deployment is the ENFORCED MERGE GATE for this
-        #   entire repository. If the conversion is wrong, nothing can merge — so it must land
-        #   alone, where the failure is unambiguous and not entangled with anything else, and
-        #   only after a real workflow run has been observed green on the converted pod.
-        - name: dind
-          image: docker:27-dind
-          securityContext:
-            privileged: true
-          env:
-            # Generate + publish TLS certs into the shared emptyDir; dockerd then listens
-            # on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
-            - name: DOCKER_TLS_CERTDIR
-              value: /certs
-          volumeMounts:
-            - name: certs
-              mountPath: /certs
-            - name: dind-storage
-              mountPath: /var/lib/docker
-          resources:
-            requests:
-              cpu: 100m
-              memory: 256Mi
-            limits:
-              memory: 2Gi
      volumes:
        - name: data
          persistentVolumeClaim:
@@ -32,11 +32,46 @@ env:
  # (10.43.0.0/16); 192.168.0.0/16 covers the home LAN.
  - name: ALLOWED_NETWORKS
    value: 10.0.0.0/8,192.168.0.0/16
-  # PLEX_CLAIM is deliberately NOT set here: a https://plex.tv/claim token has a ~4-min TTL, far
-  # shorter than the commit -> push -> Flux reconcile -> pod-roll cycle, so any token committed to git
-  # is already expired by the time the pod starts. Claiming is a one-time LIVE step (UI sign-in, or a
-  # single API POST) against the running server; the claim then persists on the /config PVC and
-  # survives pod rolls (including the roll this env change causes), so it never belongs in git.
+  # PLEX_CLAIM — the ONE-SHOT plex.tv server claim, sourced from the SOPS Secret `plex-claim`
+  # (apps/wave-0/plex-claim-secret.sops.yaml, same wave => applied in the same reconcile as this
+  # workload). Plex cannot create libraries over the mounted bulk share while the server is
+  # UNCLAIMED, so this is wired; it was previously left out on the reasoning below, and only one
+  # half of that reasoning survived contact with the upstream source.
+  #
+  # A SHORT TTL IS THE OPERATIONAL CONSTRAINT, not a reason to abstain. A https://plex.tv/claim token
+  # expires in MINUTES — read the countdown the claim page renders; do not budget against a number
+  # written in this tree. (This comment previously asserted "~4 minutes" as established fact. It was
+  # UNCITED, upstream plexinc/pms-docker documents no expiry at all, and that number was load-bearing
+  # for the whole procedure design — including a since-disproven claim that the delivery pipeline was
+  # too slow to fit inside it. Measured 2026-09-29: the full loop is ~2m45s and fits.)
+  #
+  # The live token is injected by the operator inside that window, with an explicit `flux reconcile`
+  # to collapse the wait. Scripted end to end as infra/plex/claim-ship.sh; runbooks/plex-claim.md is
+  # the procedure. SERVER CLAIMED 2026-09-29 — this env is now a permanent no-op (see below), and the
+  # committed value is the spent token from that claim rather than the original placeholder.
+  #
+  # ONE-SHOT, AND SAFE TO SIT HERE EXPIRED. Verified from plexinc/pms-docker's own first-run script
+  # (branch `master`, root/etc/cont-init.d/40-plex-first-run, confirmed byte-identical to the file in
+  # the running `:latest` image), NOT assumed:
+  #   - the exchange (line 106) is guarded on TWO conditions: PLEX_CLAIM non-empty AND the
+  #     PlexOnlineToken pref in /config empty. Once claimed, the block is SKIPPED — this env is a
+  #     no-op on every subsequent start, whatever it holds. (Upstream README: "If server is already
+  #     logged in, this parameter is ignored.") The claim itself persists on the /config PVC as before.
+  #   - an invalid/expired token on a still-unclaimed /config gets a 403 with an empty body; curl's
+  #     status is unchecked, the token `sed` yields "", the success branch at line 120 does not fire,
+  #     there is no `else` and no `exit 1`, and the script (no `set -e`) exits 0. It comes up UNCLAIMED and
+  #     SERVING — it does NOT crashloop. The image also leaves S6_BEHAVIOUR_IF_STAGE2_FAILS at its
+  #     default 0, which continues startup even when a cont-init script fails.
+  # So the earlier "a placeholder claim Secret would crashloop the pod" note was WRONG about the
+  # placeholder; it was right only about an ABSENT Secret. secretKeyRef here has no `optional` field,
+  # so the Secret must always exist in the Flux graph — which is why it is committed, encrypted, and
+  # listed in apps/wave-0/kustomization.yaml rather than hand-created at claim time. No schema change
+  # is needed: `optional` would only matter if a missing Secret had to be tolerated, and it doesn't.
+  - name: PLEX_CLAIM
+    valueFrom:
+      secretKeyRef:
+        name: plex-claim
+        key: PLEX_CLAIM
  # ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex
  # which URL to advertise to plex.tv for REMOTE clients. The only candidate is
  # https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL
@@ -56,10 +91,12 @@ storage:
                              # container's EPHEMERAL layer -> unclaimed + all state lost on restart. The image inits as
                              # root (s6-svscan PID 1), chowns /config to the plex user, then drops -> no fsGroup needed
                              # (verified in-cluster: PID1=root, "Plex Media Server"=plex). Persisting /config here also
-                              # makes the claim survive restarts once claimed. To auto-claim on first boot instead of the
-                              # UI sign-in, the operator adds a PLEX_CLAIM env from a fresh https://plex.tv/claim token
-                              # (4-min TTL) — deliberately NOT wired here: the env secretKeyRef pattern has no `optional`,
-                              # so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
+                              # makes the claim survive restarts once claimed — which is what makes the PLEX_CLAIM env
+                              # above a ONE-SHOT: the token is exchanged for a PlexOnlineToken written here, and the
+                              # exchange is then skipped on every later start. (The older note here claimed a
+                              # placeholder claim Secret "would crashloop the pod"; upstream's first-run script says
+                              # otherwise — see the PLEX_CLAIM comment in `env` above. An ABSENT Secret is the hard
+                              # failure, and that is why one is committed.)
  # THE LIBRARY (docs/04 bulk tier). One READ-ONLY mount of the export ROOT, giving /media/movies,
  # /media/tv and /media/photos — the whole tree the layout actually has (`ls /mnt/Storage/plex`:
  # downloads/ movies/ photos/ tv/).
@@ -1844,6 +1844,10 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
    convention:
      - `<svc>.yaml`                        -> wave(svc)
      - `<svc>-bootstrap-secret.sops.yaml`  -> wave(svc)   (consumed by svc's env, same apply)
+      - `<svc>-claim-secret.sops.yaml`      -> wave(svc)   (a one-shot upstream-service claim token
+        consumed by svc's env — plex's plex.tv claim. Same wave as the app for the same reason the
+        bootstrap secret is: an env `secretKeyRef` has no `optional`, so the Secret must be applied
+        in the SAME reconcile as the workload or the pod is CreateContainerConfigError.)
      - `<svc>-oauth2-proxy-credentials.sops.yaml` -> wave(svc)  (forward-auth sidecar OIDC creds)
      - `<svc>-db.yaml`                     -> wave(svc-db) (a strictly-lower wave; CNPG DB)
      - `<svc>-db-app.sops.yaml`            -> wave(svc-db) (CNPG initdb owner secret)
@@ -1870,6 +1874,11 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
        w = waves[name]
        add(w, f"{name}.yaml")
        add(w, f"{name}-bootstrap-secret.sops.yaml")
+        # A one-shot claim token for an UPSTREAM service (plex's plex.tv claim). Same wave as the
+        # app: it is read by the app's env at pod start, and an env secretKeyRef has no `optional`,
+        # so a Secret applied in a later wave would leave the pod in CreateContainerConfigError.
+        # Placed only when the file exists, so services without one are unaffected.
+        add(w, f"{name}-claim-secret.sops.yaml")
        # forward-auth (Option B): the per-app oauth2-proxy sidecar's OIDC client + cookie secret.
        # Same wave as the app (the sidecar reads it at pod start); placed only when the file exists,
        # so non-forward-auth apps are unaffected. Mirrors the bootstrap-secret naming convention.
@@ -2548,7 +2557,8 @@ def _selftest() -> int:
    # 7a) plan_apps_waves: sidecars inherit their unit's wave; not-onboarded SDs are skipped;
    #     EXTRA_UNITS anchor strictly above; a file absent from the tree is never placed.
    present = {
-        "a.yaml", "svc.yaml", "svc-bootstrap-secret.sops.yaml",
+        "a.yaml", "a-claim-secret.sops.yaml",
+        "svc.yaml", "svc-bootstrap-secret.sops.yaml",
        "svc-db.yaml", "svc-db-app.sops.yaml", "svc-db-backup-credentials.sops.yaml",
        "svc-db-backup-credentials-prod.sops.yaml",
        "gitea.yaml",
@@ -2559,6 +2569,10 @@ def _selftest() -> int:
    plan = plan_apps_waves(plan_sds, present)
    flat = {f: wv for wv, fs in plan.items() for f in fs}
    check(flat.get("a.yaml") == 0, "leaf app must land in wave 0")
+    check(flat.get("a-claim-secret.sops.yaml") == 0,
+          "a `<svc>-claim-secret.sops.yaml` (plex's plex.tv claim) must be placed in the SAME wave "
+          "as its app — an env secretKeyRef has no `optional`, so a later wave means "
+          "CreateContainerConfigError")
    check(flat.get("svc-db.yaml") == 0 and flat.get("svc-db-app.sops.yaml") == 0
          and flat.get("svc-db-backup-credentials.sops.yaml") == 0
          and flat.get("svc-db-backup-credentials-prod.sops.yaml") == 0,
@@ -1,5 +1,19 @@
 # `ci/` — portable merge gate (runnable today, without Gitea)

+> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
+> This document speaks as though no cluster exists and the platform is scaffolding. **A
+> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
+> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
+> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
+> describing a **live gap** rather than an expected stage — most importantly, L2/L3
+> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
+> coverage hole, not a phase.
+>
+> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
+> documentation rather than measurement, and `main` was unprotected with zero runners
+> registered the whole time. **Query the live API before asserting platform state.**
+
+
 **Task T-0.2 · R-TEST-02 (reproducible, single-command) · Owner WS-GITOPS (CI).**

 The same merge gate the Gitea Actions workflow runs
@@ -61,5 +61,45 @@ if [ -d "${CONF_ART}/coverage" ]; then
  cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
 fi

+# ---------------------------------------------------------------------------------------
+# DID IT ACTUALLY RUN? -- the guard without which this stage FAKES A GREEN.
+#
+# run.sh exits 0 in "scaffold mode" when it cannot reach a cluster: it emits a JUnit file
+# whose every testcase is <skipped> and returns success. That is correct FOR run.sh -- an
+# honest skip is its documented no-cluster behaviour. It is catastrophic HERE, because
+# reaching this point means the environment DECLARED CLUSTER_AVAILABLE=true, and this stage
+# would otherwise print "L2/L3 conformance passed" over 54 tests that never executed.
+#
+# MEASURED 2026-09-29, before this guard existed:
+#   CLUSTER_AVAILABLE=true ./ci/conformance.sh   ->  rc=0
+#   "[junit] wrote skipped JUnit for 54 scaffolded test(s)"
+#   "[ ok ] L2/L3 conformance passed for touched subset"
+# One Gitea variable away from a fake green on an enforcing merge gate. That is exactly the
+# HARD RULE 4 violation this repo's whole gate design exists to prevent, and nothing caught
+# it because nobody had ever set the variable.
+#
+# A skip while a cluster is DECLARED is a contradiction: either the declaration is wrong or
+# the cluster is unreachable. The merge-gate decision matrix already says this case is BLOCK
+# ("conformance skipped while CLUSTER_AVAILABLE=true -> BLOCK (it should have run)"). This
+# makes the stage itself honour that instead of reporting a pass.
+JUNIT="${CONF_ART}/report.junit.xml"
+if [ -f "$JUNIT" ]; then
+  total="$(grep -o '<testcase' "$JUNIT" | wc -l | tr -d ' ')"
+  skipped="$(grep -o '<skipped' "$JUNIT" | wc -l | tr -d ' ')"
+  if [ "${total:-0}" -gt 0 ] && [ "${skipped:-0}" -ge "${total:-0}" ]; then
+    err "CLUSTER_AVAILABLE=true, but the L2/L3 suite SKIPPED every one of its ${total} tests"
+    err "  (scaffold mode -- run.sh could not reach a cluster and exited 0)."
+    err "  This stage will NOT report that as a pass. Either the cluster is unreachable from"
+    err "  this runner, or CLUSTER_AVAILABLE is set when it should not be."
+    err "  NOTE: the in-cluster trigger path is still UNIMPLEMENTED -- see"
+    err "  docs/plans/l2-l3-conformance-ci-wiring.md. Setting CLUSTER_AVAILABLE=true does not"
+    err "  by itself make L2/L3 run; it only makes this stage attempt a LOCAL run that the CI"
+    err "  runner has no credential for (automountServiceAccountToken: false, by design)."
+    summary "- conformance-l2-l3: **BLOCKED — declared available but every test skipped** (not a pass)"
+    exit 1
+  fi
+  log "L2/L3 executed: ${total} testcase(s), ${skipped} skipped"
+fi
+
 ok "L2/L3 conformance passed for touched subset"
 summary "- conformance-l2-l3: **passed**"
@@ -1,5 +1,19 @@
 # clusters/

+> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
+> This document speaks as though no cluster exists and the platform is scaffolding. **A
+> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
+> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
+> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
+> describing a **live gap** rather than an expected stage — most importantly, L2/L3
+> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
+> coverage hole, not a phase.
+>
+> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
+> documentation rather than measurement, and `main` was unprotected with zero runners
+> registered the whole time. **Query the live API before asserting platform state.**
+
+
 Flux entrypoints and the environment-overlay model (**R-ENV-02**, **R-GITOPS-01/02/03**).

 Governing spec: [`docs/05-gitops-and-delivery.md`](../../home-network-k3s/docs/05-gitops-and-delivery.md),
@@ -1,5 +1,19 @@
 # conformance/ — the home-k3s conformance harness (WS-TEST)

+> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
+> This document speaks as though no cluster exists and the platform is scaffolding. **A
+> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
+> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
+> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
+> describing a **live gap** rather than an expected stage — most importantly, L2/L3
+> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
+> coverage hole, not a phase.
+>
+> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
+> documentation rather than measurement, and `main` was unprotected with zero runners
+> registered the whole time. **Query the live API before asserting platform state.**
+
+
 This subtree is the **definition of "done"** for the whole platform. Nothing is accepted
 on a demo, a screenshot, or "it reconciled": every requirement is proven by an
 **automated, reproducible, behavior-based test** tagged with the requirement ID(s) it
@@ -1,5 +1,19 @@
 # In-cluster conformance Job (L2/L3)

+> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
+> This document speaks as though no cluster exists and the platform is scaffolding. **A
+> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
+> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
+> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
+> describing a **live gap** rather than an expected stage — most importantly, L2/L3
+> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
+> coverage hole, not a phase.
+>
+> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
+> documentation rather than measurement, and `main` was unprotected with zero runners
+> registered the whole time. **Query the live API before asserting platform state.**
+
+
 Per **ADR-0011** and **docs/14 "Where each layer runs"**: L0/L1 run in CI without a
 cluster, but **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job**
 on the dev/test cluster that **CI triggers and waits for**. CI holds **no** cluster
@@ -38,8 +38,21 @@ agents. So the credential model must be decided by the operator before anything
 | Runner deployment | `apps/wave-1/gitea-actions.yaml` | act_runner + DinD (Docker executor). **`automountServiceAccountToken: false`** — the runner and its job containers hold NO SA token. Broad egress allowed (`- {}`), so it can *reach* the kube-API VIP `192.168.1.49:6443`, but has no credential. |
 | Secrets encryption | `.sops.yaml` (repo root) | age recipient `age175l8m…mfzg`; encrypts `data`/`stringData` of `*.sops.yaml` under `platform|apps|catalog/**` and `secrets/**`. Private key is the in-cluster `sops-age` Secret Flux uses. |

-**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the dev cluster
-is LIVE (Phase 6). So `CLUSTER_AVAILABLE` is the only remaining gate, exactly as the blueprint says.
+**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the cluster is
+LIVE (and it is now PROD; dev was decommissioned 2026-09-12).
+
+> **CORRECTION (2026-09-29): `CLUSTER_AVAILABLE` is NOT "the only remaining gate", and setting it
+> today is actively DANGEROUS.** Measured: with `CLUSTER_AVAILABLE=true` and no reachable cluster,
+> `conformance/run.sh` exits 0 in scaffold mode having marked all 54 tests `<skipped>`, and
+> `ci/conformance.sh` then printed `[ ok ] L2/L3 conformance passed for touched subset` — **rc 0, a
+> fake green**, on a merge gate that is now an enforcing required status check. The CI runner has
+> no cluster credential at all (`automountServiceAccountToken: false`, by design), so the "local
+> run" this stage attempts can never succeed; it can only skip and, before the guard added on
+> 2026-09-29, lie about it.
+>
+> `ci/conformance.sh` now BLOCKS (exit 1) when a cluster is declared but every test skipped. **Do
+> not set `CLUSTER_AVAILABLE=true` until §3's trigger path actually exists** — the variable does
+> not enable L2/L3, it only changes which way this stage fails.

 ---

@@ -43,7 +43,65 @@ list. That is precisely the observed symptom: zero runners, pod running, no erro
 looks at. Clearing the PVC is what forces a real re-registration; the script prints the
 commands.

-## Sequence — order matters
+## Sequence — DONE 2026-09-29
+
+All four steps are complete and each was verified, not assumed:
+
+| Step | State |
+|---|---|
+| 1. Runner registered | ✅ `incluster-dind` online. Root cause of its absence was a stale `/data/.runner` identity; clearing the PVC forced re-registration |
+| 2. Gate observed green on a real PR | ✅ `gate / gate (pull_request)` ~1m35s |
+| 3. `enable_status_check: true` | ✅ context `gate / gate (pull_request)` — that one ONLY |
+| 4. Collaborator + approvals | ✅ `jakman85` is owner; `required_approvals: 1` |
+
+Direct push to `main` was **provoked and rejected** (`pre-receive hook declined`) as
+`gitea-admin`, a site admin — so push protection is not merely declared, it holds for the
+account agents authenticate as.
+
+### 5. Block the admin override — the step the first pass MISSED
+
+`enable_status_check` and `required_approvals` were set on 2026-09-29 and **still did not
+bite**, because `block_admin_merge_override` was `false`. Gitea lets repo admins override
+merge requirements without it, and `jakman85` holds `owner` — so the two settings that looked
+like the gate were overridable by precisely the account that merges.
+
+**The evidence was sitting in plain sight:** PRs #5 through #9 all merged with
+`required_approvals: 1` in force and **zero approvals recorded on any of them**. Checking the
+*branch* view (`/branches/main`) hid this — it does not expose the field. Read the rule object
+(`/branch_protections/main`) and enumerate its keys.
+
+> **Do not probe for field names you assume exist.** This Gitea (1.24.3) has no
+> `apply_to_admins`; asking `jq` for it returns `null`, which reads like "set to false" but
+> means "key absent". Enumerate with `to_entries` and act on what is really there.
+
+Now `block_admin_merge_override: true`. **That is still only a setting, not evidence** — it is
+proven when a green PR is REFUSED a merge until approved, and not before.
+
+### `require-gate.sh` / `verify-then-enforce.sh`
+
+`verify-then-enforce.sh` is the re-runnable form of the whole safe order. It merges nothing —
+merging is a human's reviewed act — but it forces the Flux reconcile and then **aborts** unless
+the runner pod is healthy, a runner is online, and a real workflow goes green on it, before
+enabling enforcement and probing the push. `require-gate.sh` is the enforcement step alone.
+
+Re-run `verify-then-enforce.sh` after any future change to the act-runner Deployment.
+
+### Why these scripts exist at all, and what is still missing
+
+Gitea's repo settings live in **its own database**, not in anything Flux reconciles or the
+conformance gate can read. Nothing detects drift from what this directory declares. That is
+the root cause of the entire 2026-09-29 session: branch protection was off and the runner was
+gone for a month while the log recorded both as working, and **no check existed that could
+have caught it.**
+
+These scripts encode the desired state and can re-assert it, but they are not a detector. The
+durable fix is a conformance check that reads the live Gitea API and fails when it diverges
+from what is declared here — turning a class of silent, invisible failure into a loud one.
+**That check does not exist yet and is the highest-value remaining work in this directory.**
+
+## Historical sequence (kept for context)
+
+### Original ordering notes

 1. **Get an Actions runner registered** — `runner-token-rotate.sh`, then clear the PVC so
   the runner actually re-registers, then confirm it shows `online` in the API. Do not
@@ -0,0 +1,36 @@
+#!/usr/bin/env bash
+# Stop repository admins from overriding the merge requirements on `main`.
+#
+# WHY. Gitea lets repo admins override branch-protection merge requirements unless
+# `block_admin_merge_override` is set. It was FALSE, and jakman85 holds `owner` -- so
+# required_approvals and the required status check were both overridable by the one account
+# that actually merges. Evidence: PRs #5-#9 all merged with ZERO approvals recorded against
+# `required_approvals: 1`.
+#
+# NOTE ON FIELD NAMES: this Gitea (1.24.3) has NO `apply_to_admins` field. Querying it with jq
+# returns null, which reads like "set to false" but actually means "key absent". Enumerate the
+# real keys (`to_entries`) rather than probing names you assume exist.
+#
+# Recoverable: an owner can PATCH this back to false if it gets in the way.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
+api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
+B="https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main"
+
+echo "==> before: block_admin_merge_override = $(api "$B" | jq -r .block_admin_merge_override)"
+api -X PATCH -H 'Content-Type: application/json' \
+  -d '{"block_admin_merge_override": true}' "$B" >/dev/null
+AFTER=$(api "$B" | jq -r .block_admin_merge_override)
+echo "==> after:  block_admin_merge_override = $AFTER"
+[ "$AFTER" = true ] || { echo "PATCH did not take -- do not record this as enforced." >&2; exit 1; }
+
+api "$B" | jq -r '{enable_push,enable_status_check,status_check_contexts,required_approvals,block_admin_merge_override}'
+cat <<'NOTE'
+
+NOW PROVE IT -- the setting is not the evidence. On the next PR:
+  * let the gate go green, then try to merge WITHOUT approving it.
+  * Gitea must REFUSE. If it merges, this still is not doing its job.
+Do not record approvals as enforced until you have seen a merge refused.
+NOTE
@@ -0,0 +1,60 @@
+#!/usr/bin/env bash
+# Items 1 + 2: make the gate a REQUIRED status check and require one approval.
+#
+# RUN THIS LAST. Prerequisites, in order:
+#   a) PR #7 (dind native sidecar) is MERGED, and
+#   b) you have SEEN a real workflow run go green on the converted pod.
+# The runner Deployment is the merge gate for the whole repo. Turning on a required check
+# while the runner is broken means no PR can merge -- including the revert. (Recoverable:
+# an owner can set enable_status_check back to false. But do not rely on that.)
+#
+# Requires exactly ONE context: `gate / gate (pull_request)`.
+#   * NOT `gate / conformance (*)` -- that job is `if: false`, and a skipped Gitea job reports
+#     SUCCESS ("Has been skipped"). Requiring it would pass VACUOUSLY forever while nothing
+#     ran, which under HARD RULE 4 is worse than a deadlock: a deadlock is loud.
+# block_admin_merge_override: true is LOAD-BEARING and was missing on the first pass. Gitea
+# lets repo admins override merge requirements without it, so required_approvals and the
+# required status check are both bypassable by exactly the account that does the merging.
+# Measured: PRs #5-#9 merged with ZERO approvals recorded against required_approvals: 1.
+#
+# FIELD NAMES: this Gitea (1.24.3) has NO `apply_to_admins`. Asking jq for it returns null,
+# which reads like "false" but means "key absent" -- enumerate real keys with `to_entries`
+# instead of probing names you assume exist. That mistake cost a wasted PATCH.
+#
+#   * NOT the `(push)` variant -- after PR #6 there are no per-branch push runs at all, so a
+#     PR would never produce that context.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
+api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
+
+echo "==> updating branch protection on main"
+jq -n '{
+  enable_push: false,
+  enable_status_check: true,
+  status_check_contexts: ["gate / gate (pull_request)"],
+  required_approvals: 1,
+  block_on_rejected_reviews: true,
+  dismiss_stale_approvals: true,
+  block_on_outdated_branch: false,
+  block_admin_merge_override: true
+}' | api -X PATCH -H 'Content-Type: application/json' --data @- \
+  "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main" >/dev/null
+
+echo "==> verifying"
+api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \
+ | jq -r '{protected,user_can_push,enable_status_check,status_check_contexts,required_approvals}'
+
+cat <<'NOTE'
+
+NOTE ON required_approvals: 1 -- Gitea will not let an author approve their own PR.
+  * Agent-opened PRs (author gitea-admin) are approved by jakman85. Fine.
+  * PRs YOU open as jakman85 need gitea-admin to approve. If that is awkward, either open
+    agent work as gitea-admin and review as jakman85, or set required_approvals back to 0
+    and rely on the status check alone.
+
+Then prove it, rather than trusting the API:
+  git commit --allow-empty -m probe && git push origin main   # EXPECT: rejected
+  git reset --hard HEAD~1
+NOTE
@@ -0,0 +1,83 @@
+#!/usr/bin/env bash
+# verify-then-enforce.sh — run this AFTER you have merged PRs 8, 5, 6, 7 in that order.
+#
+# It does NOT merge anything. Merging is your reviewed act (CLAUDE.md: a human approves and
+# merges). This script does the mechanical half: prove the converted runner actually works,
+# and only then turn on enforcement.
+#
+# WHY THIS ORDER. The act-runner Deployment IS the merge gate for this repository. Enabling a
+# required status check while the runner is broken means NO PR can merge -- including the
+# revert. So PR #7 lands while the gate is still advisory, is PROVEN green on a real run, and
+# only then does enforcement go on.
+#
+# Aborts at the first checkpoint that does not pass. Safe to re-run.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
+REPO_ROOT=/var/home/keith/source/home-network-k3s-impl
+D="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+cd "$REPO_ROOT"
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
+api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
+A="https://${HOST}/api/v1/repos/${OWNER}/${REPO}"
+say() { printf '\n\033[1m==> %s\033[0m\n' "$1"; }
+die() { printf '\n\033[31mABORT: %s\033[0m\n' "$1" >&2; exit 1; }
+
+say "PRECHECK — all four PRs must be merged first"
+for n in 8 5 6 7; do
+  m=$(api "$A/pulls/$n" | jq -r .merged)
+  printf '  PR #%s merged=%s\n' "$n" "$m"
+  [ "$m" = true ] || die "PR #$n is not merged yet. Merge 8, 5, 6, 7 in that order, then re-run."
+done
+
+say "STEP 1 — force Flux to apply the sidecar now, not on the 10m interval"
+./infra/devcontainer/dev.sh run bash -c \
+  'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-1 --with-source'
+
+say "STEP 2 — CHECKPOINT: the converted pod must come up healthy"
+./infra/devcontainer/dev.sh run bash -c '
+  kubectl --context k3s-prod -n gitea-actions rollout status deploy/act-runner --timeout=180s
+  echo "--- sidecar (initContainers) ---"
+  kubectl --context k3s-prod -n gitea-actions get deploy act-runner \
+    -o jsonpath="{range .spec.template.spec.initContainers[*]}{.name} restartPolicy={.restartPolicy}{\"\n\"}{end}"
+  kubectl --context k3s-prod -n gitea-actions get pod -l app.kubernetes.io/name=act-runner' \
+  || die "converted act-runner pod is not healthy -- REVERT PR #7 before anything else"
+
+say "STEP 3 — CHECKPOINT: a runner must be ONLINE again"
+n=0
+for i in $(seq 1 40); do
+  n=$(api "https://${HOST}/api/v1/admin/actions/runners" | jq -r '[(.runners//[])[]|select(.status=="online")]|length')
+  printf '\r  online runners: %s  (%ss)   ' "$n" "$((i*5))"
+  [ "${n:-0}" -ge 1 ] && { echo; break; }
+  sleep 5
+done
+[ "${n:-0}" -ge 1 ] || die "no runner online after the conversion -- REVERT PR #7"
+
+say "STEP 4 — CHECKPOINT: a REAL workflow must go green ON THE CONVERTED POD"
+MAIN_SHA=$(api "$A/branches/main" | jq -r .commit.id)
+echo "  watching the post-merge push run on main (${MAIN_SHA:0:8})..."
+st=""
+for i in $(seq 1 60); do
+  st=$(api "$A/commits/$MAIN_SHA/status" | jq -r .state)
+  printf '\r  main gate: %-8s (%ss)   ' "$st" "$((i*10))"
+  case "$st" in success) echo; break;; failure|error) die "gate FAILED on the converted pod -- REVERT PR #7";; esac
+  sleep 10
+done
+[ "$st" = success ] || die "gate never completed on the converted pod -- REVERT PR #7; do NOT enable enforcement"
+echo "  the converted sidecar ran a real job green. Enforcement is now safe."
+
+say "STEP 5 — require the gate + one approval"
+"$D/require-gate.sh"
+
+say "STEP 6 — prove a direct push is rejected (do not trust the API's word)"
+git fetch -q origin && git checkout -q main && git reset -q --hard origin/main
+git commit -q --allow-empty -m 'protection probe'
+if git push origin main 2>&1 | grep -q 'protected branch'; then
+  echo "  CONFIRMED: push to main rejected by the pre-receive hook."
+else
+  echo "  WARNING: the push was NOT rejected -- protection is not doing what it claims."
+fi
+git reset -q --hard origin/main
+
+say "FINAL STATE"
+api "$A/branches/main" | jq -r '{protected,user_can_push,enable_status_check,status_check_contexts,required_approvals}'
@@ -0,0 +1,94 @@
+#!/usr/bin/env bash
+# plex-claim-ship.sh — run this IMMEDIATELY AFTER editing the SOPS secret with a fresh
+# plex.tv claim token. It drives the whole CI/CD loop end to end and prints a running
+# stopwatch so you can see the budget being spent.
+#
+#   branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verify
+#
+# It never sees, prints, or handles the token: the file is already encrypted by sops.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev
+OWNER=Home-Network
+REPO=home-network-k3s-impl
+SECRET=apps/wave-0/plex-claim-secret.sops.yaml
+T0=$(date +%s)
+el() { printf '\n[T+%02d:%02d] %s\n' $(( ($(date +%s)-T0)/60 )) $(( ($(date +%s)-T0)%60 )) "$1"; }
+
+cd "$(git rev-parse --show-toplevel)"
+
+# --- preflight: refuse to ship a file that is not what we think it is -------------
+git diff --quiet -- "$SECRET" && { echo "ERROR: $SECRET is unchanged. Edit it first:"; \
+  echo "  infra/devcontainer/dev.sh run sops $SECRET"; exit 1; }
+grep -q 'PLEX_CLAIM: ENC\[' "$SECRET" || { echo "ERROR: $SECRET is not encrypted. REFUSING."; exit 1; }
+[ -z "$(git status --porcelain | grep -v " $SECRET\$")" ] || echo "NOTE: other files are dirty; only $SECRET will be committed."
+
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
+api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
+
+# Branch prefix is now cosmetic. It used to matter: gate.yml's push trigger covered
+# feature/** fix/** docs/** chore/** but not feat/**, and with act_runner at capacity 1 a
+# matching prefix fired a redundant run that serialised ahead of the PR run. Those
+# per-branch push triggers were deleted, so no prefix produces a duplicate run any more.
+BRANCH="feat/plex-claim-token-$(date +%H%M%S)"
+
+el "committing on $BRANCH"
+git checkout -q -b "$BRANCH"
+git add "$SECRET"
+git commit -q -m 'chore(plex): fresh plex.tv claim token
+
+Injected inside the plex.tv claim window per runbooks/plex-claim.md.'
+SHA=$(git rev-parse HEAD)
+
+el "pushing"
+git push -q -u origin "$BRANCH"
+
+el "opening PR"
+PR=$(jq -n --arg h "$BRANCH" --arg b main --arg t 'chore(plex): fresh plex.tv claim token' \
+       '{head:$h,base:$b,title:$t,body:"Token injection inside the claim window. Wiring already reviewed in #1."}' \
+     | api -X POST -H 'Content-Type: application/json' --data @- \
+         "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls" | jq -r '.number')
+[ -n "$PR" ] && [ "$PR" != "null" ] || { echo "PR creation failed"; exit 1; }
+echo "  PR #$PR"
+
+el "waiting for the gate (expect ~1m40s)"
+for i in $(seq 1 60); do
+  st=$(api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/commits/${SHA}/status" | jq -r '.state')
+  printf '\r  gate=%s  [T+%ss]   ' "$st" "$(( $(date +%s)-T0 ))"
+  case "$st" in success) echo; break;; failure|error) echo; echo "GATE FAILED — stopping."; exit 1;; esac
+  sleep 5
+done
+[ "$st" = "success" ] || { echo; echo "gate did not finish in time"; exit 1; }
+
+el "merging PR #$PR"
+api -X POST -H 'Content-Type: application/json' -d '{"Do":"merge"}' \
+  "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls/${PR}/merge" >/dev/null
+git checkout -q main && git pull -q --ff-only origin main
+
+el "forcing the Flux reconcile (--with-source, or it applies the OLD revision)"
+./infra/devcontainer/dev.sh run bash -c \
+  'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-0 --with-source'
+
+# MANDATORY here. Only the Secret ciphertext changed, so the Deployment pod template is
+# byte-identical and NOTHING restarts on its own. PLEX_CLAIM is read once, at container
+# start. Skipping this is failure mode #2 in runbooks/plex-claim.md.
+el "rolling the pod (mandatory — a changed Secret restarts nothing)"
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex rollout restart deployment/plex && \
+   kubectl --context k3s-prod -n plex rollout status deployment/plex --timeout=120s'
+
+el "verdict"
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex logs deploy/plex --since=3m | grep -i -e claim -e token || true'
+cat <<'NOTE'
+
+  "Attempting to obtain server token..." + "Token obtained successfully"  -> CLAIMED
+  "Attempting..." with no success line                                    -> token expired; retry, nothing is broken
+  neither line                                                            -> already claimed, or env did not reach the container
+
+Confirm it persisted (prints the attribute NAME only, never its value):
+
+  infra/devcontainer/dev.sh run bash -c \
+    'kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c \
+       "grep -o PlexOnlineToken= \"/config/Library/Application Support/Plex Media Server/Preferences.xml\" | head -1"'
+NOTE
@@ -6,7 +6,7 @@ Every alert rule in this platform carries an annotation of the form

    runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"

-46 of them, over the 35 names below. Until this directory existed, that path returned **HTTP 404** on the
+46 of them, over 35 names. Until this directory existed, that path returned **HTTP 404** on the
 live cluster: every alert that reached a human handed them a dead link at the exact moment they needed
 help. These pages are what makes the link resolve.

@@ -33,7 +33,8 @@ negative is how a CLI quirk becomes real data loss. Write `backups.velero.io`,
 `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`,
 `volumes.longhorn.io`, `nodes.longhorn.io`.

-`git push origin main` **is** the deploy; there is no other apply step. Durable changes go through git.
+Merging a PR into `main` **is** the deploy; there is no other apply step. `main` is protected, so a
+direct push is rejected — durable changes go through a branch, a PR, and a green gate.

 ## Alerts you should not trust the silence of (measured 2026-09-13)

@@ -108,6 +109,13 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
 - **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
  every green signal missed.

+### Operator procedures (no alert links here)
+- **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only
+  applies after a `/config` wipe. The claim token is short-lived — read the countdown on
+  plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
+  token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
+  Includes why a changed Secret does not restart anything, and why an expired token is harmless.
+
 ### Virtualisation
 - **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.
 - **`kubevirt-launch-failed`** — a VMI reached the Failed phase. The three k3s-specific launch traps.
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -78,8 +78,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -55,8 +55,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -63,8 +63,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -87,8 +87,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -69,8 +69,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -71,8 +71,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -68,8 +68,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -7,7 +7,7 @@
 ## What this means here

 Something in the GitOps chain is stuck at Ready=False. On this cluster that is more consequential than
-it sounds, because **`git push origin main` IS the deploy** — there is no other apply step. A stuck
+it sounds, because **merging a PR into `main` IS the deploy** — there is no other apply step. A stuck
 Kustomization means git and the cluster have diverged and nothing will close the gap.

 The chain has **14 tiers**: `flux-system` → `cilium` → `platform` → `platform-config` →
@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -58,8 +58,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -87,8 +87,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -67,8 +67,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -59,8 +59,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -57,8 +57,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -81,6 +81,11 @@ configMapGenerator:
      - node-io=node-io.md
      - node-memory=node-memory.md
      - node-memory-swap=node-memory-swap.md
+      # Operator procedure, not an alert page (no alert links to it — the
+      # alert-runbooks-resolvable check treats that as advisory). It is served
+      # because the plex.tv claim window is short and is not a thing to go looking
+      # for in git: /runbooks/plex-claim has to resolve the moment it is needed.
+      - plex-claim=plex-claim.md
      - pvc-capacity=pvc-capacity.md
      - targets-down=targets-down.md
      - update-rollback=update-rollback.md
@@ -83,8 +83,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -74,8 +74,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -75,8 +75,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -52,8 +52,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -49,8 +49,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -56,8 +56,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -56,8 +56,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -227,8 +227,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -52,8 +52,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -0,0 +1,189 @@
+# plex-claim — claiming the Plex Media Server inside the token window
+
+**Alert:** none. This is an **operator procedure**, not an incident page — it is indexed here because
+it is the page you need at the moment you need it, with a stopwatch running.
+**Files:** `catalog/plex.yaml` (env `PLEX_CLAIM`), `apps/wave-0/plex-claim-secret.sops.yaml`
+(the SOPS Secret `plex-claim`, key `PLEX_CLAIM`), `apps/wave-0/plex.yaml` (the rendered golden).
+
+## What this means here
+
+Plex will not let you **create libraries** over the mounted bulk media share while the server is
+**unclaimed** — the 8.33 T NFS tree at `/media` is there, readable, and unusable. Claiming binds the
+server to a plex.tv account. It is a **one-shot** operation: the token is exchanged once for a durable
+`PlexOnlineToken` written into `Preferences.xml` on the `plex-config` Longhorn PVC, and from then on
+the `PLEX_CLAIM` env is **ignored on every start** (upstream gate:
+`[ ! -z "${PLEX_CLAIM}" ] && [ -z "${token}" ]` in `root/etc/cont-init.d/40-plex-first-run`).
+
+**A `https://plex.tv/claim` token is short-lived — minutes, not hours.** Read the countdown the
+claim page shows you; do not budget against a number written here. That is shorter than PR → gate →
+merge → Flux's 10-minute reconcile interval → pod roll if you let those run on their own timers,
+which is why this page exists: the loop is scripted and the reconcile is **forced** rather than
+waited for. Measured end to end on 2026-09-29: **~2m45s**.
+
+**An expired or wrong token does not break anything.** Verified against the image's own first-run
+script (and the copy inside `plexinc/pms-docker:latest`): a failed exchange gets a 403 with an empty
+body, `curl`'s status is never checked, the token `sed` yields empty, the `if` does not fire, there is
+no `else` and no `exit 1`, and the script (no `set -e`) exits 0. The pod comes up **unclaimed and
+serving**. So a burnt token costs you one more attempt, not an outage — and the repo deliberately
+ships a **placeholder** in that Secret for exactly this reason (the env `secretKeyRef` has no
+`optional`, so the Secret must always exist even when its value is useless).
+
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that
+hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and that default has already changed once (k3s-dev until dev
+was decommissioned on 2026-09-13, k3s-prod now). Do not run two `dev.sh` invocations at once.
+
+Mutating `kubectl` and `flux reconcile` are blocked for agents here. **This whole page is operator
+work** — the reconcile and the pod roll below are deliberate operator actions, and the durable change
+is still a git change.
+
+## First commands
+
+**Before spending a token, find out whether the server is already claimed.** This prints only the
+attribute NAME, never its value — do not widen it to print the token:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c \
+     "grep -o PlexOnlineToken= \"/config/Library/Application Support/Plex Media Server/Preferences.xml\" | head -1"'
+```
+
+`PlexOnlineToken=` printed ⇒ **already claimed**; stop, there is nothing to do and no token to burn.
+Empty output ⇒ unclaimed, continue.
+
+Is the workload healthy enough to claim at all, and is the wave reconciling?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex get deploy,pod,pvc'
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0'
+```
+
+## The procedure
+
+> **STATUS: this server was CLAIMED on 2026-09-29** and `PlexOnlineToken` is present in
+> `/config` on the `plex-config` PVC. The upstream exchange is gated on that pref being
+> *absent*, so `PLEX_CLAIM` is now a **no-op on every start** whatever it holds. You only need
+> this page again after a `/config` wipe or a rebuild onto an empty PVC.
+
+**On the clock.** The token is short-lived — **read the countdown that plex.tv/claim shows you**
+rather than trusting a number written here. (Earlier revisions of this page asserted a "~4-minute
+TTL" in four places with no citation; upstream `pms-docker` documents no expiry at all. The page
+is the only authority, and it is on screen at T+0.)
+
+**A burnt token costs nothing.** An expired token gets a 403 with an empty body, `curl`'s status is
+never checked, the `if` does not fire, there is no `else` and no `exit 1`, and the script (no
+`set -e`) exits 0. The pod comes up unclaimed and serving. So retry freely.
+
+### One command
+
+1. **T+0 — get the token.** Sign in to plex.tv as the account that should OWN this server, open
+   <https://plex.tv/claim>, copy the `claim-…` string.
+
+2. **Put it in the Secret** — in place, encrypted, never touching a plaintext file:
+
+   ```
+   infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
+   ```
+
+3. **Ship it.** This drives the whole delivery loop and prints a running stopwatch:
+
+   ```
+   ./infra/plex/claim-ship.sh
+   ```
+
+   branch → commit → push → PR → gate → merge → `flux reconcile --with-source` → `rollout restart`
+   → verdict. It refuses to run if the file is unchanged or not encrypted.
+
+**Measured end to end, 2026-09-29: ~2m45s.** Gate ~1m40s, merge + reconcile ~25s, pod roll and
+container start ~30s.
+
+### Why it is a script and not six pasted commands
+
+`main` is a **protected branch** — the direct push this page used to instruct is now rejected — so
+the deploy is a PR merged after a green gate. Three of those steps are easy to get wrong under time
+pressure, and two are silent when skipped:
+
+- **`--with-source` is load-bearing.** Without it the Kustomization applies the previously-fetched
+  revision, not your commit.
+- **The pod roll is mandatory.** Only the Secret's ciphertext changes, so the Deployment's pod
+  template is byte-identical and *nothing restarts on its own*. `PLEX_CLAIM` is read once, at
+  container start. Skip this and the new Secret sits in the cluster while the running pod keeps the
+  old value forever.
+- **The branch is named `feat/…` on purpose.** `act_runner` capacity is 1, and `gate.yml`'s push
+  trigger covers `feature/** fix/** docs/** chore/**` but *not* `feat/**`. A `chore/` branch fires a
+  second redundant run that serialises ahead of the PR run and roughly doubles the wall clock.
+
+### Reading the verdict
+
+* `Attempting to obtain server token from claim token` **and** `Token obtained successfully`
+  ⇒ **claimed.** Re-run the First-commands `PlexOnlineToken=` check to confirm it persisted, then
+  create the libraries over `/media/movies`, `/media/tv`, `/media/photos`.
+* `Attempting…` with **no** success line ⇒ the token was expired or invalid. Nothing is broken; go
+  back to step 1 with a new one.
+* **Neither line** ⇒ the exchange was skipped: `/config` already had a `PlexOnlineToken` (already
+  claimed — the expected case now), or the env did not reach the container. Check by NAME only:
+
+  ```
+  infra/devcontainer/dev.sh run bash -c \
+    'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
+       -o jsonpath="{.items[0].spec.containers[0].env[*].name}"'
+  ```
+
+## Likely causes, most likely first
+
+When the claim does not take, in order of how often it is the answer here:
+
+1. **The token expired** — you exceeded the countdown, usually by reviewing the PR instead of
+   merging it the moment the gate went green. Symptom: the "Attempting…" line with no success line.
+   Remedy: a new token, and let `claim-ship.sh` drive the loop rather than pasting steps.
+2. **The pod was never restarted** (step 5 skipped). Symptom: no claim log lines at all from a pod
+   whose `AGE` predates your push. The Secret is correct in git and in the cluster and the running
+   container still holds the placeholder.
+3. **Flux had not applied the commit yet** when you rolled the pod — step 5 run before step 4
+   finished, so the new pod read the OLD Secret. Check
+   `kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0 -o jsonpath='{.status.lastAppliedRevision}'`
+   against your commit, then roll again.
+4. **Already claimed** — the exchange is gated on `PlexOnlineToken` being absent from `/config`, so a
+   server claimed earlier ignores every later token silently. This is the benign case.
+5. **`plex.tv` unreachable.** The upstream `curl` has no `--max-time`/`--connect-timeout`, so a stalled
+   connection can leave the container sitting in cont-init with **nothing listening on 32400** — the
+   one genuine hang risk in this path. Symptom: pod Running but never Ready, log stops after
+   "Attempting…". Check egress (`plex` is `egress: internet`) rather than re-rolling repeatedly.
+6. **Wrong plex.tv account.** The claim succeeds and binds the server to whoever was signed in at
+   step 1. Unclaiming is a plex.tv-side operation on that account — see "escalate or stop".
+
+## When to escalate or stop
+
+- **Stop after two failed token attempts** and fix the timing, not the token. The third rushed attempt
+  is how people start editing live Secrets by hand, which leaves the cluster holding a value that is
+  in no git commit — precisely the drift `qbittorrent-bootstrap` had to be recovered from.
+- **Never `kubectl edit`/`patch` the `plex-claim` Secret in the cluster, and never
+  `kubectl create secret` it.** Everything durable goes through the SOPS file; a live-only Secret is
+  reverted (or adopted with the wrong value) on the next reconcile.
+- **Never `sops -d` this file into a terminal, a log, or a shared screen**, and never widen the
+  `PlexOnlineToken=` check into one that prints the value. The token is a plex.tv account credential.
+- **If the server got claimed to the wrong account, stop and get the operator.** Reversing it means
+  removing the server from that account on plex.tv and clearing `PlexOnlineToken` from
+  `Preferences.xml` on the PVC — a destructive edit of app state on the replicated tier, not a git
+  change.
+- **Do not delete the `plex-config` PVC to "start clean".** It holds the library database, metadata
+  and watch state (`policy: daily-offcluster`), and it is expensive to rebuild. The claim is a
+  two-line pref, not a reason to lose the DB.
+- After a successful claim, **leave the expired token in the SOPS file**. It is inert (the exchange is
+  skipped once claimed) and the Secret must keep existing, because the env `secretKeyRef` has no
+  `optional` — deleting it would put the pod into `CreateContainerConfigError`.
+
+---
+Source of truth: `runbooks/plex-claim.md`. Index: `/runbooks/`
+Upstream behaviour cited here is from `plexinc/pms-docker` branch `master`,
+`root/etc/cont-init.d/40-plex-first-run` (s6-overlay v2), cross-checked against the same file inside
+the running `plexinc/pms-docker:latest` image.
@@ -58,8 +58,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -60,8 +60,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -54,8 +54,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -66,8 +66,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands