Compare commits

...
Author SHA1 Message Date
jakman85andClaude Opus 5 761e25cc3f docs(runbooks): two more uncited "4-minute" claims the first pass missed
gate / gate (push) Successful in 1m34s
gate / gate (pull_request) Successful in 1m35s
gate / conformance (push) Has been skipped
gate / conformance (pull_request) Has been skipped
A systematic grep for the TTL assertion found it still in runbooks/README.md and in
runbooks/kustomization.yaml, outside the page being rewritten. Same correction: the
countdown on plex.tv/claim is the authority, not a number in our tree.

README.md also now records that the server is CLAIMED and points at claim-ship.sh with
the measured ~2m45s loop, rather than describing a push-based sequence that no longer
exists.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 01:57:47 -05:00
jakman85andClaude Opus 5 8e017fad91 docs(runbooks): the PR flow, not git push origin main -- that instruction is now blocked
gate / gate (pull_request) Successful in 1m34s
gate / gate (push) Successful in 1m34s
gate / conformance (pull_request) Has been skipped
gate / conformance (push) Has been skipped
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks
instructed is REJECTED. The wording was merely stale before; it is now actively wrong,
and it is the line an operator reads mid-incident.

Every occurrence now describes the real path: push a branch, open a PR, merge once the
gate is green -- that merge is the deploy. 35 files shared one identical block;
flux-reconcile.md, README.md and plex-claim.md each said it their own way and were
corrected individually.

plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW
CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the
page only applies after a /config wipe.

Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate ->
merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose
under time pressure and two are SILENT when skipped: --with-source (without it the
Kustomization applies the previously-fetched revision), the mandatory rollout restart
(only the ciphertext changes, so the pod template is byte-identical and nothing restarts
itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside
gate.yml's push trigger fires a redundant run that serialises ahead of the PR run).

Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream
pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority.
Measured loop: ~2m45s.

DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false,
the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction
that a skipped Gitea job reports success rather than hanging.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 01:42:35 -05:00
40 changed files with 383 additions and 131 deletions

No files matched your search

+82
View File
@@ -4781,3 +4781,85 @@ accurate before the PR flow and are stale now. Correcting them is tracked, not d
3. Land `feat/plex-claim-sops-secret` (PR #1) once the gate can actually run. 3. Land `feat/plex-claim-sops-secret` (PR #1) once the gate can actually run.
4. Claim plex, after checking it is not already claimed. 4. Claim plex, after checking it is not already claimed.
5. Correct the 37 runbooks' `git push origin main` wording. 5. Correct the 37 runbooks' `git push origin main` wording.
## 2026-09-29 — Plex is claimed; the CI/CD gate that "enforced" it never existed. **PM HANDOFF.**
Began as "inject a plex.tv claim token". Ended by discovering the delivery pipeline was fiction,
rebuilding it, and shipping the token through it. Plex is **claimed** — `PlexOnlineToken` is
present in `/config` on the `plex-config` PVC, so the exchange is now a permanent no-op and the
library scan is running over the 8.33T NFS share.
### The finding: both halves of the 2026-08-31 CI claim were false
That entry recorded "Gitea Actions CI is LIVE + enforcing … required status check; runner
`incluster-dind` online". Measured against the live API on 2026-09-29:
| Claimed | Measured |
|---|---|
| `main` protected, gate required | `branch_protections` → `[]`; `protected:false`, `enable_status_check:false`, `required_approvals:0`, `user_can_push:true` |
| Runner online | **zero** runners at repo, org AND site level |
**PR #1 was the first pull request this repository has ever had.** Its contexts sat `pending`
for 30.6 minutes across 122 polls and never reported. For a month, every "the gate enforces this"
statement rested on an assertion nobody had checked — the same fail-open class as 2026-09-18,
and the reason it hid so long is that the log itself was the evidence.
### Root cause of the dead runner
`act_runner` registers **once** and persists its identity in `/data/.runner` on the
`act-runner-data` PVC. Gitea no longer held a matching row, so the pod booted, saw `.runner`,
**skipped registration**, and polled with an identity Gitea did not recognise. Symptom: zero
runners, pod Running, nothing crashlooping, no error anyone reads. Deleting the PVC forced a
genuine re-registration (`.runner is missing` → `Runner registered successfully`).
It then did not come back for several minutes, because Flux could not recreate the PVC:
`apps-wave-1` was blocked four levels deep on a **stale** `DependencyNotReady` for
`platform-config`, which itself read `Ready=True`. Each wave re-evaluates only on its own
`interval: 10m`, so a resource deleted at the bottom of the chain can take ~40 min to return.
**Force it** with `flux reconcile kustomization apps-wave-1 --with-source` rather than waiting.
### Corrections to things this log and the runbooks asserted
- **A skipped Gitea job reports `success`, not pending.** Measured: `gate / conformance` returned
`success` / "Has been skipped". An earlier entry (#2) said requiring it would deadlock every PR.
Wrong — it would pass **vacuously**, forever, while nothing ran. Under HARD RULE 4 that is worse
than a deadlock, because a deadlock is loud. Never require it; require
`gate / gate (pull_request)` alone. Corrected in #3.
- **`gate.yml`'s push trigger omits `feat/**`** (it lists `feature/** fix/** docs/** chore/**`).
With `act_runner` capacity 1, a branch prefix that IS in that list fires a second redundant run
which serialises ahead of the PR run and doubles wall-clock. Exploited deliberately in
`infra/plex/claim-ship.sh`; the honest fix is to align the trigger list.
- **The "~4-minute TTL"** asserted in four places was **uncited**, and upstream `pms-docker`
documents no expiry at all. The authoritative reading is the countdown plex.tv/claim renders at
issue time. Removed from `runbooks/plex-claim.md`; the same claim still sits in
`catalog/plex.yaml` and the Secret's header comment (left alone — editing a `.sops.yaml`
comment risks the MAC, and touching `catalog/plex.yaml` re-renders a golden).
### Measured, once the pipeline actually worked
`gate / gate (pull_request)` 1m34s–1m58s. Full claim loop end to end **~2m45s** — comfortably
inside the window, which retires the theory that a short-lived token cannot survive a PR flow.
That theory was mine and it was wrong: the pipeline was never too slow, it simply did not exist.
### State now
- `main` **protected**, `user_can_push:false`. Runner `incluster-dind` online.
- `CLAUDE.md` is the agent delivery contract; `infra/gitea/` holds the protection + runner scripts.
- **All 37 runbooks** that instructed `git push origin main` now describe the PR flow — that
instruction had become actively blocked, not merely stale.
- `infra/plex/claim-ship.sh` drives the claim loop end to end.
### Next steps
1. **`enable_status_check: true`** with `gate / gate (pull_request)` as the only context. The
prerequisite is met: the gate has been observed green on real PRs.
2. Add `jakman85` as a repo **collaborator**, then `required_approvals: 1`. (Currently the only
org member is `gitea-admin`, which agents authenticate as — requiring approval before that is
fixed would leave nobody able to approve.)
3. Run the **push probe** against protection: `gitea-admin` is a site admin and Gitea may let it
bypass. `user_can_push:false` is the API's opinion, not proof.
4. Align `gate.yml`'s push trigger with this repo's `feat/**` prefix.
5. Convert the `dind` container to a **native sidecar** with a `startupProbe`, per the standing
FOLLOW-UP note in `apps/wave-1/gitea-actions.yaml`. It deletes both hand-rolled wait loops and
the envsubst hazard. Land it ALONE — it is the merge gate for the whole repo.
6. Create the Plex libraries over `/media/movies`, `/media/tv`, `/media/photos`.
+93
View File
@@ -0,0 +1,93 @@
#!/usr/bin/env bash
# plex-claim-ship.sh — run this IMMEDIATELY AFTER editing the SOPS secret with a fresh
# plex.tv claim token. It drives the whole CI/CD loop end to end and prints a running
# stopwatch so you can see the budget being spent.
#
# branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verify
#
# It never sees, prints, or handles the token: the file is already encrypted by sops.
set -euo pipefail
HOST=gitea.home.keithjackson.dev
OWNER=Home-Network
REPO=home-network-k3s-impl
SECRET=apps/wave-0/plex-claim-secret.sops.yaml
T0=$(date +%s)
el() { printf '\n[T+%02d:%02d] %s\n' $(( ($(date +%s)-T0)/60 )) $(( ($(date +%s)-T0)%60 )) "$1"; }
cd "$(git rev-parse --show-toplevel)"
# --- preflight: refuse to ship a file that is not what we think it is -------------
git diff --quiet -- "$SECRET" && { echo "ERROR: $SECRET is unchanged. Edit it first:"; \
echo " infra/devcontainer/dev.sh run sops $SECRET"; exit 1; }
grep -q 'PLEX_CLAIM: ENC\[' "$SECRET" || { echo "ERROR: $SECRET is not encrypted. REFUSING."; exit 1; }
[ -z "$(git status --porcelain | grep -v " $SECRET\$")" ] || echo "NOTE: other files are dirty; only $SECRET will be committed."
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
# feat/ prefix ON PURPOSE. act_runner capacity is 1, and gate.yml's PUSH trigger covers
# feature/** fix/** docs/** chore/** but NOT feat/** — so a chore/ branch would fire a
# second, redundant run that SERIALISES ahead of the PR run and doubles the wall clock.
BRANCH="feat/plex-claim-token-$(date +%H%M%S)"
el "committing on $BRANCH"
git checkout -q -b "$BRANCH"
git add "$SECRET"
git commit -q -m 'chore(plex): fresh plex.tv claim token
Injected inside the plex.tv claim window per runbooks/plex-claim.md.'
SHA=$(git rev-parse HEAD)
el "pushing"
git push -q -u origin "$BRANCH"
el "opening PR"
PR=$(jq -n --arg h "$BRANCH" --arg b main --arg t 'chore(plex): fresh plex.tv claim token' \
'{head:$h,base:$b,title:$t,body:"Token injection inside the claim window. Wiring already reviewed in #1."}' \
| api -X POST -H 'Content-Type: application/json' --data @- \
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls" | jq -r '.number')
[ -n "$PR" ] && [ "$PR" != "null" ] || { echo "PR creation failed"; exit 1; }
echo " PR #$PR"
el "waiting for the gate (expect ~1m40s)"
for i in $(seq 1 60); do
st=$(api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/commits/${SHA}/status" | jq -r '.state')
printf '\r gate=%s [T+%ss] ' "$st" "$(( $(date +%s)-T0 ))"
case "$st" in success) echo; break;; failure|error) echo; echo "GATE FAILED — stopping."; exit 1;; esac
sleep 5
done
[ "$st" = "success" ] || { echo; echo "gate did not finish in time"; exit 1; }
el "merging PR #$PR"
api -X POST -H 'Content-Type: application/json' -d '{"Do":"merge"}' \
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls/${PR}/merge" >/dev/null
git checkout -q main && git pull -q --ff-only origin main
el "forcing the Flux reconcile (--with-source, or it applies the OLD revision)"
./infra/devcontainer/dev.sh run bash -c \
'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-0 --with-source'
# MANDATORY here. Only the Secret ciphertext changed, so the Deployment pod template is
# byte-identical and NOTHING restarts on its own. PLEX_CLAIM is read once, at container
# start. Skipping this is failure mode #2 in runbooks/plex-claim.md.
el "rolling the pod (mandatory — a changed Secret restarts nothing)"
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex rollout restart deployment/plex && \
kubectl --context k3s-prod -n plex rollout status deployment/plex --timeout=120s'
el "verdict"
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex logs deploy/plex --since=3m | grep -i -e claim -e token || true'
cat <<'NOTE'
"Attempting to obtain server token..." + "Token obtained successfully" -> CLAIMED
"Attempting..." with no success line -> token expired; retry, nothing is broken
neither line -> already claimed, or env did not reach the container
Confirm it persisted (prints the attribute NAME only, never its value):
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c \
"grep -o PlexOnlineToken= \"/config/Library/Application Support/Plex Media Server/Preferences.xml\" | head -1"'
NOTE
+7 -4
View File
@@ -33,7 +33,8 @@ negative is how a CLI quirk becomes real data loss. Write `backups.velero.io`,
`backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`,
`volumes.longhorn.io`, `nodes.longhorn.io`. `volumes.longhorn.io`, `nodes.longhorn.io`.
`git push origin main` **is** the deploy; there is no other apply step. Durable changes go through git. Merging a PR into `main` **is** the deploy; there is no other apply step. `main` is protected, so a
direct push is rejected — durable changes go through a branch, a PR, and a green gate.
## Alerts you should not trust the silence of (measured 2026-09-13) ## Alerts you should not trust the silence of (measured 2026-09-13)
@@ -109,9 +110,11 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
every green signal missed. every green signal missed.
### Operator procedures (no alert links here) ### Operator procedures (no alert links here)
- **`plex-claim`** — claiming the Plex Media Server. The plex.tv claim token lives ~4 minutes, so this - **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only
is a timed sequence: token → `sops` → push → **forced** `flux reconcile` → pod roll. Includes why a applies after a `/config` wipe. The claim token is short-lived — read the countdown on
changed Secret does not restart anything, and why an expired token is harmless. plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
Includes why a changed Secret does not restart anything, and why an expired token is harmless.
### Virtualisation ### Virtualisation
- **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running. - **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.
+4 -2
View File
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -78,8 +78,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -55,8 +55,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -63,8 +63,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -87,8 +87,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -69,8 +69,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -71,8 +71,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -68,8 +68,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+5 -3
View File
@@ -7,7 +7,7 @@
## What this means here ## What this means here
Something in the GitOps chain is stuck at Ready=False. On this cluster that is more consequential than Something in the GitOps chain is stuck at Ready=False. On this cluster that is more consequential than
it sounds, because **`git push origin main` IS the deploy** — there is no other apply step. A stuck it sounds, because **merging a PR into `main` IS the deploy** — there is no other apply step. A stuck
Kustomization means git and the cluster have diverged and nothing will close the gap. Kustomization means git and the cluster have diverged and nothing will close the gap.
The chain has **14 tiers**: `flux-system` → `cilium` → `platform` → `platform-config` → The chain has **14 tiers**: `flux-system` → `cilium` → `platform` → `platform-config` →
@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -58,8 +58,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -87,8 +87,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -67,8 +67,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -59,8 +59,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -57,8 +57,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+1 -1
View File
@@ -83,7 +83,7 @@ configMapGenerator:
- node-memory-swap=node-memory-swap.md - node-memory-swap=node-memory-swap.md
# Operator procedure, not an alert page (no alert links to it — the # Operator procedure, not an alert page (no alert links to it — the
# alert-runbooks-resolvable check treats that as advisory). It is served # alert-runbooks-resolvable check treats that as advisory). It is served
# because the 4-minute plex.tv claim window is not a thing to go looking # because the plex.tv claim window is short and is not a thing to go looking
# for in git: /runbooks/plex-claim has to resolve the moment it is needed. # for in git: /runbooks/plex-claim has to resolve the moment it is needed.
- plex-claim=plex-claim.md - plex-claim=plex-claim.md
- pvc-capacity=pvc-capacity.md - pvc-capacity=pvc-capacity.md
+4 -2
View File
@@ -83,8 +83,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -74,8 +74,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -75,8 +75,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -52,8 +52,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -49,8 +49,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -56,8 +56,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -56,8 +56,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -227,8 +227,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -52,8 +52,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+59 -55
View File
@@ -1,4 +1,4 @@
# plex-claim — claiming the Plex Media Server inside the 4-minute token window # plex-claim — claiming the Plex Media Server inside the token window
**Alert:** none. This is an **operator procedure**, not an incident page — it is indexed here because **Alert:** none. This is an **operator procedure**, not an incident page — it is indexed here because
it is the page you need at the moment you need it, with a stopwatch running. it is the page you need at the moment you need it, with a stopwatch running.
@@ -14,10 +14,11 @@ server to a plex.tv account. It is a **one-shot** operation: the token is exchan
the `PLEX_CLAIM` env is **ignored on every start** (upstream gate: the `PLEX_CLAIM` env is **ignored on every start** (upstream gate:
`[ ! -z "${PLEX_CLAIM}" ] && [ -z "${token}" ]` in `root/etc/cont-init.d/40-plex-first-run`). `[ ! -z "${PLEX_CLAIM}" ] && [ -z "${token}" ]` in `root/etc/cont-init.d/40-plex-first-run`).
**A `https://plex.tv/claim` token lives ~4 minutes.** That is shorter than commit → push → Flux's **A `https://plex.tv/claim` token is short-lived — minutes, not hours.** Read the countdown the
10-minute reconcile interval → pod roll if you let those run on their own timers, which is why this claim page shows you; do not budget against a number written here. That is shorter than PR → gate →
page exists: every step below is written to be run back-to-back, with the reconcile **forced** rather merge → Flux's 10-minute reconcile interval → pod roll if you let those run on their own timers,
than waited for. Budget: token in hand at T+0, pod restarted by roughly T+2:00. which is why this page exists: the loop is scripted and the reconcile is **forced** rather than
waited for. Measured end to end on 2026-09-29: **~2m45s**.
**An expired or wrong token does not break anything.** Verified against the image's own first-run **An expired or wrong token does not break anything.** Verified against the image's own first-run
script (and the copy inside `plexinc/pms-docker:latest`): a failed exchange gets a 403 with an empty script (and the copy inside `plexinc/pms-docker:latest`): a failed exchange gets a 403 with an empty
@@ -66,81 +67,84 @@ infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0' 'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0'
``` ```
## The 4-minute procedure ## The procedure
Have this page open and do not stop between steps. > **STATUS: this server was CLAIMED on 2026-09-29** and `PlexOnlineToken` is present in
> `/config` on the `plex-config` PVC. The upstream exchange is gated on that pref being
> *absent*, so `PLEX_CLAIM` is now a **no-op on every start** whatever it holds. You only need
> this page again after a `/config` wipe or a rebuild onto an empty PVC.
**On the clock.** The token is short-lived — **read the countdown that plex.tv/claim shows you**
rather than trusting a number written here. (Earlier revisions of this page asserted a "~4-minute
TTL" in four places with no citation; upstream `pms-docker` documents no expiry at all. The page
is the only authority, and it is on screen at T+0.)
**A burnt token costs nothing.** An expired token gets a 403 with an empty body, `curl`'s status is
never checked, the `if` does not fire, there is no `else` and no `exit 1`, and the script (no
`set -e`) exits 0. The pod comes up unclaimed and serving. So retry freely.
### One command
1. **T+0 — get the token.** Sign in to plex.tv as the account that should OWN this server, open 1. **T+0 — get the token.** Sign in to plex.tv as the account that should OWN this server, open
<https://plex.tv/claim>, copy the `claim-…` string. The clock starts now. <https://plex.tv/claim>, copy the `claim-…` string.
2. **T+0:15 — put it in the Secret**, in place, encrypted, without it ever touching a plaintext file: 2. **Put it in the Secret** — in place, encrypted, never touching a plaintext file:
``` ```
infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
``` ```
Replace the `PLEX_CLAIM:` value (the placeholder `replace-me-with-a-fresh-plex-tv-claim-token`, or 3. **Ship it.** This drives the whole delivery loop and prints a running stopwatch:
a previous expired token) and save. `sops` re-encrypts on write to the committed age recipient.
3. **T+0:45 — commit and push.** `git push origin main` **is** the deploy; there is no other apply
step.
``` ```
git add apps/wave-0/plex-claim-secret.sops.yaml ./infra/plex/claim-ship.sh
git commit -m 'chore(plex): fresh plex.tv claim token'
git push origin main
``` ```
4. **T+1:00 — force the reconcile immediately. This is the one command that saves the window** — it branch → commit → push → PR → gate → merge → `flux reconcile --with-source` → `rollout restart`
pulls the new commit and re-applies wave 0 instead of waiting up to 10 minutes for the interval: → verdict. It refuses to run if the file is unchanged or not encrypted.
``` **Measured end to end, 2026-09-29: ~2m45s.** Gate ~1m40s, merge + reconcile ~25s, pod roll and
infra/devcontainer/dev.sh run bash -c \ container start ~30s.
'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-0 --with-source'
```
`--with-source` is the load-bearing flag: it reconciles the `flux-system` GitRepository first, so ### Why it is a script and not six pasted commands
the Kustomization applies **your commit** and not the previously-fetched revision.
5. **T+1:30 — roll the pod.** This step is **mandatory and easy to forget**: a changed Secret does `main` is a **protected branch** — the direct push this page used to instruct is now rejected — so
**not** restart anything. `PLEX_CLAIM` is injected by `secretKeyRef` and read only at container the deploy is a PR merged after a green gate. Three of those steps are easy to get wrong under time
start, and the Deployment's pod template is byte-identical across this change, so Flux applies the pressure, and two are silent when skipped:
new Secret and the running pod keeps the old value forever.
``` - **`--with-source` is load-bearing.** Without it the Kustomization applies the previously-fetched
infra/devcontainer/dev.sh run bash -c \ revision, not your commit.
'kubectl --context k3s-prod -n plex rollout restart deployment/plex' - **The pod roll is mandatory.** Only the Secret's ciphertext changes, so the Deployment's pod
``` template is byte-identical and *nothing restarts on its own*. `PLEX_CLAIM` is read once, at
container start. Skip this and the new Secret sits in the cluster while the running pod keeps the
old value forever.
- **The branch is named `feat/…` on purpose.** `act_runner` capacity is 1, and `gate.yml`'s push
trigger covers `feature/** fix/** docs/** chore/**` but *not* `feat/**`. A `chore/` branch fires a
second redundant run that serialises ahead of the PR run and roughly doubles the wall clock.
6. **T+2:00 — confirm the exchange happened**, in the new pod's init log: ### Reading the verdict
``` * `Attempting to obtain server token from claim token` **and** `Token obtained successfully`
infra/devcontainer/dev.sh run bash -c \ ⇒ **claimed.** Re-run the First-commands `PlexOnlineToken=` check to confirm it persisted, then
'kubectl --context k3s-prod -n plex logs deploy/plex --since=5m | grep -i -e claim -e token' create the libraries over `/media/movies`, `/media/tv`, `/media/photos`.
``` * `Attempting…` with **no** success line ⇒ the token was expired or invalid. Nothing is broken; go
back to step 1 with a new one.
* **Neither line** ⇒ the exchange was skipped: `/config` already had a `PlexOnlineToken` (already
claimed — the expected case now), or the env did not reach the container. Check by NAME only:
* `Attempting to obtain server token from claim token` **and** `Token obtained successfully` ```
⇒ **claimed.** Re-run the First-commands `PlexOnlineToken=` check to confirm it persisted, then infra/devcontainer/dev.sh run bash -c \
create the libraries over `/media/movies`, `/media/tv`, `/media/photos`. 'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
* `Attempting to obtain server token from claim token` with **no** success line ⇒ the token was -o jsonpath="{.items[0].spec.containers[0].env[*].name}"'
expired or invalid. Nothing is broken; go back to step 1 with a new token. ```
* **Neither line** ⇒ the exchange was skipped, which means `/config` already had a
`PlexOnlineToken` (already claimed) or the env did not reach the container. Check the pod actually
has the variable, by NAME only:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
-o jsonpath="{.items[0].spec.containers[0].env[*].name}"'
```
## Likely causes, most likely first ## Likely causes, most likely first
When the claim does not take, in order of how often it is the answer here: When the claim does not take, in order of how often it is the answer here:
1. **The token expired** — you spent more than ~4 minutes, usually in step 3 or waiting on step 4. 1. **The token expired** — you exceeded the countdown, usually by reviewing the PR instead of
Symptom: the "Attempting…" line with no success line. Remedy: a new token, and force the reconcile merging it the moment the gate went green. Symptom: the "Attempting…" line with no success line.
rather than waiting for it. Remedy: a new token, and let `claim-ship.sh` drive the loop rather than pasting steps.
2. **The pod was never restarted** (step 5 skipped). Symptom: no claim log lines at all from a pod 2. **The pod was never restarted** (step 5 skipped). Symptom: no claim log lines at all from a pod
whose `AGE` predates your push. The Secret is correct in git and in the cluster and the running whose `AGE` predates your push. The Secret is correct in git and in the cluster and the running
container still holds the placeholder. container still holds the placeholder.
+4 -2
View File
@@ -58,8 +58,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -60,8 +60,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -54,8 +54,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands
+4 -2
View File
@@ -66,8 +66,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy). agents here: every durable change goes through git. **`main` is a protected branch — a direct push
Where this runbook gives a mutating command it is for an operator, and it says so. is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands ## First commands