home-network-k3s-impl/runbooks/plex-claim.md

# plex-claim — claiming the Plex Media Server inside the token window

**Alert:** none. This is an **operator procedure**, not an incident page — it is indexed here because
it is the page you need at the moment you need it, with a stopwatch running.
**Files:** `catalog/plex.yaml` (env `PLEX_CLAIM`), `apps/wave-0/plex-claim-secret.sops.yaml`
(the SOPS Secret `plex-claim`, key `PLEX_CLAIM`), `apps/wave-0/plex.yaml` (the rendered golden).

## What this means here

Plex will not let you **create libraries** over the mounted bulk media share while the server is
**unclaimed** — the 8.33 T NFS tree at `/media` is there, readable, and unusable. Claiming binds the
server to a plex.tv account. It is a **one-shot** operation: the token is exchanged once for a durable
`PlexOnlineToken` written into `Preferences.xml` on the `plex-config` Longhorn PVC, and from then on
the `PLEX_CLAIM` env is **ignored on every start** (upstream gate:
`[ ! -z "${PLEX_CLAIM}" ] && [ -z "${token}" ]` in `root/etc/cont-init.d/40-plex-first-run`).

**A `https://plex.tv/claim` token is short-lived — minutes, not hours.** Read the countdown the
claim page shows you; do not budget against a number written here. That is shorter than PR → gate →
merge → Flux's 10-minute reconcile interval → pod roll if you let those run on their own timers,
which is why this page exists: the loop is scripted and the reconcile is **forced** rather than
waited for. Measured end to end on 2026-09-29: **~2m45s**.

**An expired or wrong token does not break anything.** Verified against the image's own first-run
script (and the copy inside the image then running as `plexinc/pms-docker:latest`; pinned to a version
tag since 2026-10-30): a failed exchange gets a 403 with an empty
body, `curl`'s status is never checked, the token `sed` yields empty, the `if` does not fire, there is
no `else` and no `exit 1`, and the script (no `set -e`) exits 0. The pod comes up **unclaimed and
serving**. So a burnt token costs you one more attempt, not an outage — and the repo deliberately
ships a **placeholder** in that Secret for exactly this reason (the env `secretKeyRef` has no
`optional`, so the Secret must always exist even when its value is useless).

## Running the commands in this runbook

`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that
hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and that default has already changed once (k3s-dev until dev
was decommissioned on 2026-09-13, k3s-prod now). Do not run two `dev.sh` invocations at once.

Mutating `kubectl` and `flux reconcile` are blocked for agents here. **This whole page is operator
work** — the reconcile and the pod roll below are deliberate operator actions, and the durable change
is still a git change.

## First commands

**Before spending a token, find out whether the server is already claimed.** This prints only the
attribute NAME, never its value — do not widen it to print the token:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c \
     "grep -o PlexOnlineToken= \"/config/Library/Application Support/Plex Media Server/Preferences.xml\" | head -1"'
```

`PlexOnlineToken=` printed ⇒ **already claimed**; stop, there is nothing to do and no token to burn.
Empty output ⇒ unclaimed, continue.

Is the workload healthy enough to claim at all, and is the wave reconciling?

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex get deploy,pod,pvc'
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0'
```

## The procedure

> **STATUS: this server was CLAIMED on 2026-09-29** and `PlexOnlineToken` is present in
> `/config` on the `plex-config` PVC. The upstream exchange is gated on that pref being
> *absent*, so `PLEX_CLAIM` is now a **no-op on every start** whatever it holds. You only need
> this page again after a `/config` wipe or a rebuild onto an empty PVC.

**On the clock.** The token is short-lived — **read the countdown that plex.tv/claim shows you**
rather than trusting a number written here. (Earlier revisions of this page asserted a "~4-minute
TTL" in four places with no citation; upstream `pms-docker` documents no expiry at all. The page
is the only authority, and it is on screen at T+0.)

**A burnt token costs nothing.** An expired token gets a 403 with an empty body, `curl`'s status is
never checked, the `if` does not fire, there is no `else` and no `exit 1`, and the script (no
`set -e`) exits 0. The pod comes up unclaimed and serving. So retry freely.

### One command

1. **T+0 — get the token.** Sign in to plex.tv as the account that should OWN this server, open
   <https://plex.tv/claim>, copy the `claim-…` string.

2. **Put it in the Secret** — in place, encrypted, never touching a plaintext file:

   ```
   infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
   ```

3. **Ship it.** This drives the whole delivery loop and prints a running stopwatch:

   ```
   ./infra/plex/claim-ship.sh
   ```

   branch → commit → push → PR → gate → merge → `flux reconcile --with-source` → `rollout restart`
   → verdict. It refuses to run if the file is unchanged or not encrypted.

**Measured end to end, 2026-09-29: ~2m45s.** Gate ~1m40s, merge + reconcile ~25s, pod roll and
container start ~30s.

### Why it is a script and not six pasted commands

`main` is a **protected branch** — the direct push this page used to instruct is now rejected — so
the deploy is a PR merged after a green gate. Three of those steps are easy to get wrong under time
pressure, and two are silent when skipped:

- **`--with-source` is load-bearing.** Without it the Kustomization applies the previously-fetched
  revision, not your commit.
- **The pod roll is mandatory.** Only the Secret's ciphertext changes, so the Deployment's pod
  template is byte-identical and *nothing restarts on its own*. `PLEX_CLAIM` is read once, at
  container start. Skip this and the new Secret sits in the cluster while the running pod keeps the
  old value forever.
- **The branch is named `feat/…` on purpose.** `act_runner` capacity is 1, and `gate.yml`'s push
  trigger covers `feature/** fix/** docs/** chore/**` but *not* `feat/**`. A `chore/` branch fires a
  second redundant run that serialises ahead of the PR run and roughly doubles the wall clock.

### Reading the verdict

* `Attempting to obtain server token from claim token` **and** `Token obtained successfully`
  ⇒ **claimed.** Re-run the First-commands `PlexOnlineToken=` check to confirm it persisted, then
  create the libraries over `/media/movies`, `/media/tv`, `/media/photos`.
* `Attempting…` with **no** success line ⇒ the token was expired or invalid. Nothing is broken; go
  back to step 1 with a new one.
* **Neither line** ⇒ the exchange was skipped: `/config` already had a `PlexOnlineToken` (already
  claimed — the expected case now), or the env did not reach the container. Check by NAME only:

  ```
  infra/devcontainer/dev.sh run bash -c \
    'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
       -o jsonpath="{.items[0].spec.containers[0].env[*].name}"'
  ```

## Likely causes, most likely first

When the claim does not take, in order of how often it is the answer here:

1. **The token expired** — you exceeded the countdown, usually by reviewing the PR instead of
   merging it the moment the gate went green. Symptom: the "Attempting…" line with no success line.
   Remedy: a new token, and let `claim-ship.sh` drive the loop rather than pasting steps.
2. **The pod was never restarted** (step 5 skipped). Symptom: no claim log lines at all from a pod
   whose `AGE` predates your push. The Secret is correct in git and in the cluster and the running
   container still holds the placeholder.
3. **Flux had not applied the commit yet** when you rolled the pod — step 5 run before step 4
   finished, so the new pod read the OLD Secret. Check
   `kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0 -o jsonpath='{.status.lastAppliedRevision}'`
   against your commit, then roll again.
4. **Already claimed** — the exchange is gated on `PlexOnlineToken` being absent from `/config`, so a
   server claimed earlier ignores every later token silently. This is the benign case.
5. **`plex.tv` unreachable.** The upstream `curl` has no `--max-time`/`--connect-timeout`, so a stalled
   connection can leave the container sitting in cont-init with **nothing listening on 32400** — the
   one genuine hang risk in this path. Symptom: pod Running but never Ready, log stops after
   "Attempting…". Check egress (`plex` is `egress: internet`) rather than re-rolling repeatedly.
6. **Wrong plex.tv account.** The claim succeeds and binds the server to whoever was signed in at
   step 1. Unclaiming is a plex.tv-side operation on that account — see "escalate or stop".

## When to escalate or stop

- **Stop after two failed token attempts** and fix the timing, not the token. The third rushed attempt
  is how people start editing live Secrets by hand, which leaves the cluster holding a value that is
  in no git commit — precisely the drift `qbittorrent-bootstrap` had to be recovered from.
- **Never `kubectl edit`/`patch` the `plex-claim` Secret in the cluster, and never
  `kubectl create secret` it.** Everything durable goes through the SOPS file; a live-only Secret is
  reverted (or adopted with the wrong value) on the next reconcile.
- **Never `sops -d` this file into a terminal, a log, or a shared screen**, and never widen the
  `PlexOnlineToken=` check into one that prints the value. The token is a plex.tv account credential.
- **If the server got claimed to the wrong account, stop and get the operator.** Reversing it means
  removing the server from that account on plex.tv and clearing `PlexOnlineToken` from
  `Preferences.xml` on the PVC — a destructive edit of app state on the replicated tier, not a git
  change.
- **Do not delete the `plex-config` PVC to "start clean".** It holds the library database, metadata
  and watch state (`policy: daily-offcluster`), and it is expensive to rebuild. The claim is a
  two-line pref, not a reason to lose the DB.
- After a successful claim, **leave the expired token in the SOPS file**. It is inert (the exchange is
  skipped once claimed) and the Secret must keep existing, because the env `secretKeyRef` has no
  `optional` — deleting it would put the pod into `CreateContainerConfigError`.

---
Source of truth: `runbooks/plex-claim.md`. Index: `/runbooks/`
Upstream behaviour cited here is from `plexinc/pms-docker` branch `master`,
`root/etc/cont-init.d/40-plex-first-run` (s6-overlay v2), cross-checked against the same file inside
the image that was running at the time as `plexinc/pms-docker:latest` (the deployment has been pinned
to a version tag, `catalog/plex.yaml` `source.image`, since 2026-10-30).