home-network-k3s-impl/runbooks/plex-claim.md

# plex-claim — claiming the Plex Media Server inside the 4-minute token window

**Alert:** none. This is an **operator procedure**, not an incident page — it is indexed here because
it is the page you need at the moment you need it, with a stopwatch running.
**Files:** `catalog/plex.yaml` (env `PLEX_CLAIM`), `apps/wave-0/plex-claim-secret.sops.yaml`
(the SOPS Secret `plex-claim`, key `PLEX_CLAIM`), `apps/wave-0/plex.yaml` (the rendered golden).

## What this means here

Plex will not let you **create libraries** over the mounted bulk media share while the server is
**unclaimed** — the 8.33 T NFS tree at `/media` is there, readable, and unusable. Claiming binds the
server to a plex.tv account. It is a **one-shot** operation: the token is exchanged once for a durable
`PlexOnlineToken` written into `Preferences.xml` on the `plex-config` Longhorn PVC, and from then on
the `PLEX_CLAIM` env is **ignored on every start** (upstream gate:
`[ ! -z "${PLEX_CLAIM}" ] && [ -z "${token}" ]` in `root/etc/cont-init.d/40-plex-first-run`).

**A `https://plex.tv/claim` token lives ~4 minutes.** That is shorter than commit → push → Flux's
10-minute reconcile interval → pod roll if you let those run on their own timers, which is why this
page exists: every step below is written to be run back-to-back, with the reconcile **forced** rather
than waited for. Budget: token in hand at T+0, pod restarted by roughly T+2:00.

**An expired or wrong token does not break anything.** Verified against the image's own first-run
script (and the copy inside `plexinc/pms-docker:latest`): a failed exchange gets a 403 with an empty
body, `curl`'s status is never checked, the token `sed` yields empty, the `if` does not fire, there is
no `else` and no `exit 1`, and the script (no `set -e`) exits 0. The pod comes up **unclaimed and
serving**. So a burnt token costs you one more attempt, not an outage — and the repo deliberately
ships a **placeholder** in that Secret for exactly this reason (the env `secretKeyRef` has no
`optional`, so the Secret must always exist even when its value is useless).

## Running the commands in this runbook

`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that
hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and that default has already changed once (k3s-dev until dev
was decommissioned on 2026-09-13, k3s-prod now). Do not run two `dev.sh` invocations at once.

Mutating `kubectl` and `flux reconcile` are blocked for agents here. **This whole page is operator
work** — the reconcile and the pod roll below are deliberate operator actions, and the durable change
is still a git change.

## First commands

**Before spending a token, find out whether the server is already claimed.** This prints only the
attribute NAME, never its value — do not widen it to print the token:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c \
     "grep -o PlexOnlineToken= \"/config/Library/Application Support/Plex Media Server/Preferences.xml\" | head -1"'
```

`PlexOnlineToken=` printed ⇒ **already claimed**; stop, there is nothing to do and no token to burn.
Empty output ⇒ unclaimed, continue.

Is the workload healthy enough to claim at all, and is the wave reconciling?

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex get deploy,pod,pvc'
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0'
```

## The 4-minute procedure

Have this page open and do not stop between steps.

1. **T+0 — get the token.** Sign in to plex.tv as the account that should OWN this server, open
   <https://plex.tv/claim>, copy the `claim-…` string. The clock starts now.

2. **T+0:15 — put it in the Secret**, in place, encrypted, without it ever touching a plaintext file:

   ```
   infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
   ```

   Replace the `PLEX_CLAIM:` value (the placeholder `replace-me-with-a-fresh-plex-tv-claim-token`, or
   a previous expired token) and save. `sops` re-encrypts on write to the committed age recipient.

3. **T+0:45 — commit and push.** `git push origin main` **is** the deploy; there is no other apply
   step.

   ```
   git add apps/wave-0/plex-claim-secret.sops.yaml
   git commit -m 'chore(plex): fresh plex.tv claim token'
   git push origin main
   ```

4. **T+1:00 — force the reconcile immediately. This is the one command that saves the window** — it
   pulls the new commit and re-applies wave 0 instead of waiting up to 10 minutes for the interval:

   ```
   infra/devcontainer/dev.sh run bash -c \
     'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-0 --with-source'
   ```

   `--with-source` is the load-bearing flag: it reconciles the `flux-system` GitRepository first, so
   the Kustomization applies **your commit** and not the previously-fetched revision.

5. **T+1:30 — roll the pod.** This step is **mandatory and easy to forget**: a changed Secret does
   **not** restart anything. `PLEX_CLAIM` is injected by `secretKeyRef` and read only at container
   start, and the Deployment's pod template is byte-identical across this change, so Flux applies the
   new Secret and the running pod keeps the old value forever.

   ```
   infra/devcontainer/dev.sh run bash -c \
     'kubectl --context k3s-prod -n plex rollout restart deployment/plex'
   ```

6. **T+2:00 — confirm the exchange happened**, in the new pod's init log:

   ```
   infra/devcontainer/dev.sh run bash -c \
     'kubectl --context k3s-prod -n plex logs deploy/plex --since=5m | grep -i -e claim -e token'
   ```

   * `Attempting to obtain server token from claim token` **and** `Token obtained successfully`
     ⇒ **claimed.** Re-run the First-commands `PlexOnlineToken=` check to confirm it persisted, then
     create the libraries over `/media/movies`, `/media/tv`, `/media/photos`.
   * `Attempting to obtain server token from claim token` with **no** success line ⇒ the token was
     expired or invalid. Nothing is broken; go back to step 1 with a new token.
   * **Neither line** ⇒ the exchange was skipped, which means `/config` already had a
     `PlexOnlineToken` (already claimed) or the env did not reach the container. Check the pod actually
     has the variable, by NAME only:

     ```
     infra/devcontainer/dev.sh run bash -c \
       'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
          -o jsonpath="{.items[0].spec.containers[0].env[*].name}"'
     ```

## Likely causes, most likely first

When the claim does not take, in order of how often it is the answer here:

1. **The token expired** — you spent more than ~4 minutes, usually in step 3 or waiting on step 4.
   Symptom: the "Attempting…" line with no success line. Remedy: a new token, and force the reconcile
   rather than waiting for it.
2. **The pod was never restarted** (step 5 skipped). Symptom: no claim log lines at all from a pod
   whose `AGE` predates your push. The Secret is correct in git and in the cluster and the running
   container still holds the placeholder.
3. **Flux had not applied the commit yet** when you rolled the pod — step 5 run before step 4
   finished, so the new pod read the OLD Secret. Check
   `kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0 -o jsonpath='{.status.lastAppliedRevision}'`
   against your commit, then roll again.
4. **Already claimed** — the exchange is gated on `PlexOnlineToken` being absent from `/config`, so a
   server claimed earlier ignores every later token silently. This is the benign case.
5. **`plex.tv` unreachable.** The upstream `curl` has no `--max-time`/`--connect-timeout`, so a stalled
   connection can leave the container sitting in cont-init with **nothing listening on 32400** — the
   one genuine hang risk in this path. Symptom: pod Running but never Ready, log stops after
   "Attempting…". Check egress (`plex` is `egress: internet`) rather than re-rolling repeatedly.
6. **Wrong plex.tv account.** The claim succeeds and binds the server to whoever was signed in at
   step 1. Unclaiming is a plex.tv-side operation on that account — see "escalate or stop".

## When to escalate or stop

- **Stop after two failed token attempts** and fix the timing, not the token. The third rushed attempt
  is how people start editing live Secrets by hand, which leaves the cluster holding a value that is
  in no git commit — precisely the drift `qbittorrent-bootstrap` had to be recovered from.
- **Never `kubectl edit`/`patch` the `plex-claim` Secret in the cluster, and never
  `kubectl create secret` it.** Everything durable goes through the SOPS file; a live-only Secret is
  reverted (or adopted with the wrong value) on the next reconcile.
- **Never `sops -d` this file into a terminal, a log, or a shared screen**, and never widen the
  `PlexOnlineToken=` check into one that prints the value. The token is a plex.tv account credential.
- **If the server got claimed to the wrong account, stop and get the operator.** Reversing it means
  removing the server from that account on plex.tv and clearing `PlexOnlineToken` from
  `Preferences.xml` on the PVC — a destructive edit of app state on the replicated tier, not a git
  change.
- **Do not delete the `plex-config` PVC to "start clean".** It holds the library database, metadata
  and watch state (`policy: daily-offcluster`), and it is expensive to rebuild. The claim is a
  two-line pref, not a reason to lose the DB.
- After a successful claim, **leave the expired token in the SOPS file**. It is inert (the exchange is
  skipped once claimed) and the Secret must keep existing, because the env `secretKeyRef` has no
  `optional` — deleting it would put the pod into `CreateContainerConfigError`.

---
Source of truth: `runbooks/plex-claim.md`. Index: `/runbooks/`
Upstream behaviour cited here is from `plexinc/pms-docker` branch `master`,
`root/etc/cont-init.d/40-plex-first-run` (s6-overlay v2), cross-checked against the same file inside
the running `plexinc/pms-docker:latest` image.