home-network-k3s-impl/runbooks/plex-claim.md

plex-claim — claiming the Plex Media Server inside the 4-minute token window

Alert: none. This is an operator procedure, not an incident page — it is indexed here because it is the page you need at the moment you need it, with a stopwatch running. Files: catalog/plex.yaml (env PLEX_CLAIM), apps/wave-0/plex-claim-secret.sops.yaml (the SOPS Secret plex-claim, key PLEX_CLAIM), apps/wave-0/plex.yaml (the rendered golden).

What this means here

Plex will not let you create libraries over the mounted bulk media share while the server is unclaimed — the 8.33 T NFS tree at /media is there, readable, and unusable. Claiming binds the server to a plex.tv account. It is a one-shot operation: the token is exchanged once for a durable PlexOnlineToken written into Preferences.xml on the plex-config Longhorn PVC, and from then on the PLEX_CLAIM env is ignored on every start (upstream gate: [ ! -z "${PLEX_CLAIM}" ] && [ -z "${token}" ] in root/etc/cont-init.d/40-plex-first-run).

A https://plex.tv/claim token lives ~4 minutes. That is shorter than commit → push → Flux's 10-minute reconcile interval → pod roll if you let those run on their own timers, which is why this page exists: every step below is written to be run back-to-back, with the reconcile forced rather than waited for. Budget: token in hand at T+0, pod restarted by roughly T+2:00.

An expired or wrong token does not break anything. Verified against the image's own first-run script (and the copy inside plexinc/pms-docker:latest): a failed exchange gets a 403 with an empty body, curl's status is never checked, the token sed yields empty, the if does not fire, there is no else and no exit 1, and the script (no set -e) exits 0. The pod comes up unclaimed and serving. So a burnt token costs you one more attempt, not an outage — and the repo deliberately ships a placeholder in that Secret for exactly this reason (the env secretKeyRef has no optional, so the Secret must always exist even when its value is useless).

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once (k3s-dev until dev was decommissioned on 2026-09-13, k3s-prod now). Do not run two dev.sh invocations at once.

Mutating kubectl and flux reconcile are blocked for agents here. This whole page is operator work — the reconcile and the pod roll below are deliberate operator actions, and the durable change is still a git change.

First commands

Before spending a token, find out whether the server is already claimed. This prints only the attribute NAME, never its value — do not widen it to print the token:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c \
     "grep -o PlexOnlineToken= \"/config/Library/Application Support/Plex Media Server/Preferences.xml\" | head -1"'

PlexOnlineToken= printed ⇒ already claimed; stop, there is nothing to do and no token to burn. Empty output ⇒ unclaimed, continue.

Is the workload healthy enough to claim at all, and is the wave reconciling?

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex get deploy,pod,pvc'
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0'

The 4-minute procedure

Have this page open and do not stop between steps.

1. T+0 — get the token. Sign in to plex.tv as the account that should OWN this server, open https://plex.tv/claim, copy the claim-… string. The clock starts now.

2. T+0:15 — put it in the Secret, in place, encrypted, without it ever touching a plaintext file:

   infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
   

   Replace the PLEX_CLAIM: value (the placeholder replace-me-with-a-fresh-plex-tv-claim-token, or a previous expired token) and save. sops re-encrypts on write to the committed age recipient.

3. T+0:45 — commit and push. git push origin main is the deploy; there is no other apply step.

   git add apps/wave-0/plex-claim-secret.sops.yaml
   git commit -m 'chore(plex): fresh plex.tv claim token'
   git push origin main
   

4. T+1:00 — force the reconcile immediately. This is the one command that saves the window — it pulls the new commit and re-applies wave 0 instead of waiting up to 10 minutes for the interval:

   infra/devcontainer/dev.sh run bash -c \
     'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-0 --with-source'
   

   --with-source is the load-bearing flag: it reconciles the flux-system GitRepository first, so the Kustomization applies your commit and not the previously-fetched revision.

5. T+1:30 — roll the pod. This step is mandatory and easy to forget: a changed Secret does not restart anything. PLEX_CLAIM is injected by secretKeyRef and read only at container start, and the Deployment's pod template is byte-identical across this change, so Flux applies the new Secret and the running pod keeps the old value forever.

   infra/devcontainer/dev.sh run bash -c \
     'kubectl --context k3s-prod -n plex rollout restart deployment/plex'
   

6. T+2:00 — confirm the exchange happened, in the new pod's init log:

   infra/devcontainer/dev.sh run bash -c \
     'kubectl --context k3s-prod -n plex logs deploy/plex --since=5m | grep -i -e claim -e token'
   

   • Attempting to obtain server token from claim token and Token obtained successfully ⇒ claimed. Re-run the First-commands PlexOnlineToken= check to confirm it persisted, then create the libraries over /media/movies, /media/tv, /media/photos.

   • Attempting to obtain server token from claim token with no success line ⇒ the token was expired or invalid. Nothing is broken; go back to step 1 with a new token.

   • Neither line ⇒ the exchange was skipped, which means /config already had a PlexOnlineToken (already claimed) or the env did not reach the container. Check the pod actually has the variable, by NAME only:

     infra/devcontainer/dev.sh run bash -c \
       'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
          -o jsonpath="{.items[0].spec.containers[0].env[*].name}"'
     

Likely causes, most likely first

When the claim does not take, in order of how often it is the answer here:

1. The token expired — you spent more than ~4 minutes, usually in step 3 or waiting on step 4. Symptom: the "Attempting…" line with no success line. Remedy: a new token, and force the reconcile rather than waiting for it.
2. The pod was never restarted (step 5 skipped). Symptom: no claim log lines at all from a pod whose AGE predates your push. The Secret is correct in git and in the cluster and the running container still holds the placeholder.
3. Flux had not applied the commit yet when you rolled the pod — step 5 run before step 4 finished, so the new pod read the OLD Secret. Check kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0 -o jsonpath='{.status.lastAppliedRevision}' against your commit, then roll again.
4. Already claimed — the exchange is gated on PlexOnlineToken being absent from /config, so a server claimed earlier ignores every later token silently. This is the benign case.
5. plex.tv unreachable. The upstream curl has no --max-time/--connect-timeout, so a stalled connection can leave the container sitting in cont-init with nothing listening on 32400 — the one genuine hang risk in this path. Symptom: pod Running but never Ready, log stops after "Attempting…". Check egress (plex is egress: internet) rather than re-rolling repeatedly.
6. Wrong plex.tv account. The claim succeeds and binds the server to whoever was signed in at step 1. Unclaiming is a plex.tv-side operation on that account — see "escalate or stop".

When to escalate or stop

• Stop after two failed token attempts and fix the timing, not the token. The third rushed attempt is how people start editing live Secrets by hand, which leaves the cluster holding a value that is in no git commit — precisely the drift qbittorrent-bootstrap had to be recovered from.
• Never kubectl edit/patch the plex-claim Secret in the cluster, and never kubectl create secret it. Everything durable goes through the SOPS file; a live-only Secret is reverted (or adopted with the wrong value) on the next reconcile.
• Never sops -d this file into a terminal, a log, or a shared screen, and never widen the PlexOnlineToken= check into one that prints the value. The token is a plex.tv account credential.
• If the server got claimed to the wrong account, stop and get the operator. Reversing it means removing the server from that account on plex.tv and clearing PlexOnlineToken from Preferences.xml on the PVC — a destructive edit of app state on the replicated tier, not a git change.
• Do not delete the plex-config PVC to "start clean". It holds the library database, metadata and watch state (policy: daily-offcluster), and it is expensive to rebuild. The claim is a two-line pref, not a reason to lose the DB.
• After a successful claim, leave the expired token in the SOPS file. It is inert (the exchange is skipped once claimed) and the Secret must keep existing, because the env secretKeyRef has no optional — deleting it would put the pod into CreateContainerConfigError.

---

Source of truth: runbooks/plex-claim.md. Index: /runbooks/ Upstream behaviour cited here is from plexinc/pms-docker branch master, root/etc/cont-init.d/40-plex-first-run (s6-overlay v2), cross-checked against the same file inside the running plexinc/pms-docker:latest image.