From 4fd75ccdaa50c38b5bd8d1c46263ec0c39bd65d2 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Mon, 28 Sep 2026 22:26:44 -0500 Subject: [PATCH] feat(plex): wire PLEX_CLAIM from a SOPS Secret so the server can be claimed Plex cannot create libraries over the already-mounted 8.33T NFS bulk share while the server is UNCLAIMED. catalog/plex.yaml carried a long argument for leaving PLEX_CLAIM unwired; the operator has decided to override it. This implements the wiring in the way that respects the half of that argument which is still true. WHAT WAS TRUE AND STILL IS: a https://plex.tv/claim token has a ~4-minute TTL, far shorter than commit -> push -> Flux reconcile -> pod-roll on its own timers. So the committed value is a documented PLACEHOLDER and the live token is injected by the operator inside that window, with the reconcile FORCED rather than waited for. runbooks/plex-claim.md is that timed procedure. WHAT WAS NOT TRUE: the old comment claimed "a placeholder/absent claim Secret would crashloop the pod". Half of that is wrong, and the difference is the whole design. Established from plexinc/pms-docker's own first-run script (branch `master`, root/etc/cont-init.d/40-plex-first-run, s6-overlay v2), cross-checked byte-identical against the same file inside the running :latest image: * the claim exchange (line 106) is gated on PLEX_CLAIM being non-empty AND the PlexOnlineToken pref in /config being empty. Once the server is claimed the block is skipped entirely, so the env is a NO-OP on every later start whatever it holds ("If server is already logged in, this parameter is ignored" -- upstream README). * with an invalid/expired token on a still-unclaimed /config, plex.tv answers 403 with an empty body; curl's status is never checked, the token sed yields empty, the success branch at line 120 does not fire, there is no else branch and no non-zero exit, and the script (no `set -e`) exits 0. The pod comes up UNCLAIMED AND SERVING. It does not crashloop. * the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS (default 0 = continue even if a cont-init script failed), so even a non-zero exit would not halt startup. An ABSENT Secret is the real hard failure (secretKeyRef has no `optional`, so it is CreateContainerConfigError), which is why the Secret is committed, encrypted and listed in the wave kustomization rather than hand-created at claim time. NO SCHEMA CHANGE. `optional` on secretKeyRef would only be needed if the pod had to tolerate a missing Secret or could not start without a VALID token. Neither holds: the Secret always exists in the Flux graph, and an invalid token is non-fatal. schemas/servicedefinition.schema.json is untouched, and so are catalog/render.py's env emission and the renderer contract. Changes: * apps/wave-0/plex-claim-secret.sops.yaml -- new Opaque Secret plex-claim/plex, key PLEX_CLAIM, SOPS-encrypted to the committed age recipient. Its header carries NO dollar-brace expressions on purpose: kustomize preserves body comments into the stream envsubst reads (per conformance/tests/l0-static/flux-postbuild-substitution), so quoting the upstream shell conditions verbatim would have been a live substitution hazard on apps-wave-0, not prose. * catalog/plex.yaml -- the PLEX_CLAIM env via valueFrom.secretKeyRef, the "deliberately NOT set" block rewritten around the TTL + one-shot design, and the stale crashloop claim in the /config storage comment corrected. * apps/wave-0/plex.yaml -- re-rendered with the canonical command (`--all catalog -o apps --wave-layout --no-image-automation`). No L1 golden under conformance/tests/l1-rendering covers plex, so this is the only rendered surface that moves. * catalog/render.py -- plan_apps_waves places `-claim-secret.sops.yaml` in the app's OWN wave, like the bootstrap secret, plus a selftest assertion. Without this, the next `--emit-waves` would have silently dropped the Secret from apps/wave-0/kustomization.yaml and put the pod into CreateContainerConfigError. apps/wave-0/kustomization.yaml is the hand-pruned subset (vm-demo/keycloak are rendered but deliberately unlisted), so its one new line was applied without re-emitting the rest. * runbooks/plex-claim.md + its kustomization.yaml generator entry + README index -- the operator procedure that fits in 4 minutes, including the single forced-reconcile command and the mandatory pod roll (a changed Secret does NOT restart anything: PLEX_CLAIM is read only at container start and the pod template is byte-identical across this change). Touched requirements: R-SEC-01 (secrets encrypted at rest), R-GITOPS-02 (ServiceDefinition onboarding contract). Co-Authored-By: Claude Opus 5 (1M context) --- apps/wave-0/kustomization.yaml | 1 + apps/wave-0/plex-claim-secret.sops.yaml | 69 +++++++++ apps/wave-0/plex.yaml | 5 + catalog/plex.yaml | 49 +++++-- catalog/render.py | 16 +- runbooks/README.md | 7 +- runbooks/kustomization.yaml | 5 + runbooks/plex-claim.md | 185 ++++++++++++++++++++++++ 8 files changed, 326 insertions(+), 11 deletions(-) create mode 100644 apps/wave-0/plex-claim-secret.sops.yaml create mode 100644 runbooks/plex-claim.md diff --git a/apps/wave-0/kustomization.yaml b/apps/wave-0/kustomization.yaml index eb649a6..50b5244 100644 --- a/apps/wave-0/kustomization.yaml +++ b/apps/wave-0/kustomization.yaml @@ -17,6 +17,7 @@ resources: - pihole-dns.yaml - pihole-oauth2-proxy-credentials.sops.yaml - pihole.yaml +- plex-claim-secret.sops.yaml - plex.yaml - qbittorrent-bootstrap-secret.sops.yaml - qbittorrent-oauth2-proxy-credentials.sops.yaml diff --git a/apps/wave-0/plex-claim-secret.sops.yaml b/apps/wave-0/plex-claim-secret.sops.yaml new file mode 100644 index 0000000..d4671f5 --- /dev/null +++ b/apps/wave-0/plex-claim-secret.sops.yaml @@ -0,0 +1,69 @@ +# Plex plex.tv server-claim token (WS-APPS). SOPS-encrypted (age). +# Consumed by catalog/plex.yaml as env PLEX_CLAIM via valueFrom.secretKeyRef. +# +# NOTE ON THESE COMMENTS: they carry NO dollar-brace expressions, deliberately. This file is applied +# by the apps-wave-0 Flux Kustomization, which runs envsubst over the built output, and kustomize +# preserves body comments into that stream (see conformance/tests/l0-static/flux-postbuild- +# substitution). A braced token quoted from the upstream shell script would be a live hazard here, +# not prose, so the upstream conditions below are described in words. +# +# THE VALUE SHIPPED WITH THIS FILE IS A PLACEHOLDER, DELIBERATELY — the literal string +# `replace-me-with-a-fresh-plex-tv-claim-token`, not a credential. A https://plex.tv/claim token has +# a ~4-minute TTL, far shorter than commit -> push -> Flux reconcile -> pod-roll, so no token +# committed here is ever still valid when the pod starts. This file exists to make the Secret — and +# therefore the env reference — ALWAYS RESOLVABLE IN THE FLUX GRAPH, not to carry a working token. +# The real token is injected by the operator inside that 4-minute window: runbooks/plex-claim.md. +# +# WHY SHIPPING A PLACEHOLDER IS SAFE (verified against plexinc/pms-docker branch `master`, the +# s6-overlay v2 script root/etc/cont-init.d/40-plex-first-run, cross-checked byte-identical against +# the same file extracted from the running `plexinc/pms-docker:latest` image): +# - line 106 guards the claim exchange on TWO conditions: PLEX_CLAIM non-empty AND the +# PlexOnlineToken pref read out of /config empty. Once the server is claimed that block is +# skipped entirely, so whatever this Secret holds is a NO-OP on every later start. Upstream +# README: "If server is already logged in, this parameter is ignored." +# - on a still-UNCLAIMED /config, an invalid or expired token makes plex.tv answer 403 with an +# empty body. curl's exit status is never checked, the sed for the authentication-token element +# yields the empty string, and the success branch at line 120 simply does not fire. There is no +# else branch and no non-zero exit; the script has no `set -e` and ends by touching +# /.firstRunComplete and echoing, so it exits 0. The server comes up UNCLAIMED and serving. +# It does NOT crashloop. +# - the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS, whose default is 0 ("continue silently even +# if a cont-init script failed"), so even a hypothetical non-zero exit would not halt startup. +# A MISSING Secret would NOT be safe: the ServiceDefinition schema's valueFrom.secretKeyRef shape +# has no `optional` field, so an absent Secret is a hard CreateContainerConfigError. That is exactly +# why this file is committed and listed in apps/wave-0/kustomization.yaml instead of being created +# by hand at claim time. +# +# INJECTION / ROTATION (operator, not agents) — runbooks/plex-claim.md has the timed procedure: +# infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml +# Never decrypt this file into a shell, a log, or a shared terminal. +apiVersion: v1 +kind: Secret +metadata: + name: plex-claim + namespace: plex + labels: + app.kubernetes.io/name: plex +type: Opaque +stringData: + PLEX_CLAIM: ENC[AES256_GCM,data:8O9nvFw+6OR+53mXvHfe321aSmYUQcZuAqLKZpcSlUwNyeU+Nbp/tRPBlg==,iv:EbvvOsXlXjAFOewnWbK28ErmUdWcJJN0Q5Gm53El3ZM=,tag:wXaInGFQ+wqUt765vvjCDA==,type:str] +sops: + kms: [] + gcp_kms: [] + azure_kv: [] + hc_vault: [] + age: + - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg + enc: | + -----BEGIN AGE ENCRYPTED FILE----- + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2dWtqOVNQalZ2QlRxOXJa + cDc1MTdtWENkdWIxS1JId3dIU1JDSmk0ODNrCk1YUzhFWXR3bUxyV2NNcmFOakdz + THl1Q284dFJxTy9Rd1h0eXJJaVZtdmcKLS0tIERuN0ZFeW9zdi9aNFRYNWxmUS9s + QUtYdHEvN042ajNoNktOLzlYQmQrc1kK4NHozMbXVzNGYbUBRbK6a9Uw/sVPAfe6 + RW5KWzZ4I/hloqjN3zzVISoSMieMkBGDnF5qZS5N6FqIc6/HrsDD8A== + -----END AGE ENCRYPTED FILE----- + lastmodified: "2026-09-29T03:25:20Z" + mac: ENC[AES256_GCM,data:u5uVmOdXN78UaKUMkPZzqeW84222+tcG57rFKIKcvRKwJkzMTOuSFOeQpKh4eK4KZXVKK7+Q3mFf5G8a+jjuKjYbjSCpR38AF3TJv3xrj8fhnqR+8QHRp8tsgA0jJZBS55N7KKDyOztLFo49sI7mNnlrUwhWTQUkbLgN1LxF5MA=,iv:P4W9G2mI/3VTkvSrDZvnkJ0Qxb1nFmXcF35CrSNhako=,tag:yVsg7xsUALUoCJH0cHfQjw==,type:str] + pgp: [] + encrypted_regex: ^(data|stringData)$ + version: 3.9.4 diff --git a/apps/wave-0/plex.yaml b/apps/wave-0/plex.yaml index d189d25..5c3b21f 100644 --- a/apps/wave-0/plex.yaml +++ b/apps/wave-0/plex.yaml @@ -132,6 +132,11 @@ spec: value: Etc/UTC - name: ALLOWED_NETWORKS value: 10.0.0.0/8,192.168.0.0/16 + - name: PLEX_CLAIM + valueFrom: + secretKeyRef: + name: plex-claim + key: PLEX_CLAIM volumeMounts: - name: config mountPath: /config diff --git a/catalog/plex.yaml b/catalog/plex.yaml index be48d97..2ae7b37 100644 --- a/catalog/plex.yaml +++ b/catalog/plex.yaml @@ -32,11 +32,40 @@ env: # (10.43.0.0/16); 192.168.0.0/16 covers the home LAN. - name: ALLOWED_NETWORKS value: 10.0.0.0/8,192.168.0.0/16 - # PLEX_CLAIM is deliberately NOT set here: a https://plex.tv/claim token has a ~4-min TTL, far - # shorter than the commit -> push -> Flux reconcile -> pod-roll cycle, so any token committed to git - # is already expired by the time the pod starts. Claiming is a one-time LIVE step (UI sign-in, or a - # single API POST) against the running server; the claim then persists on the /config PVC and - # survives pod rolls (including the roll this env change causes), so it never belongs in git. + # PLEX_CLAIM — the ONE-SHOT plex.tv server claim, sourced from the SOPS Secret `plex-claim` + # (apps/wave-0/plex-claim-secret.sops.yaml, same wave => applied in the same reconcile as this + # workload). Plex cannot create libraries over the mounted bulk share while the server is + # UNCLAIMED, so this is wired; it was previously left out on the reasoning below, and only one + # half of that reasoning survived contact with the upstream source. + # + # THE 4-MINUTE TTL IS STILL TRUE, and it is now the OPERATIONAL constraint rather than a reason to + # abstain. A https://plex.tv/claim token expires in ~4 minutes — far shorter than commit -> push -> + # Flux reconcile -> pod-roll if that cycle is left to its own timers. So the committed value is a + # documented PLACEHOLDER and the live token is injected by the operator inside that window, with an + # explicit `flux reconcile` to collapse the wait: runbooks/plex-claim.md is the timed procedure. + # + # ONE-SHOT, AND SAFE TO SIT HERE EXPIRED. Verified from plexinc/pms-docker's own first-run script + # (branch `master`, root/etc/cont-init.d/40-plex-first-run, confirmed byte-identical to the file in + # the running `:latest` image), NOT assumed: + # - the exchange (line 106) is guarded on TWO conditions: PLEX_CLAIM non-empty AND the + # PlexOnlineToken pref in /config empty. Once claimed, the block is SKIPPED — this env is a + # no-op on every subsequent start, whatever it holds. (Upstream README: "If server is already + # logged in, this parameter is ignored.") The claim itself persists on the /config PVC as before. + # - an invalid/expired token on a still-unclaimed /config gets a 403 with an empty body; curl's + # status is unchecked, the token `sed` yields "", the success branch at line 120 does not fire, + # there is no `else` and no `exit 1`, and the script (no `set -e`) exits 0. It comes up UNCLAIMED and + # SERVING — it does NOT crashloop. The image also leaves S6_BEHAVIOUR_IF_STAGE2_FAILS at its + # default 0, which continues startup even when a cont-init script fails. + # So the earlier "a placeholder claim Secret would crashloop the pod" note was WRONG about the + # placeholder; it was right only about an ABSENT Secret. secretKeyRef here has no `optional` field, + # so the Secret must always exist in the Flux graph — which is why it is committed, encrypted, and + # listed in apps/wave-0/kustomization.yaml rather than hand-created at claim time. No schema change + # is needed: `optional` would only matter if a missing Secret had to be tolerated, and it doesn't. + - name: PLEX_CLAIM + valueFrom: + secretKeyRef: + name: plex-claim + key: PLEX_CLAIM # ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex # which URL to advertise to plex.tv for REMOTE clients. The only candidate is # https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL @@ -56,10 +85,12 @@ storage: # container's EPHEMERAL layer -> unclaimed + all state lost on restart. The image inits as # root (s6-svscan PID 1), chowns /config to the plex user, then drops -> no fsGroup needed # (verified in-cluster: PID1=root, "Plex Media Server"=plex). Persisting /config here also - # makes the claim survive restarts once claimed. To auto-claim on first boot instead of the - # UI sign-in, the operator adds a PLEX_CLAIM env from a fresh https://plex.tv/claim token - # (4-min TTL) — deliberately NOT wired here: the env secretKeyRef pattern has no `optional`, - # so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off. + # makes the claim survive restarts once claimed — which is what makes the PLEX_CLAIM env + # above a ONE-SHOT: the token is exchanged for a PlexOnlineToken written here, and the + # exchange is then skipped on every later start. (The older note here claimed a + # placeholder claim Secret "would crashloop the pod"; upstream's first-run script says + # otherwise — see the PLEX_CLAIM comment in `env` above. An ABSENT Secret is the hard + # failure, and that is why one is committed.) # THE LIBRARY (docs/04 bulk tier). One READ-ONLY mount of the export ROOT, giving /media/movies, # /media/tv and /media/photos — the whole tree the layout actually has (`ls /mnt/Storage/plex`: # downloads/ movies/ photos/ tv/). diff --git a/catalog/render.py b/catalog/render.py index 1e8ceeb..cc6c9a5 100644 --- a/catalog/render.py +++ b/catalog/render.py @@ -1844,6 +1844,10 @@ def plan_apps_waves(sds: list, present_files: set) -> dict: convention: - `.yaml` -> wave(svc) - `-bootstrap-secret.sops.yaml` -> wave(svc) (consumed by svc's env, same apply) + - `-claim-secret.sops.yaml` -> wave(svc) (a one-shot upstream-service claim token + consumed by svc's env — plex's plex.tv claim. Same wave as the app for the same reason the + bootstrap secret is: an env `secretKeyRef` has no `optional`, so the Secret must be applied + in the SAME reconcile as the workload or the pod is CreateContainerConfigError.) - `-oauth2-proxy-credentials.sops.yaml` -> wave(svc) (forward-auth sidecar OIDC creds) - `-db.yaml` -> wave(svc-db) (a strictly-lower wave; CNPG DB) - `-db-app.sops.yaml` -> wave(svc-db) (CNPG initdb owner secret) @@ -1870,6 +1874,11 @@ def plan_apps_waves(sds: list, present_files: set) -> dict: w = waves[name] add(w, f"{name}.yaml") add(w, f"{name}-bootstrap-secret.sops.yaml") + # A one-shot claim token for an UPSTREAM service (plex's plex.tv claim). Same wave as the + # app: it is read by the app's env at pod start, and an env secretKeyRef has no `optional`, + # so a Secret applied in a later wave would leave the pod in CreateContainerConfigError. + # Placed only when the file exists, so services without one are unaffected. + add(w, f"{name}-claim-secret.sops.yaml") # forward-auth (Option B): the per-app oauth2-proxy sidecar's OIDC client + cookie secret. # Same wave as the app (the sidecar reads it at pod start); placed only when the file exists, # so non-forward-auth apps are unaffected. Mirrors the bootstrap-secret naming convention. @@ -2548,7 +2557,8 @@ def _selftest() -> int: # 7a) plan_apps_waves: sidecars inherit their unit's wave; not-onboarded SDs are skipped; # EXTRA_UNITS anchor strictly above; a file absent from the tree is never placed. present = { - "a.yaml", "svc.yaml", "svc-bootstrap-secret.sops.yaml", + "a.yaml", "a-claim-secret.sops.yaml", + "svc.yaml", "svc-bootstrap-secret.sops.yaml", "svc-db.yaml", "svc-db-app.sops.yaml", "svc-db-backup-credentials.sops.yaml", "svc-db-backup-credentials-prod.sops.yaml", "gitea.yaml", @@ -2559,6 +2569,10 @@ def _selftest() -> int: plan = plan_apps_waves(plan_sds, present) flat = {f: wv for wv, fs in plan.items() for f in fs} check(flat.get("a.yaml") == 0, "leaf app must land in wave 0") + check(flat.get("a-claim-secret.sops.yaml") == 0, + "a `-claim-secret.sops.yaml` (plex's plex.tv claim) must be placed in the SAME wave " + "as its app — an env secretKeyRef has no `optional`, so a later wave means " + "CreateContainerConfigError") check(flat.get("svc-db.yaml") == 0 and flat.get("svc-db-app.sops.yaml") == 0 and flat.get("svc-db-backup-credentials.sops.yaml") == 0 and flat.get("svc-db-backup-credentials-prod.sops.yaml") == 0, diff --git a/runbooks/README.md b/runbooks/README.md index 0874231..5dbe033 100644 --- a/runbooks/README.md +++ b/runbooks/README.md @@ -6,7 +6,7 @@ Every alert rule in this platform carries an annotation of the form runbook_url: "https://${BASE_DOMAIN}/runbooks/" -46 of them, over the 35 names below. Until this directory existed, that path returned **HTTP 404** on the +46 of them, over 35 names. Until this directory existed, that path returned **HTTP 404** on the live cluster: every alert that reached a human handed them a dead link at the exact moment they needed help. These pages are what makes the link resolve. @@ -108,6 +108,11 @@ Each affected runbook says so on its own page and gives the by-hand equivalent. - **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that every green signal missed. +### Operator procedures (no alert links here) +- **`plex-claim`** — claiming the Plex Media Server. The plex.tv claim token lives ~4 minutes, so this + is a timed sequence: token → `sops` → push → **forced** `flux reconcile` → pod roll. Includes why a + changed Secret does not restart anything, and why an expired token is harmless. + ### Virtualisation - **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running. - **`kubevirt-launch-failed`** — a VMI reached the Failed phase. The three k3s-specific launch traps. diff --git a/runbooks/kustomization.yaml b/runbooks/kustomization.yaml index 3259aa5..a0bd184 100644 --- a/runbooks/kustomization.yaml +++ b/runbooks/kustomization.yaml @@ -81,6 +81,11 @@ configMapGenerator: - node-io=node-io.md - node-memory=node-memory.md - node-memory-swap=node-memory-swap.md + # Operator procedure, not an alert page (no alert links to it — the + # alert-runbooks-resolvable check treats that as advisory). It is served + # because the 4-minute plex.tv claim window is not a thing to go looking + # for in git: /runbooks/plex-claim has to resolve the moment it is needed. + - plex-claim=plex-claim.md - pvc-capacity=pvc-capacity.md - targets-down=targets-down.md - update-rollback=update-rollback.md diff --git a/runbooks/plex-claim.md b/runbooks/plex-claim.md new file mode 100644 index 0000000..791745f --- /dev/null +++ b/runbooks/plex-claim.md @@ -0,0 +1,185 @@ +# plex-claim — claiming the Plex Media Server inside the 4-minute token window + +**Alert:** none. This is an **operator procedure**, not an incident page — it is indexed here because +it is the page you need at the moment you need it, with a stopwatch running. +**Files:** `catalog/plex.yaml` (env `PLEX_CLAIM`), `apps/wave-0/plex-claim-secret.sops.yaml` +(the SOPS Secret `plex-claim`, key `PLEX_CLAIM`), `apps/wave-0/plex.yaml` (the rendered golden). + +## What this means here + +Plex will not let you **create libraries** over the mounted bulk media share while the server is +**unclaimed** — the 8.33 T NFS tree at `/media` is there, readable, and unusable. Claiming binds the +server to a plex.tv account. It is a **one-shot** operation: the token is exchanged once for a durable +`PlexOnlineToken` written into `Preferences.xml` on the `plex-config` Longhorn PVC, and from then on +the `PLEX_CLAIM` env is **ignored on every start** (upstream gate: +`[ ! -z "${PLEX_CLAIM}" ] && [ -z "${token}" ]` in `root/etc/cont-init.d/40-plex-first-run`). + +**A `https://plex.tv/claim` token lives ~4 minutes.** That is shorter than commit → push → Flux's +10-minute reconcile interval → pod roll if you let those run on their own timers, which is why this +page exists: every step below is written to be run back-to-back, with the reconcile **forced** rather +than waited for. Budget: token in hand at T+0, pod restarted by roughly T+2:00. + +**An expired or wrong token does not break anything.** Verified against the image's own first-run +script (and the copy inside `plexinc/pms-docker:latest`): a failed exchange gets a 403 with an empty +body, `curl`'s status is never checked, the token `sed` yields empty, the `if` does not fire, there is +no `else` and no `exit 1`, and the script (no `set -e`) exits 0. The pod comes up **unclaimed and +serving**. So a burnt token costs you one more attempt, not an outage — and the repo deliberately +ships a **placeholder** in that Secret for exactly this reason (the env `secretKeyRef` has no +`optional`, so the Secret must always exist even when its value is useless). + +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that +hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and that default has already changed once (k3s-dev until dev +was decommissioned on 2026-09-13, k3s-prod now). Do not run two `dev.sh` invocations at once. + +Mutating `kubectl` and `flux reconcile` are blocked for agents here. **This whole page is operator +work** — the reconcile and the pod roll below are deliberate operator actions, and the durable change +is still a git change. + +## First commands + +**Before spending a token, find out whether the server is already claimed.** This prints only the +attribute NAME, never its value — do not widen it to print the token: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c \ + "grep -o PlexOnlineToken= \"/config/Library/Application Support/Plex Media Server/Preferences.xml\" | head -1"' +``` + +`PlexOnlineToken=` printed ⇒ **already claimed**; stop, there is nothing to do and no token to burn. +Empty output ⇒ unclaimed, continue. + +Is the workload healthy enough to claim at all, and is the wave reconciling? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n plex get deploy,pod,pvc' +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0' +``` + +## The 4-minute procedure + +Have this page open and do not stop between steps. + +1. **T+0 — get the token.** Sign in to plex.tv as the account that should OWN this server, open + , copy the `claim-…` string. The clock starts now. + +2. **T+0:15 — put it in the Secret**, in place, encrypted, without it ever touching a plaintext file: + + ``` + infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml + ``` + + Replace the `PLEX_CLAIM:` value (the placeholder `replace-me-with-a-fresh-plex-tv-claim-token`, or + a previous expired token) and save. `sops` re-encrypts on write to the committed age recipient. + +3. **T+0:45 — commit and push.** `git push origin main` **is** the deploy; there is no other apply + step. + + ``` + git add apps/wave-0/plex-claim-secret.sops.yaml + git commit -m 'chore(plex): fresh plex.tv claim token' + git push origin main + ``` + +4. **T+1:00 — force the reconcile immediately. This is the one command that saves the window** — it + pulls the new commit and re-applies wave 0 instead of waiting up to 10 minutes for the interval: + + ``` + infra/devcontainer/dev.sh run bash -c \ + 'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-0 --with-source' + ``` + + `--with-source` is the load-bearing flag: it reconciles the `flux-system` GitRepository first, so + the Kustomization applies **your commit** and not the previously-fetched revision. + +5. **T+1:30 — roll the pod.** This step is **mandatory and easy to forget**: a changed Secret does + **not** restart anything. `PLEX_CLAIM` is injected by `secretKeyRef` and read only at container + start, and the Deployment's pod template is byte-identical across this change, so Flux applies the + new Secret and the running pod keeps the old value forever. + + ``` + infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n plex rollout restart deployment/plex' + ``` + +6. **T+2:00 — confirm the exchange happened**, in the new pod's init log: + + ``` + infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n plex logs deploy/plex --since=5m | grep -i -e claim -e token' + ``` + + * `Attempting to obtain server token from claim token` **and** `Token obtained successfully` + ⇒ **claimed.** Re-run the First-commands `PlexOnlineToken=` check to confirm it persisted, then + create the libraries over `/media/movies`, `/media/tv`, `/media/photos`. + * `Attempting to obtain server token from claim token` with **no** success line ⇒ the token was + expired or invalid. Nothing is broken; go back to step 1 with a new token. + * **Neither line** ⇒ the exchange was skipped, which means `/config` already had a + `PlexOnlineToken` (already claimed) or the env did not reach the container. Check the pod actually + has the variable, by NAME only: + + ``` + infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \ + -o jsonpath="{.items[0].spec.containers[0].env[*].name}"' + ``` + +## Likely causes, most likely first + +When the claim does not take, in order of how often it is the answer here: + +1. **The token expired** — you spent more than ~4 minutes, usually in step 3 or waiting on step 4. + Symptom: the "Attempting…" line with no success line. Remedy: a new token, and force the reconcile + rather than waiting for it. +2. **The pod was never restarted** (step 5 skipped). Symptom: no claim log lines at all from a pod + whose `AGE` predates your push. The Secret is correct in git and in the cluster and the running + container still holds the placeholder. +3. **Flux had not applied the commit yet** when you rolled the pod — step 5 run before step 4 + finished, so the new pod read the OLD Secret. Check + `kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0 -o jsonpath='{.status.lastAppliedRevision}'` + against your commit, then roll again. +4. **Already claimed** — the exchange is gated on `PlexOnlineToken` being absent from `/config`, so a + server claimed earlier ignores every later token silently. This is the benign case. +5. **`plex.tv` unreachable.** The upstream `curl` has no `--max-time`/`--connect-timeout`, so a stalled + connection can leave the container sitting in cont-init with **nothing listening on 32400** — the + one genuine hang risk in this path. Symptom: pod Running but never Ready, log stops after + "Attempting…". Check egress (`plex` is `egress: internet`) rather than re-rolling repeatedly. +6. **Wrong plex.tv account.** The claim succeeds and binds the server to whoever was signed in at + step 1. Unclaiming is a plex.tv-side operation on that account — see "escalate or stop". + +## When to escalate or stop + +- **Stop after two failed token attempts** and fix the timing, not the token. The third rushed attempt + is how people start editing live Secrets by hand, which leaves the cluster holding a value that is + in no git commit — precisely the drift `qbittorrent-bootstrap` had to be recovered from. +- **Never `kubectl edit`/`patch` the `plex-claim` Secret in the cluster, and never + `kubectl create secret` it.** Everything durable goes through the SOPS file; a live-only Secret is + reverted (or adopted with the wrong value) on the next reconcile. +- **Never `sops -d` this file into a terminal, a log, or a shared screen**, and never widen the + `PlexOnlineToken=` check into one that prints the value. The token is a plex.tv account credential. +- **If the server got claimed to the wrong account, stop and get the operator.** Reversing it means + removing the server from that account on plex.tv and clearing `PlexOnlineToken` from + `Preferences.xml` on the PVC — a destructive edit of app state on the replicated tier, not a git + change. +- **Do not delete the `plex-config` PVC to "start clean".** It holds the library database, metadata + and watch state (`policy: daily-offcluster`), and it is expensive to rebuild. The claim is a + two-line pref, not a reason to lose the DB. +- After a successful claim, **leave the expired token in the SOPS file**. It is inert (the exchange is + skipped once claimed) and the Secret must keep existing, because the env `secretKeyRef` has no + `optional` — deleting it would put the pod into `CreateContainerConfigError`. + +--- +Source of truth: `runbooks/plex-claim.md`. Index: `/runbooks/` +Upstream behaviour cited here is from `plexinc/pms-docker` branch `master`, +`root/etc/cont-init.d/40-plex-first-run` (s6-overlay v2), cross-checked against the same file inside +the running `plexinc/pms-docker:latest` image. -- 2.54.0