feat(plex): wire PLEX_CLAIM from a SOPS Secret so the server can be claimed

Plex cannot create libraries over the already-mounted 8.33T NFS bulk share while
the server is UNCLAIMED. catalog/plex.yaml carried a long argument for leaving
PLEX_CLAIM unwired; the operator has decided to override it. This implements the
wiring in the way that respects the half of that argument which is still true.

WHAT WAS TRUE AND STILL IS: a https://plex.tv/claim token has a ~4-minute TTL,
far shorter than commit -> push -> Flux reconcile -> pod-roll on its own timers.
So the committed value is a documented PLACEHOLDER and the live token is injected
by the operator inside that window, with the reconcile FORCED rather than waited
for. runbooks/plex-claim.md is that timed procedure.

WHAT WAS NOT TRUE: the old comment claimed "a placeholder/absent claim Secret
would crashloop the pod". Half of that is wrong, and the difference is the whole
design. Established from plexinc/pms-docker's own first-run script (branch
`master`, root/etc/cont-init.d/40-plex-first-run, s6-overlay v2), cross-checked
byte-identical against the same file inside the running :latest image:

  * the claim exchange (line 106) is gated on PLEX_CLAIM being non-empty AND the
    PlexOnlineToken pref in /config being empty. Once the server is claimed the
    block is skipped entirely, so the env is a NO-OP on every later start
    whatever it holds ("If server is already logged in, this parameter is
    ignored" -- upstream README).
  * with an invalid/expired token on a still-unclaimed /config, plex.tv answers
    403 with an empty body; curl's status is never checked, the token sed yields
    empty, the success branch at line 120 does not fire, there is no else branch
    and no non-zero exit, and the script (no `set -e`) exits 0. The pod comes up
    UNCLAIMED AND SERVING. It does not crashloop.
  * the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS (default 0 = continue even
    if a cont-init script failed), so even a non-zero exit would not halt
    startup.

An ABSENT Secret is the real hard failure (secretKeyRef has no `optional`, so it
is CreateContainerConfigError), which is why the Secret is committed, encrypted
and listed in the wave kustomization rather than hand-created at claim time.

NO SCHEMA CHANGE. `optional` on secretKeyRef would only be needed if the pod had
to tolerate a missing Secret or could not start without a VALID token. Neither
holds: the Secret always exists in the Flux graph, and an invalid token is
non-fatal. schemas/servicedefinition.schema.json is untouched, and so are
catalog/render.py's env emission and the renderer contract.

Changes:
  * apps/wave-0/plex-claim-secret.sops.yaml -- new Opaque Secret plex-claim/plex,
    key PLEX_CLAIM, SOPS-encrypted to the committed age recipient. Its header
    carries NO dollar-brace expressions on purpose: kustomize preserves body
    comments into the stream envsubst reads (per
    conformance/tests/l0-static/flux-postbuild-substitution), so quoting the
    upstream shell conditions verbatim would have been a live substitution
    hazard on apps-wave-0, not prose.
  * catalog/plex.yaml -- the PLEX_CLAIM env via valueFrom.secretKeyRef, the
    "deliberately NOT set" block rewritten around the TTL + one-shot design, and
    the stale crashloop claim in the /config storage comment corrected.
  * apps/wave-0/plex.yaml -- re-rendered with the canonical command
    (`--all catalog -o apps --wave-layout --no-image-automation`). No L1 golden
    under conformance/tests/l1-rendering covers plex, so this is the only
    rendered surface that moves.
  * catalog/render.py -- plan_apps_waves places `<svc>-claim-secret.sops.yaml`
    in the app's OWN wave, like the bootstrap secret, plus a selftest assertion.
    Without this, the next `--emit-waves` would have silently dropped the Secret
    from apps/wave-0/kustomization.yaml and put the pod into
    CreateContainerConfigError. apps/wave-0/kustomization.yaml is the hand-pruned
    subset (vm-demo/keycloak are rendered but deliberately unlisted), so its one
    new line was applied without re-emitting the rest.
  * runbooks/plex-claim.md + its kustomization.yaml generator entry + README
    index -- the operator procedure that fits in 4 minutes, including the single
    forced-reconcile command and the mandatory pod roll (a changed Secret does
    NOT restart anything: PLEX_CLAIM is read only at container start and the pod
    template is byte-identical across this change).

Touched requirements: R-SEC-01 (secrets encrypted at rest), R-GITOPS-02
(ServiceDefinition onboarding contract).

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

catalog/plex.yaml

@@ -32,11 +32,40 @@ env:
   # (10.43.0.0/16); 192.168.0.0/16 covers the home LAN.
   - name: ALLOWED_NETWORKS
     value: 10.0.0.0/8,192.168.0.0/16
-  # PLEX_CLAIM is deliberately NOT set here: a https://plex.tv/claim token has a ~4-min TTL, far
-  # shorter than the commit -> push -> Flux reconcile -> pod-roll cycle, so any token committed to git
-  # is already expired by the time the pod starts. Claiming is a one-time LIVE step (UI sign-in, or a
-  # single API POST) against the running server; the claim then persists on the /config PVC and
-  # survives pod rolls (including the roll this env change causes), so it never belongs in git.
+  # PLEX_CLAIM — the ONE-SHOT plex.tv server claim, sourced from the SOPS Secret `plex-claim`
+  # (apps/wave-0/plex-claim-secret.sops.yaml, same wave => applied in the same reconcile as this
+  # workload). Plex cannot create libraries over the mounted bulk share while the server is
+  # UNCLAIMED, so this is wired; it was previously left out on the reasoning below, and only one
+  # half of that reasoning survived contact with the upstream source.
+  #
+  # THE 4-MINUTE TTL IS STILL TRUE, and it is now the OPERATIONAL constraint rather than a reason to
+  # abstain. A https://plex.tv/claim token expires in ~4 minutes — far shorter than commit -> push ->
+  # Flux reconcile -> pod-roll if that cycle is left to its own timers. So the committed value is a
+  # documented PLACEHOLDER and the live token is injected by the operator inside that window, with an
+  # explicit `flux reconcile` to collapse the wait: runbooks/plex-claim.md is the timed procedure.
+  #
+  # ONE-SHOT, AND SAFE TO SIT HERE EXPIRED. Verified from plexinc/pms-docker's own first-run script
+  # (branch `master`, root/etc/cont-init.d/40-plex-first-run, confirmed byte-identical to the file in
+  # the running `:latest` image), NOT assumed:
+  #   - the exchange (line 106) is guarded on TWO conditions: PLEX_CLAIM non-empty AND the
+  #     PlexOnlineToken pref in /config empty. Once claimed, the block is SKIPPED — this env is a
+  #     no-op on every subsequent start, whatever it holds. (Upstream README: "If server is already
+  #     logged in, this parameter is ignored.") The claim itself persists on the /config PVC as before.
+  #   - an invalid/expired token on a still-unclaimed /config gets a 403 with an empty body; curl's
+  #     status is unchecked, the token `sed` yields "", the success branch at line 120 does not fire,
+  #     there is no `else` and no `exit 1`, and the script (no `set -e`) exits 0. It comes up UNCLAIMED and
+  #     SERVING — it does NOT crashloop. The image also leaves S6_BEHAVIOUR_IF_STAGE2_FAILS at its
+  #     default 0, which continues startup even when a cont-init script fails.
+  # So the earlier "a placeholder claim Secret would crashloop the pod" note was WRONG about the
+  # placeholder; it was right only about an ABSENT Secret. secretKeyRef here has no `optional` field,
+  # so the Secret must always exist in the Flux graph — which is why it is committed, encrypted, and
+  # listed in apps/wave-0/kustomization.yaml rather than hand-created at claim time. No schema change
+  # is needed: `optional` would only matter if a missing Secret had to be tolerated, and it doesn't.
+  - name: PLEX_CLAIM
+    valueFrom:
+      secretKeyRef:
+        name: plex-claim
+        key: PLEX_CLAIM
   # ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex
   # which URL to advertise to plex.tv for REMOTE clients. The only candidate is
   # https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL
@@ -56,10 +85,12 @@ storage:
                               # container's EPHEMERAL layer -> unclaimed + all state lost on restart. The image inits as
                               # root (s6-svscan PID 1), chowns /config to the plex user, then drops -> no fsGroup needed
                               # (verified in-cluster: PID1=root, "Plex Media Server"=plex). Persisting /config here also
-                              # makes the claim survive restarts once claimed. To auto-claim on first boot instead of the
-                              # UI sign-in, the operator adds a PLEX_CLAIM env from a fresh https://plex.tv/claim token
-                              # (4-min TTL) — deliberately NOT wired here: the env secretKeyRef pattern has no `optional`,
-                              # so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
+                              # makes the claim survive restarts once claimed — which is what makes the PLEX_CLAIM env
+                              # above a ONE-SHOT: the token is exchanged for a PlexOnlineToken written here, and the
+                              # exchange is then skipped on every later start. (The older note here claimed a
+                              # placeholder claim Secret "would crashloop the pod"; upstream's first-run script says
+                              # otherwise — see the PLEX_CLAIM comment in `env` above. An ABSENT Secret is the hard
+                              # failure, and that is why one is committed.)
   # THE LIBRARY (docs/04 bulk tier). One READ-ONLY mount of the export ROOT, giving /media/movies,
   # /media/tv and /media/photos — the whole tree the layout actually has (`ls /mnt/Storage/plex`:
   # downloads/ movies/ photos/ tv/).

catalog/render.py

@@ -1844,6 +1844,10 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
     convention:
       - `<svc>.yaml`                        -> wave(svc)
       - `<svc>-bootstrap-secret.sops.yaml`  -> wave(svc)   (consumed by svc's env, same apply)
+      - `<svc>-claim-secret.sops.yaml`      -> wave(svc)   (a one-shot upstream-service claim token
+        consumed by svc's env — plex's plex.tv claim. Same wave as the app for the same reason the
+        bootstrap secret is: an env `secretKeyRef` has no `optional`, so the Secret must be applied
+        in the SAME reconcile as the workload or the pod is CreateContainerConfigError.)
       - `<svc>-oauth2-proxy-credentials.sops.yaml` -> wave(svc)  (forward-auth sidecar OIDC creds)
       - `<svc>-db.yaml`                     -> wave(svc-db) (a strictly-lower wave; CNPG DB)
       - `<svc>-db-app.sops.yaml`            -> wave(svc-db) (CNPG initdb owner secret)
@@ -1870,6 +1874,11 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
         w = waves[name]
         add(w, f"{name}.yaml")
         add(w, f"{name}-bootstrap-secret.sops.yaml")
+        # A one-shot claim token for an UPSTREAM service (plex's plex.tv claim). Same wave as the
+        # app: it is read by the app's env at pod start, and an env secretKeyRef has no `optional`,
+        # so a Secret applied in a later wave would leave the pod in CreateContainerConfigError.
+        # Placed only when the file exists, so services without one are unaffected.
+        add(w, f"{name}-claim-secret.sops.yaml")
         # forward-auth (Option B): the per-app oauth2-proxy sidecar's OIDC client + cookie secret.
         # Same wave as the app (the sidecar reads it at pod start); placed only when the file exists,
         # so non-forward-auth apps are unaffected. Mirrors the bootstrap-secret naming convention.
@@ -2548,7 +2557,8 @@ def _selftest() -> int:
     # 7a) plan_apps_waves: sidecars inherit their unit's wave; not-onboarded SDs are skipped;
     #     EXTRA_UNITS anchor strictly above; a file absent from the tree is never placed.
     present = {
-        "a.yaml", "svc.yaml", "svc-bootstrap-secret.sops.yaml",
+        "a.yaml", "a-claim-secret.sops.yaml",
+        "svc.yaml", "svc-bootstrap-secret.sops.yaml",
         "svc-db.yaml", "svc-db-app.sops.yaml", "svc-db-backup-credentials.sops.yaml",
         "svc-db-backup-credentials-prod.sops.yaml",
         "gitea.yaml",
@@ -2559,6 +2569,10 @@ def _selftest() -> int:
     plan = plan_apps_waves(plan_sds, present)
     flat = {f: wv for wv, fs in plan.items() for f in fs}
     check(flat.get("a.yaml") == 0, "leaf app must land in wave 0")
+    check(flat.get("a-claim-secret.sops.yaml") == 0,
+          "a `<svc>-claim-secret.sops.yaml` (plex's plex.tv claim) must be placed in the SAME wave "
+          "as its app — an env secretKeyRef has no `optional`, so a later wave means "
+          "CreateContainerConfigError")
     check(flat.get("svc-db.yaml") == 0 and flat.get("svc-db-app.sops.yaml") == 0
           and flat.get("svc-db-backup-credentials.sops.yaml") == 0
           and flat.get("svc-db-backup-credentials-prod.sops.yaml") == 0,