Compare commits

..
Author SHA1 Message Date
jakman85 d3a0d587a4 Added gitea action runner secret
gate / gate (push) Successful in 1m32s
gate / gate (pull_request) Successful in 1m34s
gate / conformance (push) Has been skipped
gate / conformance (pull_request) Has been skipped
2026-09-29 00:50:02 -05:00
jakman85andClaude Opus 5 fe862ab527 infra(gitea): recover the Actions runner before anything requires the gate
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
gate / gate (pull_request) Successful in 1m32s
gate / conformance (pull_request) Has been skipped
No runner is registered at repo, org or site level, so gate.yml has never executed.
PR #1 sat pending for 30.6 minutes across 122 polls and never reported. Requiring the
gate as a status check in that state would lock the repo permanently.

Adds runner-token-rotate.sh: mints a fresh instance-level registration token from the
Gitea API and writes it into the SOPS secret via sops set, without ever printing the
value.

Records the likely root cause, which a new token alone does NOT fix. act_runner
registers once and persists its identity in /data/.runner on the act-runner-data PVC.
If Gitea no longer holds a matching runner row, the pod starts, sees .runner, SKIPS
registration entirely, and polls with an identity Gitea does not recognise. Symptom:
zero runners, pod running, nothing obviously broken. Clearing the PVC is what forces a
genuine re-registration.

The README now sequences runner-first, enforcement-second, and requires CONFIRMING the
runner reports online rather than assuming it. That assumption is the whole defect:
DECISIONS-LOG recorded the runner as online on 2026-08-31 and nobody checked since.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 00:43:47 -05:00
jakman85andClaude Opus 5 a2665c0511 infra(gitea): script the branch protection that was never applied
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
gate / gate (pull_request) Successful in 1m33s
gate / conformance (pull_request) Has been skipped
Gitea repo settings live in its own database, not in a manifest Flux reconciles, so
they are drift-prone and invisible to the conformance gate -- which is how main came
to accept direct pushes while the log recorded it as protected.

Adds infra/gitea/branch-protection.sh (blocks direct pushes to main) and a README
carrying the measured before-state and the ORDER the remaining steps must happen in.

The script deliberately leaves enable_status_check:false and required_approvals:0.
Both would deadlock today: there is no runner to satisfy a required context, and the
only human account is not yet a repo collaborator, so no one could supply an approval.
The README sequences both.

It prints a push probe to run afterwards. The API response is not proof -- Gitea may
let site admins bypass protection, and that has to be observed, not assumed.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 00:20:12 -05:00
jakman85andClaude Opus 5 5a9e455149 docs: delivery contract for agents + record that the merge gate does not exist
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
gate / gate (pull_request) Successful in 1m33s
gate / conformance (pull_request) Has been skipped
Going to land the plex claim wiring through the documented PR flow surfaced that
the flow has never run. PR #1 is the first pull request this repo has ever had.

Verified against the live Gitea, not inferred from the docs:
  * branch_protections is [] -- main is protected:false, enable_status_check:false,
    status_check_contexts:[], required_approvals:0, user_can_push:true
  * ZERO Actions runners registered at repo, org and site level

DECISIONS-LOG 2026-08-31 asserts a "required status check" and a runner
"incluster-dind online". Both are false of the live system. The gate is neither
enforced nor able to run, and that went unnoticed because the log claimed
otherwise -- the same fail-open class as the two holes found on 2026-09-18.

Adds CLAUDE.md, the first agent-facing delivery doc here: the
branch -> PR -> gate -> approval -> merge rule, stated alongside the honest fact
that enforcement is currently weaker than the rule, so agents follow the rule
rather than what they can get away with. Carries the toolbox gate invocation, the
two git-worktree workarounds, the rc 0/1/3 contract, and secret handling.

Records the ordering trap: enabling a required status check before a runner exists
deadlocks every PR permanently, including the one that would fix it.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 00:11:58 -05:00
14 changed files with 371 additions and 329 deletions

No files matched your search

+115
View File
@@ -0,0 +1,115 @@
# CLAUDE.md — how work lands in this repo
Read this before doing anything durable. It is the delivery contract for AI coding
sessions and agents. `DECISIONS-LOG.md`'s last entry is the current PM handoff and
takes precedence on *what* to work on; this file governs *how* it lands.
## 1. Never push to `main`. Ever.
Every durable change reaches `main` exactly one way:
branch -> push the branch -> open a PR -> CI gate green -> human approval -> merge
There is no other path. Specifically:
- **Do not `git push origin main`.**
- **Do not `git push origin HEAD:main`.** Same thing wearing a hat.
- **Do not merge your own PR.** A human approves and clicks merge.
- **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
> **Many runbooks under `runbooks/` still say `git push origin main` IS the deploy.**
> That wording is **stale** — it predates the PR flow. It describes what Flux does
> (main is what the cluster follows), not what you are permitted to do. Honour this
> file, not that line. Fixing those runbooks is tracked work.
## 2. Enforcement is currently WEAKER than this rule. Follow the rule anyway.
Verified against the live Gitea on 2026-09-29:
| Expected | Actual |
|---|---|
| `main` protected, direct push rejected | **No branch protection.** `protected: false`, `user_can_push: true` |
| PR blocked until the gate passes | **`enable_status_check: false`**, `status_check_contexts: []` |
| Gate runs on every PR | **No Actions runner is registered** (repo, org and site level all report zero) |
| Approval required before merge | **`required_approvals: 0`** |
So nothing mechanically stops you from pushing to `main` right now. **That is a gap
being closed, not permission.** `DECISIONS-LOG.md` (2026-08-31) claims the gate is a
"required status check" with a runner online; that claim is not true of the live
system, which is how this went unnoticed.
**Consequence for you:** because CI cannot run, the **local toolbox gate is the only
real gate**. Run it and quote its numbers in your report. Never describe a PR as
"green" on the strength of CI that did not execute.
## 3. Gate before you push
./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
Exit codes are a contract — do not flatten them:
| rc | Meaning |
|---|---|
| 0 | PASS |
| 1 | BLOCKED — something failed |
| 3 | UNDETERMINED — a stage did not run. **Never report this as green.** |
`spec-immutability` reaches rc 3 whenever the sibling spec repo is not checked out
beside this one; that single skip is expected and tolerated. Any *other* skip is a
real gap in your run.
**Never gate on the host.** `promtool` and `kustomize` are not on the host `$PATH`,
so checks SKIP and you get a false rc 3. Only the pinned toolbox result counts.
### Running the gate from a git worktree
Two unrelated failures, both with known workarounds — put both in any agent brief:
1. **`dev.sh` refuses to start:** `infra/ansible/.cache/prod.kubeconfig` is untracked
and exists only in the main checkout. Do **not** copy it (it is a credential).
Use `DEV_KUBECONFIG=/dev/null` — the gate contacts no cluster, so L2/L3 scaffold-skip
normally.
2. **Every `git` call inside the container fails:** a worktree's `.git` is a pointer to a
gitdir outside the mounted root. `dev.sh` takes no extra-mount flag, so inject one via
the `ENGINE` hook — a shim that intercepts `run` and re-execs
`podman run -v /path/to/main/.git:/path/to/main/.git:ro "$@"`. A traceback in
`tracked_files()` is this bug, not a broken tree.
## 4. Branch naming
Use `feat/`, `fix/`, `docs/`, `chore/`. Note that `.gitea/workflows/gate.yml`'s **push**
trigger lists `feature/**` — not `feat/**` — so branch pushes get no fast-feedback run.
The `pull_request` trigger fires regardless of branch name, so the PR gate is unaffected.
## 5. Cluster access
`kubectl` does not work on the operator workstation; the toolchain lives only in the
pinned container. Always name the context explicitly:
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
`dev.sh` merges both clusters and picks a current-context **per invocation**; that
default has already changed once. Never rely on it. Do not run two `dev.sh`
invocations concurrently.
**Mutating `kubectl` and `flux reconcile` are operator actions, not agent actions.**
## 6. Secrets
- **Never `sops -d` a file into a terminal, a log, or a transcript.**
- **Never print a Secret's `.data`/`.stringData`** when checking whether it exists —
list key names or object names only. A password has been leaked into a transcript
this way before.
- Encrypted files are `*.sops.yaml` or live under `secrets/`. The root `.sops.yaml` is
the single creation-rules file; do not add a second one in a subtree.
- A short-lived credential (e.g. a plex.tv claim token, ~minutes) cannot survive the PR
flow. Do not design around committing one — read the expiry from the issuing page and
treat a burnt token as free to retry.
## 7. Before you finish
Append a handoff entry to `DECISIONS-LOG.md` describing what you changed, what you
verified, and what you did **not** verify. State gaps plainly — an honest
"UNVERIFIED" is worth more than an optimistic "green", and this repo has already been
burned by the difference.
+65
View File
@@ -4716,3 +4716,68 @@ asleep. `git worktree remove` preserves branches and their commits, so only that
6. **Five `public` services have no working public path** — waiting on WAN forwarding, deliberately.
7. Backlog cherry-picks: `feat/egress-gateway-staleness-alert`, two `rescued/*` fixes, two rendered
manifests nothing applies (the keycloak one would stand up a conflicting instance).
## 2026-09-29 — The CI/CD merge gate does not exist: no runner, no branch protection. **PM HANDOFF.**
Went to land the plex claim wiring through the documented PR flow and discovered the flow has
never run. **PR #1 is the first pull request this repository has ever had.** It is still queued.
### What is actually true of the live Gitea (verified via API, 2026-09-29)
| Claimed (DECISIONS-LOG 2026-08-31) | Measured |
|---|---|
| "required status check" on `main` | `branch_protections` returns `[]`. `protected: false`, `enable_status_check: false`, `status_check_contexts: []`, `required_approvals: 0`, `user_can_push: true` |
| "runner `incluster-dind` online (ns gitea-actions)" | **Zero runners registered** at repo, org and site level |
Both halves of that entry are false. Nothing enforces the gate, and the gate cannot run.
PR #1's two contexts (`gate / gate`, `gate / conformance`) sat `pending` for 14+ minutes with
nothing to execute them, and will sit there indefinitely.
This is a **fail-open governance hole**, the same class as the two found on 2026-09-18 — and it
was invisible precisely because the log asserted the opposite. Every "the gate is enforcing"
statement since 2026-08-31 rested on an unverified claim.
### Ordering trap — read before fixing
**Do not enable required status checks until a runner is live.** Requiring
`gate / gate (pull_request)` with no runner deadlocks every PR permanently, including the PR
that would fix it. Sequence: runner first, verify the gate actually runs and passes on a real
PR, *then* turn on enforcement with the context name CI actually reported.
Related: `gate.yml`'s `conformance` job is `if: false` by design (honest skip, never fake green),
but it still publishes a `gate / conformance (pull_request)` context. If that context is ever
listed as required, it deadlocks every PR for the same reason. Wire it or exclude it.
### Delivery contract written down
Added `CLAUDE.md` at the repo root — the first agent-facing delivery doc here. It states the
branch -> PR -> gate -> approval -> merge rule, records that enforcement is currently weaker than
the rule (so agents follow the rule rather than what they can get away with), and carries the
toolbox-gate invocation, the two git-worktree workarounds, the rc 0/1/3 contract, and the secret
handling rules.
It also flags that **37 of 39 runbooks still instruct `git push origin main`**. Those were
accurate before the PR flow and are stale now. Correcting them is tracked, not done.
### Also found, not fixed
- `gate.yml`'s push trigger lists `feature/**`; this repo's branches use `feat/**`, so branch
pushes get no fast-feedback run. PR trigger is unaffected.
- The gitea-actions runner is GitOps-managed (`apps/wave-1/gitea-actions.yaml` +
`gitea-actions-runner-secret.sops.yaml`), so it is deployed but not registering — most likely a
stale registration token in that SOPS secret. Needs in-cluster diagnosis (operator).
- **Plex is still unclaimed and it was never checked** whether it is already claimed. Do that
before spending a token — `runbooks/plex-claim.md` §First commands.
- `runbooks/plex-claim.md` asserts a "~4-minute TTL" for a plex.tv claim token in four places,
**uncited**, and the whole stopwatch procedure is built on it. Upstream `pms-docker`'s README
says nothing about expiry. The number is plausible community folklore, not a measured fact.
The authoritative reading is the countdown rendered on plex.tv/claim at issue time.
### Next steps
1. **Operator:** diagnose the gitea-actions runner; get one registered.
2. **Operator:** enable branch protection on `main` — push blocked, approvals required — and only
then add the gate as a required status check.
3. Land `feat/plex-claim-sops-secret` (PR #1) once the gate can actually run.
4. Claim plex, after checking it is not already claimed.
5. Correct the 37 runbooks' `git push origin main` wording.
-1
View File
@@ -17,7 +17,6 @@ resources:
- pihole-dns.yaml
- pihole-oauth2-proxy-credentials.sops.yaml
- pihole.yaml
- plex-claim-secret.sops.yaml
- plex.yaml
- qbittorrent-bootstrap-secret.sops.yaml
- qbittorrent-oauth2-proxy-credentials.sops.yaml
-69
View File
@@ -1,69 +0,0 @@
# Plex plex.tv server-claim token (WS-APPS). SOPS-encrypted (age).
# Consumed by catalog/plex.yaml as env PLEX_CLAIM via valueFrom.secretKeyRef.
#
# NOTE ON THESE COMMENTS: they carry NO dollar-brace expressions, deliberately. This file is applied
# by the apps-wave-0 Flux Kustomization, which runs envsubst over the built output, and kustomize
# preserves body comments into that stream (see conformance/tests/l0-static/flux-postbuild-
# substitution). A braced token quoted from the upstream shell script would be a live hazard here,
# not prose, so the upstream conditions below are described in words.
#
# THE VALUE SHIPPED WITH THIS FILE IS A PLACEHOLDER, DELIBERATELY — the literal string
# `replace-me-with-a-fresh-plex-tv-claim-token`, not a credential. A https://plex.tv/claim token has
# a ~4-minute TTL, far shorter than commit -> push -> Flux reconcile -> pod-roll, so no token
# committed here is ever still valid when the pod starts. This file exists to make the Secret — and
# therefore the env reference — ALWAYS RESOLVABLE IN THE FLUX GRAPH, not to carry a working token.
# The real token is injected by the operator inside that 4-minute window: runbooks/plex-claim.md.
#
# WHY SHIPPING A PLACEHOLDER IS SAFE (verified against plexinc/pms-docker branch `master`, the
# s6-overlay v2 script root/etc/cont-init.d/40-plex-first-run, cross-checked byte-identical against
# the same file extracted from the running `plexinc/pms-docker:latest` image):
# - line 106 guards the claim exchange on TWO conditions: PLEX_CLAIM non-empty AND the
# PlexOnlineToken pref read out of /config empty. Once the server is claimed that block is
# skipped entirely, so whatever this Secret holds is a NO-OP on every later start. Upstream
# README: "If server is already logged in, this parameter is ignored."
# - on a still-UNCLAIMED /config, an invalid or expired token makes plex.tv answer 403 with an
# empty body. curl's exit status is never checked, the sed for the authentication-token element
# yields the empty string, and the success branch at line 120 simply does not fire. There is no
# else branch and no non-zero exit; the script has no `set -e` and ends by touching
# /.firstRunComplete and echoing, so it exits 0. The server comes up UNCLAIMED and serving.
# It does NOT crashloop.
# - the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS, whose default is 0 ("continue silently even
# if a cont-init script failed"), so even a hypothetical non-zero exit would not halt startup.
# A MISSING Secret would NOT be safe: the ServiceDefinition schema's valueFrom.secretKeyRef shape
# has no `optional` field, so an absent Secret is a hard CreateContainerConfigError. That is exactly
# why this file is committed and listed in apps/wave-0/kustomization.yaml instead of being created
# by hand at claim time.
#
# INJECTION / ROTATION (operator, not agents) — runbooks/plex-claim.md has the timed procedure:
# infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
# Never decrypt this file into a shell, a log, or a shared terminal.
apiVersion: v1
kind: Secret
metadata:
name: plex-claim
namespace: plex
labels:
app.kubernetes.io/name: plex
type: Opaque
stringData:
PLEX_CLAIM: ENC[AES256_GCM,data:8O9nvFw+6OR+53mXvHfe321aSmYUQcZuAqLKZpcSlUwNyeU+Nbp/tRPBlg==,iv:EbvvOsXlXjAFOewnWbK28ErmUdWcJJN0Q5Gm53El3ZM=,tag:wXaInGFQ+wqUt765vvjCDA==,type:str]
sops:
kms: []
gcp_kms: []
azure_kv: []
hc_vault: []
age:
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2dWtqOVNQalZ2QlRxOXJa
cDc1MTdtWENkdWIxS1JId3dIU1JDSmk0ODNrCk1YUzhFWXR3bUxyV2NNcmFOakdz
THl1Q284dFJxTy9Rd1h0eXJJaVZtdmcKLS0tIERuN0ZFeW9zdi9aNFRYNWxmUS9s
QUtYdHEvN042ajNoNktOLzlYQmQrc1kK4NHozMbXVzNGYbUBRbK6a9Uw/sVPAfe6
RW5KWzZ4I/hloqjN3zzVISoSMieMkBGDnF5qZS5N6FqIc6/HrsDD8A==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-09-29T03:25:20Z"
mac: ENC[AES256_GCM,data:u5uVmOdXN78UaKUMkPZzqeW84222+tcG57rFKIKcvRKwJkzMTOuSFOeQpKh4eK4KZXVKK7+Q3mFf5G8a+jjuKjYbjSCpR38AF3TJv3xrj8fhnqR+8QHRp8tsgA0jJZBS55N7KKDyOztLFo49sI7mNnlrUwhWTQUkbLgN1LxF5MA=,iv:P4W9G2mI/3VTkvSrDZvnkJ0Qxb1nFmXcF35CrSNhako=,tag:yVsg7xsUALUoCJH0cHfQjw==,type:str]
pgp: []
encrypted_regex: ^(data|stringData)$
version: 3.9.4
-5
View File
@@ -132,11 +132,6 @@ spec:
value: Etc/UTC
- name: ALLOWED_NETWORKS
value: 10.0.0.0/8,192.168.0.0/16
- name: PLEX_CLAIM
valueFrom:
secretKeyRef:
name: plex-claim
key: PLEX_CLAIM
volumeMounts:
- name: config
mountPath: /config
@@ -14,7 +14,7 @@ metadata:
app.kubernetes.io/name: act-runner
type: Opaque
stringData:
token: ENC[AES256_GCM,data:W48mEtBqGeukX13lMhrhrer+zB21cLVp0acy3m1Xy/oo89DqO8FX/Q==,iv:wfTUU7RiFLdEc1fmEFS9DfE2SmGCFLPK17m0IKspft0=,tag:UhoXA47p7386KHcYNbPzCg==,type:str]
token: ENC[AES256_GCM,data:f574mAUkLvMGq0TYycwCd4tduMuBgQLiTg/140kuRDgFUYYm1ZoSjQ==,iv:Tn7gFWBLm2hHk/BbfVX7qIZegwZp9ah1cMou5i2O91E=,tag:TkdSiMI4sdCNn2JuN++Pig==,type:str]
sops:
kms: []
gcp_kms: []
@@ -30,8 +30,8 @@ sops:
MDZLUE52QzJXbFlrNWxOcVhJbDNnY00KxCpFkbhhCuk4Fa8CC3C4+R9AcsUnaXQR
UjAfwKFoXAneXUih7TTXMbR/6+VgQBww5Bmwpn7jVLnzP0N0pIZVTA==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-08-26T04:48:14Z"
mac: ENC[AES256_GCM,data:b99/zkL332RIT3FYBa1sR3o9+pwlmrUgKvVHawbjWYSJZ7ERUfO+LXuhKJ0HKrp6i1JfkurZlRO27iBcfrBisAXS9KLaqmsCJf4kKfFc4skANrmKCzprrXUmZexr1uiE8eDUzjNCLNxSCZfnFR6d/AkMXJqi6Hsj9/O+wH2lWsc=,iv:3DIPszSdxK81ERO3o/nTs33qQYHaAKLNcGP4qlZViFc=,tag:/B6VKkzMUwh0byNUicS5wQ==,type:str]
lastmodified: "2026-09-29T05:48:35Z"
mac: ENC[AES256_GCM,data:nobqr794nrdRYFrd8Ct8bw9M8NtftJXawRGk9zrL+pTWahwlWrwXVtOAxMGEPVojz3WemsxTJESzmzy0gcT1QcrjpYTsF2MVHcr6+umk8q9gAh2enyxy/0QkYiSt6XjvRVqfqleLanEkeIq1FPf2zhSLSJaiw4/oZTfkCHD1/Zg=,iv:JxaP0htmGcvE4ZU1YAmGF4A6JkejV29NNygyMl1OKF8=,tag:G70NvcPyVuagXhugWoUfTw==,type:str]
pgp: []
encrypted_regex: ^(data|stringData)$
version: 3.9.4
+9 -40
View File
@@ -32,40 +32,11 @@ env:
# (10.43.0.0/16); 192.168.0.0/16 covers the home LAN.
- name: ALLOWED_NETWORKS
value: 10.0.0.0/8,192.168.0.0/16
# PLEX_CLAIM — the ONE-SHOT plex.tv server claim, sourced from the SOPS Secret `plex-claim`
# (apps/wave-0/plex-claim-secret.sops.yaml, same wave => applied in the same reconcile as this
# workload). Plex cannot create libraries over the mounted bulk share while the server is
# UNCLAIMED, so this is wired; it was previously left out on the reasoning below, and only one
# half of that reasoning survived contact with the upstream source.
#
# THE 4-MINUTE TTL IS STILL TRUE, and it is now the OPERATIONAL constraint rather than a reason to
# abstain. A https://plex.tv/claim token expires in ~4 minutes — far shorter than commit -> push ->
# Flux reconcile -> pod-roll if that cycle is left to its own timers. So the committed value is a
# documented PLACEHOLDER and the live token is injected by the operator inside that window, with an
# explicit `flux reconcile` to collapse the wait: runbooks/plex-claim.md is the timed procedure.
#
# ONE-SHOT, AND SAFE TO SIT HERE EXPIRED. Verified from plexinc/pms-docker's own first-run script
# (branch `master`, root/etc/cont-init.d/40-plex-first-run, confirmed byte-identical to the file in
# the running `:latest` image), NOT assumed:
# - the exchange (line 106) is guarded on TWO conditions: PLEX_CLAIM non-empty AND the
# PlexOnlineToken pref in /config empty. Once claimed, the block is SKIPPED — this env is a
# no-op on every subsequent start, whatever it holds. (Upstream README: "If server is already
# logged in, this parameter is ignored.") The claim itself persists on the /config PVC as before.
# - an invalid/expired token on a still-unclaimed /config gets a 403 with an empty body; curl's
# status is unchecked, the token `sed` yields "", the success branch at line 120 does not fire,
# there is no `else` and no `exit 1`, and the script (no `set -e`) exits 0. It comes up UNCLAIMED and
# SERVING — it does NOT crashloop. The image also leaves S6_BEHAVIOUR_IF_STAGE2_FAILS at its
# default 0, which continues startup even when a cont-init script fails.
# So the earlier "a placeholder claim Secret would crashloop the pod" note was WRONG about the
# placeholder; it was right only about an ABSENT Secret. secretKeyRef here has no `optional` field,
# so the Secret must always exist in the Flux graph — which is why it is committed, encrypted, and
# listed in apps/wave-0/kustomization.yaml rather than hand-created at claim time. No schema change
# is needed: `optional` would only matter if a missing Secret had to be tolerated, and it doesn't.
- name: PLEX_CLAIM
valueFrom:
secretKeyRef:
name: plex-claim
key: PLEX_CLAIM
# PLEX_CLAIM is deliberately NOT set here: a https://plex.tv/claim token has a ~4-min TTL, far
# shorter than the commit -> push -> Flux reconcile -> pod-roll cycle, so any token committed to git
# is already expired by the time the pod starts. Claiming is a one-time LIVE step (UI sign-in, or a
# single API POST) against the running server; the claim then persists on the /config PVC and
# survives pod rolls (including the roll this env change causes), so it never belongs in git.
# ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex
# which URL to advertise to plex.tv for REMOTE clients. The only candidate is
# https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL
@@ -85,12 +56,10 @@ storage:
# container's EPHEMERAL layer -> unclaimed + all state lost on restart. The image inits as
# root (s6-svscan PID 1), chowns /config to the plex user, then drops -> no fsGroup needed
# (verified in-cluster: PID1=root, "Plex Media Server"=plex). Persisting /config here also
# makes the claim survive restarts once claimed — which is what makes the PLEX_CLAIM env
# above a ONE-SHOT: the token is exchanged for a PlexOnlineToken written here, and the
# exchange is then skipped on every later start. (The older note here claimed a
# placeholder claim Secret "would crashloop the pod"; upstream's first-run script says
# otherwise — see the PLEX_CLAIM comment in `env` above. An ABSENT Secret is the hard
# failure, and that is why one is committed.)
# makes the claim survive restarts once claimed. To auto-claim on first boot instead of the
# UI sign-in, the operator adds a PLEX_CLAIM env from a fresh https://plex.tv/claim token
# (4-min TTL) — deliberately NOT wired here: the env secretKeyRef pattern has no `optional`,
# so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
# THE LIBRARY (docs/04 bulk tier). One READ-ONLY mount of the export ROOT, giving /media/movies,
# /media/tv and /media/photos — the whole tree the layout actually has (`ls /mnt/Storage/plex`:
# downloads/ movies/ photos/ tv/).
+1 -15
View File
@@ -1844,10 +1844,6 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
convention:
- `<svc>.yaml` -> wave(svc)
- `<svc>-bootstrap-secret.sops.yaml` -> wave(svc) (consumed by svc's env, same apply)
- `<svc>-claim-secret.sops.yaml` -> wave(svc) (a one-shot upstream-service claim token
consumed by svc's env — plex's plex.tv claim. Same wave as the app for the same reason the
bootstrap secret is: an env `secretKeyRef` has no `optional`, so the Secret must be applied
in the SAME reconcile as the workload or the pod is CreateContainerConfigError.)
- `<svc>-oauth2-proxy-credentials.sops.yaml` -> wave(svc) (forward-auth sidecar OIDC creds)
- `<svc>-db.yaml` -> wave(svc-db) (a strictly-lower wave; CNPG DB)
- `<svc>-db-app.sops.yaml` -> wave(svc-db) (CNPG initdb owner secret)
@@ -1874,11 +1870,6 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
w = waves[name]
add(w, f"{name}.yaml")
add(w, f"{name}-bootstrap-secret.sops.yaml")
# A one-shot claim token for an UPSTREAM service (plex's plex.tv claim). Same wave as the
# app: it is read by the app's env at pod start, and an env secretKeyRef has no `optional`,
# so a Secret applied in a later wave would leave the pod in CreateContainerConfigError.
# Placed only when the file exists, so services without one are unaffected.
add(w, f"{name}-claim-secret.sops.yaml")
# forward-auth (Option B): the per-app oauth2-proxy sidecar's OIDC client + cookie secret.
# Same wave as the app (the sidecar reads it at pod start); placed only when the file exists,
# so non-forward-auth apps are unaffected. Mirrors the bootstrap-secret naming convention.
@@ -2557,8 +2548,7 @@ def _selftest() -> int:
# 7a) plan_apps_waves: sidecars inherit their unit's wave; not-onboarded SDs are skipped;
# EXTRA_UNITS anchor strictly above; a file absent from the tree is never placed.
present = {
"a.yaml", "a-claim-secret.sops.yaml",
"svc.yaml", "svc-bootstrap-secret.sops.yaml",
"a.yaml", "svc.yaml", "svc-bootstrap-secret.sops.yaml",
"svc-db.yaml", "svc-db-app.sops.yaml", "svc-db-backup-credentials.sops.yaml",
"svc-db-backup-credentials-prod.sops.yaml",
"gitea.yaml",
@@ -2569,10 +2559,6 @@ def _selftest() -> int:
plan = plan_apps_waves(plan_sds, present)
flat = {f: wv for wv, fs in plan.items() for f in fs}
check(flat.get("a.yaml") == 0, "leaf app must land in wave 0")
check(flat.get("a-claim-secret.sops.yaml") == 0,
"a `<svc>-claim-secret.sops.yaml` (plex's plex.tv claim) must be placed in the SAME wave "
"as its app — an env secretKeyRef has no `optional`, so a later wave means "
"CreateContainerConfigError")
check(flat.get("svc-db.yaml") == 0 and flat.get("svc-db-app.sops.yaml") == 0
and flat.get("svc-db-backup-credentials.sops.yaml") == 0
and flat.get("svc-db-backup-credentials-prod.sops.yaml") == 0,
+59
View File
@@ -0,0 +1,59 @@
# infra/gitea — platform settings that are NOT GitOps-managed
Gitea's own repo settings (branch protection, required status checks, approvals) live
in Gitea's database, not in a manifest Flux reconciles. They are therefore **drift-prone
and invisible to the gate** — exactly how the state below went unnoticed.
## The gap this closes
Verified against the live API on 2026-09-29, against a `DECISIONS-LOG.md` entry
(2026-08-31) asserting the opposite:
| Claimed | Measured |
|---|---|
| `main` is a protected branch, gate is a required status check | `branch_protections` → `[]`; `protected:false`, `enable_status_check:false`, `required_approvals:0`, `user_can_push:true` |
| Runner `incluster-dind` online | **zero** runners registered at repo, org and site level |
So `main` accepted direct pushes, and the gate could not run at all. `CLAUDE.md` is the
rule; this directory is how the rule gets enforced by the machine.
## `branch-protection.sh`
Blocks direct pushes to `main`. Run it, then run the push probe it prints — **verify,
do not trust the API response**, since Gitea may let site admins bypass protection.
./infra/gitea/branch-protection.sh
It authenticates with the credential `git` already has stored for this host, and never
prints the secret.
## `runner-token-rotate.sh`
Mints a fresh instance-level registration token from the Gitea API and writes it into
`apps/wave-1/gitea-actions-runner-secret.sops.yaml` via `sops set`. **The token value is
never printed.**
./infra/gitea/runner-token-rotate.sh
A fresh token is usually **not sufficient on its own**. `act_runner` registers exactly once
and persists its identity in `/data/.runner` on the `act-runner-data` PVC. If Gitea no
longer holds a matching runner row, the pod starts, sees `.runner`, **skips registration**,
and polls with an identity Gitea does not recognise — so it never appears in the runner
list. That is precisely the observed symptom: zero runners, pod running, no error anyone
looks at. Clearing the PVC is what forces a real re-registration; the script prints the
commands.
## Sequence — order matters
1. **Get an Actions runner registered** — `runner-token-rotate.sh`, then clear the PVC so
the runner actually re-registers, then confirm it shows `online` in the API. Do not
skip the confirmation: the whole reason this went unnoticed is that a runner was
*recorded* as online without anyone checking.
2. **Watch the gate actually report on a real PR**, and note the context name it produced.
3. **Only then set `enable_status_check: true`** with that exact context.
4. Add the human reviewer as a repo collaborator, then raise `required_approvals` to 1.
> **Do not do step 3 before step 1.** A required status check with no runner leaves every
> PR permanently pending — including the PR that would fix it. The same applies to
> `gate.yml`'s `conformance` job: it is `if: false` and never runs, yet still publishes a
> `gate / conformance (pull_request)` context. Never list that one as required.
+57
View File
@@ -0,0 +1,57 @@
#!/usr/bin/env bash
# Enable branch protection on main for home-network-k3s-impl.
#
# Sets enable_push:false -> no direct pushes to main; changes land only by merging a PR.
#
# enable_status_check is deliberately FALSE. There is currently NO Gitea Actions runner
# registered (verified 2026-09-29: zero at repo, org and site level), so requiring the
# gate context would deadlock EVERY PR permanently -- including the PR that fixes it.
# Turn it on only after a runner is live and you have seen the gate actually report on
# a real PR, using the context name CI itself produced.
#
# required_approvals is 0 for the same class of reason: jakman85 is a site admin but NOT
# a repo collaborator, so requiring an approval now could leave nobody able to approve.
# Add the collaborator first, then raise this to 1.
set -euo pipefail
HOST=gitea.home.keithjackson.dev
OWNER=Home-Network
REPO=home-network-k3s-impl
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"
P="$(sed -n 's/^password=//p' <<<"$creds")"
[ -n "$U" ] && [ -n "$P" ] || { echo "no stored credential for $HOST" >&2; exit 1; }
echo "authenticating as '${U}' (secret not shown)"
echo "==> creating branch protection on 'main'"
jq -n '{
branch_name: "main",
enable_push: false,
enable_status_check: false,
status_check_contexts: [],
required_approvals: 0,
block_on_rejected_reviews: true,
block_on_outdated_branch: false,
dismiss_stale_approvals: true
}' | curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
-X POST -H 'Content-Type: application/json' --data @- \
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections" \
| jq -r 'if .branch_name then "created" else "API said: \(.message // tostring)" end'
echo "==> verifying (this is the part that matters)"
curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \
| jq -r '{protected, user_can_push, required_approvals, enable_status_check, status_check_contexts}'
cat <<'NOTE'
Now prove it actually blocks you -- do not take the API's word for it. From a clean
checkout on main, with nothing to lose:
git commit --allow-empty -m 'protection probe'
git push origin main # EXPECT: rejected by the remote
git reset --hard HEAD~1 # tidy up either way
If that push SUCCEEDS, Gitea is letting site admins bypass protection and this is not
yet doing what you want.
NOTE
+61
View File
@@ -0,0 +1,61 @@
#!/usr/bin/env bash
# Mint a fresh Gitea Actions registration token and write it into the SOPS secret.
#
# The token value is NEVER printed. It goes from the Gitea API into `sops set` and
# nowhere else. (It is briefly visible in this host's process list as a sops argument;
# acceptable on a single-operator workstation, and far better than echoing it.)
#
# This ALONE may not bring the runner back -- see "IMPORTANT" at the end of the output.
set -euo pipefail
HOST=gitea.home.keithjackson.dev
SECRET=apps/wave-1/gitea-actions-runner-secret.sops.yaml
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
cd "$REPO_ROOT"
[ -f "$SECRET" ] || { echo "not in the repo root, or $SECRET is missing" >&2; exit 1; }
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"
P="$(sed -n 's/^password=//p' <<<"$creds")"
[ -n "$U" ] && [ -n "$P" ] || { echo "no stored git credential for $HOST" >&2; exit 1; }
echo "==> minting an instance-level registration token as '${U}'"
TOKEN="$(curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
"https://${HOST}/api/v1/admin/runners/registration-token" | jq -r '.token // empty')"
[ -n "$TOKEN" ] || { echo "Gitea did not return a token" >&2; exit 1; }
echo "==> got a ${#TOKEN}-character token (value not shown)"
echo "==> writing it into ${SECRET} via sops"
DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run \
sops set "$SECRET" '["stringData"]["token"]' "\"${TOKEN}\""
unset TOKEN
echo "==> done. Verify the file is still encrypted (no plaintext token):"
grep -q 'ENC\[' "$SECRET" && echo " OK - stringData is still SOPS-encrypted" \
|| { echo " REFUSING TO CONTINUE: $SECRET is not encrypted" >&2; exit 1; }
cat <<'NOTE'
Next: commit this on a branch, open a PR, and merge it so Flux applies the new Secret.
IMPORTANT -- a new token is usually NOT sufficient on its own.
act_runner registers exactly ONCE and then persists its identity in /data/.runner on the
act-runner-data PVC. If Gitea no longer has a matching runner row (DB restore, the row was
deleted, the instance was rebuilt), the pod starts, sees .runner, SKIPS registration, and
polls with an identity Gitea does not recognise -- so it never appears in the runner list,
which is exactly the observed symptom (zero runners, pod running).
So after the Secret lands, force a genuine re-registration by clearing that state:
kubectl --context k3s-prod -n gitea-actions scale deploy/act-runner --replicas=0
kubectl --context k3s-prod -n gitea-actions delete pvc act-runner-data
kubectl --context k3s-prod -n gitea-actions scale deploy/act-runner --replicas=1
(The PVC holds only runner registration state -- no build artifacts, nothing durable.
Deleting it is the supported way to re-register. Flux recreates it from the manifest.)
Then confirm it actually registered -- do not assume:
curl -s -u '<user>' https://gitea.home.keithjackson.dev/api/v1/admin/actions/runners | jq
A runner must appear with status "online" and the labels ubuntu-latest / homelab.
NOTE
+1 -6
View File
@@ -6,7 +6,7 @@ Every alert rule in this platform carries an annotation of the form
runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"
46 of them, over 35 names. Until this directory existed, that path returned **HTTP 404** on the
46 of them, over the 35 names below. Until this directory existed, that path returned **HTTP 404** on the
live cluster: every alert that reached a human handed them a dead link at the exact moment they needed
help. These pages are what makes the link resolve.
@@ -108,11 +108,6 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
- **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
every green signal missed.
### Operator procedures (no alert links here)
- **`plex-claim`** — claiming the Plex Media Server. The plex.tv claim token lives ~4 minutes, so this
is a timed sequence: token → `sops` → push → **forced** `flux reconcile` → pod roll. Includes why a
changed Secret does not restart anything, and why an expired token is harmless.
### Virtualisation
- **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.
- **`kubevirt-launch-failed`** — a VMI reached the Failed phase. The three k3s-specific launch traps.
-5
View File
@@ -81,11 +81,6 @@ configMapGenerator:
- node-io=node-io.md
- node-memory=node-memory.md
- node-memory-swap=node-memory-swap.md
# Operator procedure, not an alert page (no alert links to it — the
# alert-runbooks-resolvable check treats that as advisory). It is served
# because the 4-minute plex.tv claim window is not a thing to go looking
# for in git: /runbooks/plex-claim has to resolve the moment it is needed.
- plex-claim=plex-claim.md
- pvc-capacity=pvc-capacity.md
- targets-down=targets-down.md
- update-rollback=update-rollback.md
-185
View File
@@ -1,185 +0,0 @@
# plex-claim — claiming the Plex Media Server inside the 4-minute token window
**Alert:** none. This is an **operator procedure**, not an incident page — it is indexed here because
it is the page you need at the moment you need it, with a stopwatch running.
**Files:** `catalog/plex.yaml` (env `PLEX_CLAIM`), `apps/wave-0/plex-claim-secret.sops.yaml`
(the SOPS Secret `plex-claim`, key `PLEX_CLAIM`), `apps/wave-0/plex.yaml` (the rendered golden).
## What this means here
Plex will not let you **create libraries** over the mounted bulk media share while the server is
**unclaimed** — the 8.33 T NFS tree at `/media` is there, readable, and unusable. Claiming binds the
server to a plex.tv account. It is a **one-shot** operation: the token is exchanged once for a durable
`PlexOnlineToken` written into `Preferences.xml` on the `plex-config` Longhorn PVC, and from then on
the `PLEX_CLAIM` env is **ignored on every start** (upstream gate:
`[ ! -z "${PLEX_CLAIM}" ] && [ -z "${token}" ]` in `root/etc/cont-init.d/40-plex-first-run`).
**A `https://plex.tv/claim` token lives ~4 minutes.** That is shorter than commit → push → Flux's
10-minute reconcile interval → pod roll if you let those run on their own timers, which is why this
page exists: every step below is written to be run back-to-back, with the reconcile **forced** rather
than waited for. Budget: token in hand at T+0, pod restarted by roughly T+2:00.
**An expired or wrong token does not break anything.** Verified against the image's own first-run
script (and the copy inside `plexinc/pms-docker:latest`): a failed exchange gets a 403 with an empty
body, `curl`'s status is never checked, the token `sed` yields empty, the `if` does not fire, there is
no `else` and no `exit 1`, and the script (no `set -e`) exits 0. The pod comes up **unclaimed and
serving**. So a burnt token costs you one more attempt, not an outage — and the repo deliberately
ships a **placeholder** in that Secret for exactly this reason (the env `secretKeyRef` has no
`optional`, so the Secret must always exist even when its value is useless).
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that
hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and that default has already changed once (k3s-dev until dev
was decommissioned on 2026-09-13, k3s-prod now). Do not run two `dev.sh` invocations at once.
Mutating `kubectl` and `flux reconcile` are blocked for agents here. **This whole page is operator
work** — the reconcile and the pod roll below are deliberate operator actions, and the durable change
is still a git change.
## First commands
**Before spending a token, find out whether the server is already claimed.** This prints only the
attribute NAME, never its value — do not widen it to print the token:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c \
"grep -o PlexOnlineToken= \"/config/Library/Application Support/Plex Media Server/Preferences.xml\" | head -1"'
```
`PlexOnlineToken=` printed ⇒ **already claimed**; stop, there is nothing to do and no token to burn.
Empty output ⇒ unclaimed, continue.
Is the workload healthy enough to claim at all, and is the wave reconciling?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex get deploy,pod,pvc'
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0'
```
## The 4-minute procedure
Have this page open and do not stop between steps.
1. **T+0 — get the token.** Sign in to plex.tv as the account that should OWN this server, open
<https://plex.tv/claim>, copy the `claim-…` string. The clock starts now.
2. **T+0:15 — put it in the Secret**, in place, encrypted, without it ever touching a plaintext file:
```
infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
```
Replace the `PLEX_CLAIM:` value (the placeholder `replace-me-with-a-fresh-plex-tv-claim-token`, or
a previous expired token) and save. `sops` re-encrypts on write to the committed age recipient.
3. **T+0:45 — commit and push.** `git push origin main` **is** the deploy; there is no other apply
step.
```
git add apps/wave-0/plex-claim-secret.sops.yaml
git commit -m 'chore(plex): fresh plex.tv claim token'
git push origin main
```
4. **T+1:00 — force the reconcile immediately. This is the one command that saves the window** — it
pulls the new commit and re-applies wave 0 instead of waiting up to 10 minutes for the interval:
```
infra/devcontainer/dev.sh run bash -c \
'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-0 --with-source'
```
`--with-source` is the load-bearing flag: it reconciles the `flux-system` GitRepository first, so
the Kustomization applies **your commit** and not the previously-fetched revision.
5. **T+1:30 — roll the pod.** This step is **mandatory and easy to forget**: a changed Secret does
**not** restart anything. `PLEX_CLAIM` is injected by `secretKeyRef` and read only at container
start, and the Deployment's pod template is byte-identical across this change, so Flux applies the
new Secret and the running pod keeps the old value forever.
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex rollout restart deployment/plex'
```
6. **T+2:00 — confirm the exchange happened**, in the new pod's init log:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex logs deploy/plex --since=5m | grep -i -e claim -e token'
```
* `Attempting to obtain server token from claim token` **and** `Token obtained successfully`
⇒ **claimed.** Re-run the First-commands `PlexOnlineToken=` check to confirm it persisted, then
create the libraries over `/media/movies`, `/media/tv`, `/media/photos`.
* `Attempting to obtain server token from claim token` with **no** success line ⇒ the token was
expired or invalid. Nothing is broken; go back to step 1 with a new token.
* **Neither line** ⇒ the exchange was skipped, which means `/config` already had a
`PlexOnlineToken` (already claimed) or the env did not reach the container. Check the pod actually
has the variable, by NAME only:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
-o jsonpath="{.items[0].spec.containers[0].env[*].name}"'
```
## Likely causes, most likely first
When the claim does not take, in order of how often it is the answer here:
1. **The token expired** — you spent more than ~4 minutes, usually in step 3 or waiting on step 4.
Symptom: the "Attempting…" line with no success line. Remedy: a new token, and force the reconcile
rather than waiting for it.
2. **The pod was never restarted** (step 5 skipped). Symptom: no claim log lines at all from a pod
whose `AGE` predates your push. The Secret is correct in git and in the cluster and the running
container still holds the placeholder.
3. **Flux had not applied the commit yet** when you rolled the pod — step 5 run before step 4
finished, so the new pod read the OLD Secret. Check
`kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0 -o jsonpath='{.status.lastAppliedRevision}'`
against your commit, then roll again.
4. **Already claimed** — the exchange is gated on `PlexOnlineToken` being absent from `/config`, so a
server claimed earlier ignores every later token silently. This is the benign case.
5. **`plex.tv` unreachable.** The upstream `curl` has no `--max-time`/`--connect-timeout`, so a stalled
connection can leave the container sitting in cont-init with **nothing listening on 32400** — the
one genuine hang risk in this path. Symptom: pod Running but never Ready, log stops after
"Attempting…". Check egress (`plex` is `egress: internet`) rather than re-rolling repeatedly.
6. **Wrong plex.tv account.** The claim succeeds and binds the server to whoever was signed in at
step 1. Unclaiming is a plex.tv-side operation on that account — see "escalate or stop".
## When to escalate or stop
- **Stop after two failed token attempts** and fix the timing, not the token. The third rushed attempt
is how people start editing live Secrets by hand, which leaves the cluster holding a value that is
in no git commit — precisely the drift `qbittorrent-bootstrap` had to be recovered from.
- **Never `kubectl edit`/`patch` the `plex-claim` Secret in the cluster, and never
`kubectl create secret` it.** Everything durable goes through the SOPS file; a live-only Secret is
reverted (or adopted with the wrong value) on the next reconcile.
- **Never `sops -d` this file into a terminal, a log, or a shared screen**, and never widen the
`PlexOnlineToken=` check into one that prints the value. The token is a plex.tv account credential.
- **If the server got claimed to the wrong account, stop and get the operator.** Reversing it means
removing the server from that account on plex.tv and clearing `PlexOnlineToken` from
`Preferences.xml` on the PVC — a destructive edit of app state on the replicated tier, not a git
change.
- **Do not delete the `plex-config` PVC to "start clean".** It holds the library database, metadata
and watch state (`policy: daily-offcluster`), and it is expensive to rebuild. The claim is a
two-line pref, not a reason to lose the DB.
- After a successful claim, **leave the expired token in the SOPS file**. It is inert (the exchange is
skipped once claimed) and the Secret must keep existing, because the env `secretKeyRef` has no
`optional` — deleting it would put the pod into `CreateContainerConfigError`.
---
Source of truth: `runbooks/plex-claim.md`. Index: `/runbooks/`
Upstream behaviour cited here is from `plexinc/pms-docker` branch `master`,
`root/etc/cont-init.d/40-plex-first-run` (s6-overlay v2), cross-checked against the same file inside
the running `plexinc/pms-docker:latest` image.