From 8e017fad919a54a75595f96c36f65832d3180077 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 29 Sep 2026 01:42:35 -0500 Subject: [PATCH 1/2] docs(runbooks): the PR flow, not `git push origin main` -- that instruction is now blocked main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) --- DECISIONS-LOG.md | 82 ++++++++++++++++++++ infra/plex/claim-ship.sh | 93 +++++++++++++++++++++++ runbooks/README.md | 3 +- runbooks/backup-failure.md | 6 +- runbooks/backup-missed.md | 6 +- runbooks/backup-partial.md | 6 +- runbooks/cdi-import-stuck.md | 6 +- runbooks/cert-expiry.md | 6 +- runbooks/cert-issuance.md | 6 +- runbooks/cnpg-archiving.md | 6 +- runbooks/cnpg-backup-missed.md | 6 +- runbooks/cnpg-exporter.md | 6 +- runbooks/cnpg-primary-spread.md | 6 +- runbooks/cnpg-wal-growth.md | 6 +- runbooks/denied-flows.md | 6 +- runbooks/flux-reconcile.md | 8 +- runbooks/flux-suspended.md | 6 +- runbooks/image-automation.md | 6 +- runbooks/image-policy-stalled.md | 6 +- runbooks/keycloak-client-secrets.md | 6 +- runbooks/keycloak-db.md | 6 +- runbooks/keycloak-down.md | 6 +- runbooks/kubevirt-control-plane.md | 6 +- runbooks/kubevirt-crashloop.md | 6 +- runbooks/kubevirt-launch-failed.md | 6 +- runbooks/longhorn-capacity.md | 6 +- runbooks/longhorn-degraded.md | 6 +- runbooks/longhorn-faulted.md | 6 +- runbooks/monitoring-capacity.md | 6 +- runbooks/node-cpu.md | 6 +- runbooks/node-disk.md | 6 +- runbooks/node-down.md | 6 +- runbooks/node-io.md | 6 +- runbooks/node-memory.md | 6 +- runbooks/plex-claim.md | 114 ++++++++++++++-------------- runbooks/pvc-capacity.md | 6 +- runbooks/targets-down.md | 6 +- runbooks/update-rollback.md | 6 +- runbooks/vpn-down.md | 6 +- 39 files changed, 377 insertions(+), 127 deletions(-) create mode 100755 infra/plex/claim-ship.sh diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index bee4174..3713547 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -4781,3 +4781,85 @@ accurate before the PR flow and are stale now. Correcting them is tracked, not d 3. Land `feat/plex-claim-sops-secret` (PR #1) once the gate can actually run. 4. Claim plex, after checking it is not already claimed. 5. Correct the 37 runbooks' `git push origin main` wording. + +## 2026-09-29 — Plex is claimed; the CI/CD gate that "enforced" it never existed. **PM HANDOFF.** + +Began as "inject a plex.tv claim token". Ended by discovering the delivery pipeline was fiction, +rebuilding it, and shipping the token through it. Plex is **claimed** — `PlexOnlineToken` is +present in `/config` on the `plex-config` PVC, so the exchange is now a permanent no-op and the +library scan is running over the 8.33T NFS share. + +### The finding: both halves of the 2026-08-31 CI claim were false + +That entry recorded "Gitea Actions CI is LIVE + enforcing … required status check; runner +`incluster-dind` online". Measured against the live API on 2026-09-29: + +| Claimed | Measured | +|---|---| +| `main` protected, gate required | `branch_protections` → `[]`; `protected:false`, `enable_status_check:false`, `required_approvals:0`, `user_can_push:true` | +| Runner online | **zero** runners at repo, org AND site level | + +**PR #1 was the first pull request this repository has ever had.** Its contexts sat `pending` +for 30.6 minutes across 122 polls and never reported. For a month, every "the gate enforces this" +statement rested on an assertion nobody had checked — the same fail-open class as 2026-09-18, +and the reason it hid so long is that the log itself was the evidence. + +### Root cause of the dead runner + +`act_runner` registers **once** and persists its identity in `/data/.runner` on the +`act-runner-data` PVC. Gitea no longer held a matching row, so the pod booted, saw `.runner`, +**skipped registration**, and polled with an identity Gitea did not recognise. Symptom: zero +runners, pod Running, nothing crashlooping, no error anyone reads. Deleting the PVC forced a +genuine re-registration (`.runner is missing` → `Runner registered successfully`). + +It then did not come back for several minutes, because Flux could not recreate the PVC: +`apps-wave-1` was blocked four levels deep on a **stale** `DependencyNotReady` for +`platform-config`, which itself read `Ready=True`. Each wave re-evaluates only on its own +`interval: 10m`, so a resource deleted at the bottom of the chain can take ~40 min to return. +**Force it** with `flux reconcile kustomization apps-wave-1 --with-source` rather than waiting. + +### Corrections to things this log and the runbooks asserted + +- **A skipped Gitea job reports `success`, not pending.** Measured: `gate / conformance` returned + `success` / "Has been skipped". An earlier entry (#2) said requiring it would deadlock every PR. + Wrong — it would pass **vacuously**, forever, while nothing ran. Under HARD RULE 4 that is worse + than a deadlock, because a deadlock is loud. Never require it; require + `gate / gate (pull_request)` alone. Corrected in #3. +- **`gate.yml`'s push trigger omits `feat/**`** (it lists `feature/** fix/** docs/** chore/**`). + With `act_runner` capacity 1, a branch prefix that IS in that list fires a second redundant run + which serialises ahead of the PR run and doubles wall-clock. Exploited deliberately in + `infra/plex/claim-ship.sh`; the honest fix is to align the trigger list. +- **The "~4-minute TTL"** asserted in four places was **uncited**, and upstream `pms-docker` + documents no expiry at all. The authoritative reading is the countdown plex.tv/claim renders at + issue time. Removed from `runbooks/plex-claim.md`; the same claim still sits in + `catalog/plex.yaml` and the Secret's header comment (left alone — editing a `.sops.yaml` + comment risks the MAC, and touching `catalog/plex.yaml` re-renders a golden). + +### Measured, once the pipeline actually worked + +`gate / gate (pull_request)` 1m34s–1m58s. Full claim loop end to end **~2m45s** — comfortably +inside the window, which retires the theory that a short-lived token cannot survive a PR flow. +That theory was mine and it was wrong: the pipeline was never too slow, it simply did not exist. + +### State now + +- `main` **protected**, `user_can_push:false`. Runner `incluster-dind` online. +- `CLAUDE.md` is the agent delivery contract; `infra/gitea/` holds the protection + runner scripts. +- **All 37 runbooks** that instructed `git push origin main` now describe the PR flow — that + instruction had become actively blocked, not merely stale. +- `infra/plex/claim-ship.sh` drives the claim loop end to end. + +### Next steps + +1. **`enable_status_check: true`** with `gate / gate (pull_request)` as the only context. The + prerequisite is met: the gate has been observed green on real PRs. +2. Add `jakman85` as a repo **collaborator**, then `required_approvals: 1`. (Currently the only + org member is `gitea-admin`, which agents authenticate as — requiring approval before that is + fixed would leave nobody able to approve.) +3. Run the **push probe** against protection: `gitea-admin` is a site admin and Gitea may let it + bypass. `user_can_push:false` is the API's opinion, not proof. +4. Align `gate.yml`'s push trigger with this repo's `feat/**` prefix. +5. Convert the `dind` container to a **native sidecar** with a `startupProbe`, per the standing + FOLLOW-UP note in `apps/wave-1/gitea-actions.yaml`. It deletes both hand-rolled wait loops and + the envsubst hazard. Land it ALONE — it is the merge gate for the whole repo. +6. Create the Plex libraries over `/media/movies`, `/media/tv`, `/media/photos`. diff --git a/infra/plex/claim-ship.sh b/infra/plex/claim-ship.sh new file mode 100755 index 0000000..4dce82a --- /dev/null +++ b/infra/plex/claim-ship.sh @@ -0,0 +1,93 @@ +#!/usr/bin/env bash +# plex-claim-ship.sh — run this IMMEDIATELY AFTER editing the SOPS secret with a fresh +# plex.tv claim token. It drives the whole CI/CD loop end to end and prints a running +# stopwatch so you can see the budget being spent. +# +# branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verify +# +# It never sees, prints, or handles the token: the file is already encrypted by sops. +set -euo pipefail +HOST=gitea.home.keithjackson.dev +OWNER=Home-Network +REPO=home-network-k3s-impl +SECRET=apps/wave-0/plex-claim-secret.sops.yaml +T0=$(date +%s) +el() { printf '\n[T+%02d:%02d] %s\n' $(( ($(date +%s)-T0)/60 )) $(( ($(date +%s)-T0)%60 )) "$1"; } + +cd "$(git rev-parse --show-toplevel)" + +# --- preflight: refuse to ship a file that is not what we think it is ------------- +git diff --quiet -- "$SECRET" && { echo "ERROR: $SECRET is unchanged. Edit it first:"; \ + echo " infra/devcontainer/dev.sh run sops $SECRET"; exit 1; } +grep -q 'PLEX_CLAIM: ENC\[' "$SECRET" || { echo "ERROR: $SECRET is not encrypted. REFUSING."; exit 1; } +[ -z "$(git status --porcelain | grep -v " $SECRET\$")" ] || echo "NOTE: other files are dirty; only $SECRET will be committed." + +creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)" +U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")" +api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; } + +# feat/ prefix ON PURPOSE. act_runner capacity is 1, and gate.yml's PUSH trigger covers +# feature/** fix/** docs/** chore/** but NOT feat/** — so a chore/ branch would fire a +# second, redundant run that SERIALISES ahead of the PR run and doubles the wall clock. +BRANCH="feat/plex-claim-token-$(date +%H%M%S)" + +el "committing on $BRANCH" +git checkout -q -b "$BRANCH" +git add "$SECRET" +git commit -q -m 'chore(plex): fresh plex.tv claim token + +Injected inside the plex.tv claim window per runbooks/plex-claim.md.' +SHA=$(git rev-parse HEAD) + +el "pushing" +git push -q -u origin "$BRANCH" + +el "opening PR" +PR=$(jq -n --arg h "$BRANCH" --arg b main --arg t 'chore(plex): fresh plex.tv claim token' \ + '{head:$h,base:$b,title:$t,body:"Token injection inside the claim window. Wiring already reviewed in #1."}' \ + | api -X POST -H 'Content-Type: application/json' --data @- \ + "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls" | jq -r '.number') +[ -n "$PR" ] && [ "$PR" != "null" ] || { echo "PR creation failed"; exit 1; } +echo " PR #$PR" + +el "waiting for the gate (expect ~1m40s)" +for i in $(seq 1 60); do + st=$(api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/commits/${SHA}/status" | jq -r '.state') + printf '\r gate=%s [T+%ss] ' "$st" "$(( $(date +%s)-T0 ))" + case "$st" in success) echo; break;; failure|error) echo; echo "GATE FAILED — stopping."; exit 1;; esac + sleep 5 +done +[ "$st" = "success" ] || { echo; echo "gate did not finish in time"; exit 1; } + +el "merging PR #$PR" +api -X POST -H 'Content-Type: application/json' -d '{"Do":"merge"}' \ + "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls/${PR}/merge" >/dev/null +git checkout -q main && git pull -q --ff-only origin main + +el "forcing the Flux reconcile (--with-source, or it applies the OLD revision)" +./infra/devcontainer/dev.sh run bash -c \ + 'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-0 --with-source' + +# MANDATORY here. Only the Secret ciphertext changed, so the Deployment pod template is +# byte-identical and NOTHING restarts on its own. PLEX_CLAIM is read once, at container +# start. Skipping this is failure mode #2 in runbooks/plex-claim.md. +el "rolling the pod (mandatory — a changed Secret restarts nothing)" +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n plex rollout restart deployment/plex && \ + kubectl --context k3s-prod -n plex rollout status deployment/plex --timeout=120s' + +el "verdict" +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n plex logs deploy/plex --since=3m | grep -i -e claim -e token || true' +cat <<'NOTE' + + "Attempting to obtain server token..." + "Token obtained successfully" -> CLAIMED + "Attempting..." with no success line -> token expired; retry, nothing is broken + neither line -> already claimed, or env did not reach the container + +Confirm it persisted (prints the attribute NAME only, never its value): + + infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c \ + "grep -o PlexOnlineToken= \"/config/Library/Application Support/Plex Media Server/Preferences.xml\" | head -1"' +NOTE diff --git a/runbooks/README.md b/runbooks/README.md index 5dbe033..a2f9db8 100644 --- a/runbooks/README.md +++ b/runbooks/README.md @@ -33,7 +33,8 @@ negative is how a CLI quirk becomes real data loss. Write `backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. -`git push origin main` **is** the deploy; there is no other apply step. Durable changes go through git. +Merging a PR into `main` **is** the deploy; there is no other apply step. `main` is protected, so a +direct push is rejected — durable changes go through a branch, a PR, and a green gate. ## Alerts you should not trust the silence of (measured 2026-09-13) diff --git a/runbooks/backup-failure.md b/runbooks/backup-failure.md index b87a65c..f4422ed 100644 --- a/runbooks/backup-failure.md +++ b/runbooks/backup-failure.md @@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/backup-missed.md b/runbooks/backup-missed.md index a5b8958..a4db81f 100644 --- a/runbooks/backup-missed.md +++ b/runbooks/backup-missed.md @@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/backup-partial.md b/runbooks/backup-partial.md index af522af..dc977f1 100644 --- a/runbooks/backup-partial.md +++ b/runbooks/backup-partial.md @@ -78,8 +78,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/cdi-import-stuck.md b/runbooks/cdi-import-stuck.md index 204559e..6bdb0e3 100644 --- a/runbooks/cdi-import-stuck.md +++ b/runbooks/cdi-import-stuck.md @@ -55,8 +55,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/cert-expiry.md b/runbooks/cert-expiry.md index eceabc6..9e2ddc0 100644 --- a/runbooks/cert-expiry.md +++ b/runbooks/cert-expiry.md @@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/cert-issuance.md b/runbooks/cert-issuance.md index b278f30..b736830 100644 --- a/runbooks/cert-issuance.md +++ b/runbooks/cert-issuance.md @@ -63,8 +63,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/cnpg-archiving.md b/runbooks/cnpg-archiving.md index 4505c6d..6a80734 100644 --- a/runbooks/cnpg-archiving.md +++ b/runbooks/cnpg-archiving.md @@ -87,8 +87,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/cnpg-backup-missed.md b/runbooks/cnpg-backup-missed.md index c274167..76b2663 100644 --- a/runbooks/cnpg-backup-missed.md +++ b/runbooks/cnpg-backup-missed.md @@ -69,8 +69,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/cnpg-exporter.md b/runbooks/cnpg-exporter.md index 7714561..17fb803 100644 --- a/runbooks/cnpg-exporter.md +++ b/runbooks/cnpg-exporter.md @@ -71,8 +71,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/cnpg-primary-spread.md b/runbooks/cnpg-primary-spread.md index 7287b68..e61b804 100644 --- a/runbooks/cnpg-primary-spread.md +++ b/runbooks/cnpg-primary-spread.md @@ -68,8 +68,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/cnpg-wal-growth.md b/runbooks/cnpg-wal-growth.md index 4d11b76..1de8531 100644 --- a/runbooks/cnpg-wal-growth.md +++ b/runbooks/cnpg-wal-growth.md @@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/denied-flows.md b/runbooks/denied-flows.md index a44af99..e205728 100644 --- a/runbooks/denied-flows.md +++ b/runbooks/denied-flows.md @@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/flux-reconcile.md b/runbooks/flux-reconcile.md index 70d12bb..3e45a9d 100644 --- a/runbooks/flux-reconcile.md +++ b/runbooks/flux-reconcile.md @@ -7,7 +7,7 @@ ## What this means here Something in the GitOps chain is stuck at Ready=False. On this cluster that is more consequential than -it sounds, because **`git push origin main` IS the deploy** — there is no other apply step. A stuck +it sounds, because **merging a PR into `main` IS the deploy** — there is no other apply step. A stuck Kustomization means git and the cluster have diverged and nothing will close the gap. The chain has **14 tiers**: `flux-system` → `cilium` → `platform` → `platform-config` → @@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/flux-suspended.md b/runbooks/flux-suspended.md index 3a766bc..69f5222 100644 --- a/runbooks/flux-suspended.md +++ b/runbooks/flux-suspended.md @@ -58,8 +58,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/image-automation.md b/runbooks/image-automation.md index 596f1e5..596f3fe 100644 --- a/runbooks/image-automation.md +++ b/runbooks/image-automation.md @@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/image-policy-stalled.md b/runbooks/image-policy-stalled.md index 2edcf7d..32ce2a2 100644 --- a/runbooks/image-policy-stalled.md +++ b/runbooks/image-policy-stalled.md @@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/keycloak-client-secrets.md b/runbooks/keycloak-client-secrets.md index 9c72463..48d0057 100644 --- a/runbooks/keycloak-client-secrets.md +++ b/runbooks/keycloak-client-secrets.md @@ -87,8 +87,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/keycloak-db.md b/runbooks/keycloak-db.md index 0fd0d7a..f6bc981 100644 --- a/runbooks/keycloak-db.md +++ b/runbooks/keycloak-db.md @@ -67,8 +67,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/keycloak-down.md b/runbooks/keycloak-down.md index d14948d..3343589 100644 --- a/runbooks/keycloak-down.md +++ b/runbooks/keycloak-down.md @@ -59,8 +59,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/kubevirt-control-plane.md b/runbooks/kubevirt-control-plane.md index c6c3cfb..6dc9768 100644 --- a/runbooks/kubevirt-control-plane.md +++ b/runbooks/kubevirt-control-plane.md @@ -57,8 +57,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/kubevirt-crashloop.md b/runbooks/kubevirt-crashloop.md index 9d612cb..23527d9 100644 --- a/runbooks/kubevirt-crashloop.md +++ b/runbooks/kubevirt-crashloop.md @@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/kubevirt-launch-failed.md b/runbooks/kubevirt-launch-failed.md index 6308be6..3117e07 100644 --- a/runbooks/kubevirt-launch-failed.md +++ b/runbooks/kubevirt-launch-failed.md @@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/longhorn-capacity.md b/runbooks/longhorn-capacity.md index 62ea29e..17ce8c4 100644 --- a/runbooks/longhorn-capacity.md +++ b/runbooks/longhorn-capacity.md @@ -83,8 +83,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/longhorn-degraded.md b/runbooks/longhorn-degraded.md index c74143c..eee4f34 100644 --- a/runbooks/longhorn-degraded.md +++ b/runbooks/longhorn-degraded.md @@ -74,8 +74,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/longhorn-faulted.md b/runbooks/longhorn-faulted.md index 3b210ad..8506223 100644 --- a/runbooks/longhorn-faulted.md +++ b/runbooks/longhorn-faulted.md @@ -75,8 +75,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/monitoring-capacity.md b/runbooks/monitoring-capacity.md index 3a56dad..4fd99a8 100644 --- a/runbooks/monitoring-capacity.md +++ b/runbooks/monitoring-capacity.md @@ -52,8 +52,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/node-cpu.md b/runbooks/node-cpu.md index 3a5e4a6..82f5a24 100644 --- a/runbooks/node-cpu.md +++ b/runbooks/node-cpu.md @@ -49,8 +49,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/node-disk.md b/runbooks/node-disk.md index a9392fc..ee824a8 100644 --- a/runbooks/node-disk.md +++ b/runbooks/node-disk.md @@ -56,8 +56,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/node-down.md b/runbooks/node-down.md index 6974d70..41f6caa 100644 --- a/runbooks/node-down.md +++ b/runbooks/node-down.md @@ -56,8 +56,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/node-io.md b/runbooks/node-io.md index 781315b..b8cb942 100644 --- a/runbooks/node-io.md +++ b/runbooks/node-io.md @@ -227,8 +227,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/node-memory.md b/runbooks/node-memory.md index 78c6784..b02a671 100644 --- a/runbooks/node-memory.md +++ b/runbooks/node-memory.md @@ -52,8 +52,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/plex-claim.md b/runbooks/plex-claim.md index 791745f..d9a255b 100644 --- a/runbooks/plex-claim.md +++ b/runbooks/plex-claim.md @@ -1,4 +1,4 @@ -# plex-claim — claiming the Plex Media Server inside the 4-minute token window +# plex-claim — claiming the Plex Media Server inside the token window **Alert:** none. This is an **operator procedure**, not an incident page — it is indexed here because it is the page you need at the moment you need it, with a stopwatch running. @@ -14,10 +14,11 @@ server to a plex.tv account. It is a **one-shot** operation: the token is exchan the `PLEX_CLAIM` env is **ignored on every start** (upstream gate: `[ ! -z "${PLEX_CLAIM}" ] && [ -z "${token}" ]` in `root/etc/cont-init.d/40-plex-first-run`). -**A `https://plex.tv/claim` token lives ~4 minutes.** That is shorter than commit → push → Flux's -10-minute reconcile interval → pod roll if you let those run on their own timers, which is why this -page exists: every step below is written to be run back-to-back, with the reconcile **forced** rather -than waited for. Budget: token in hand at T+0, pod restarted by roughly T+2:00. +**A `https://plex.tv/claim` token is short-lived — minutes, not hours.** Read the countdown the +claim page shows you; do not budget against a number written here. That is shorter than PR → gate → +merge → Flux's 10-minute reconcile interval → pod roll if you let those run on their own timers, +which is why this page exists: the loop is scripted and the reconcile is **forced** rather than +waited for. Measured end to end on 2026-09-29: **~2m45s**. **An expired or wrong token does not break anything.** Verified against the image's own first-run script (and the copy inside `plexinc/pms-docker:latest`): a failed exchange gets a 403 with an empty @@ -66,81 +67,84 @@ infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0' ``` -## The 4-minute procedure +## The procedure -Have this page open and do not stop between steps. +> **STATUS: this server was CLAIMED on 2026-09-29** and `PlexOnlineToken` is present in +> `/config` on the `plex-config` PVC. The upstream exchange is gated on that pref being +> *absent*, so `PLEX_CLAIM` is now a **no-op on every start** whatever it holds. You only need +> this page again after a `/config` wipe or a rebuild onto an empty PVC. + +**On the clock.** The token is short-lived — **read the countdown that plex.tv/claim shows you** +rather than trusting a number written here. (Earlier revisions of this page asserted a "~4-minute +TTL" in four places with no citation; upstream `pms-docker` documents no expiry at all. The page +is the only authority, and it is on screen at T+0.) + +**A burnt token costs nothing.** An expired token gets a 403 with an empty body, `curl`'s status is +never checked, the `if` does not fire, there is no `else` and no `exit 1`, and the script (no +`set -e`) exits 0. The pod comes up unclaimed and serving. So retry freely. + +### One command 1. **T+0 — get the token.** Sign in to plex.tv as the account that should OWN this server, open - , copy the `claim-…` string. The clock starts now. + , copy the `claim-…` string. -2. **T+0:15 — put it in the Secret**, in place, encrypted, without it ever touching a plaintext file: +2. **Put it in the Secret** — in place, encrypted, never touching a plaintext file: ``` infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml ``` - Replace the `PLEX_CLAIM:` value (the placeholder `replace-me-with-a-fresh-plex-tv-claim-token`, or - a previous expired token) and save. `sops` re-encrypts on write to the committed age recipient. - -3. **T+0:45 — commit and push.** `git push origin main` **is** the deploy; there is no other apply - step. +3. **Ship it.** This drives the whole delivery loop and prints a running stopwatch: ``` - git add apps/wave-0/plex-claim-secret.sops.yaml - git commit -m 'chore(plex): fresh plex.tv claim token' - git push origin main + ./infra/plex/claim-ship.sh ``` -4. **T+1:00 — force the reconcile immediately. This is the one command that saves the window** — it - pulls the new commit and re-applies wave 0 instead of waiting up to 10 minutes for the interval: + branch → commit → push → PR → gate → merge → `flux reconcile --with-source` → `rollout restart` + → verdict. It refuses to run if the file is unchanged or not encrypted. - ``` - infra/devcontainer/dev.sh run bash -c \ - 'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-0 --with-source' - ``` +**Measured end to end, 2026-09-29: ~2m45s.** Gate ~1m40s, merge + reconcile ~25s, pod roll and +container start ~30s. - `--with-source` is the load-bearing flag: it reconciles the `flux-system` GitRepository first, so - the Kustomization applies **your commit** and not the previously-fetched revision. +### Why it is a script and not six pasted commands -5. **T+1:30 — roll the pod.** This step is **mandatory and easy to forget**: a changed Secret does - **not** restart anything. `PLEX_CLAIM` is injected by `secretKeyRef` and read only at container - start, and the Deployment's pod template is byte-identical across this change, so Flux applies the - new Secret and the running pod keeps the old value forever. +`main` is a **protected branch** — the direct push this page used to instruct is now rejected — so +the deploy is a PR merged after a green gate. Three of those steps are easy to get wrong under time +pressure, and two are silent when skipped: - ``` - infra/devcontainer/dev.sh run bash -c \ - 'kubectl --context k3s-prod -n plex rollout restart deployment/plex' - ``` +- **`--with-source` is load-bearing.** Without it the Kustomization applies the previously-fetched + revision, not your commit. +- **The pod roll is mandatory.** Only the Secret's ciphertext changes, so the Deployment's pod + template is byte-identical and *nothing restarts on its own*. `PLEX_CLAIM` is read once, at + container start. Skip this and the new Secret sits in the cluster while the running pod keeps the + old value forever. +- **The branch is named `feat/…` on purpose.** `act_runner` capacity is 1, and `gate.yml`'s push + trigger covers `feature/** fix/** docs/** chore/**` but *not* `feat/**`. A `chore/` branch fires a + second redundant run that serialises ahead of the PR run and roughly doubles the wall clock. -6. **T+2:00 — confirm the exchange happened**, in the new pod's init log: +### Reading the verdict - ``` - infra/devcontainer/dev.sh run bash -c \ - 'kubectl --context k3s-prod -n plex logs deploy/plex --since=5m | grep -i -e claim -e token' - ``` +* `Attempting to obtain server token from claim token` **and** `Token obtained successfully` + ⇒ **claimed.** Re-run the First-commands `PlexOnlineToken=` check to confirm it persisted, then + create the libraries over `/media/movies`, `/media/tv`, `/media/photos`. +* `Attempting…` with **no** success line ⇒ the token was expired or invalid. Nothing is broken; go + back to step 1 with a new one. +* **Neither line** ⇒ the exchange was skipped: `/config` already had a `PlexOnlineToken` (already + claimed — the expected case now), or the env did not reach the container. Check by NAME only: - * `Attempting to obtain server token from claim token` **and** `Token obtained successfully` - ⇒ **claimed.** Re-run the First-commands `PlexOnlineToken=` check to confirm it persisted, then - create the libraries over `/media/movies`, `/media/tv`, `/media/photos`. - * `Attempting to obtain server token from claim token` with **no** success line ⇒ the token was - expired or invalid. Nothing is broken; go back to step 1 with a new token. - * **Neither line** ⇒ the exchange was skipped, which means `/config` already had a - `PlexOnlineToken` (already claimed) or the env did not reach the container. Check the pod actually - has the variable, by NAME only: - - ``` - infra/devcontainer/dev.sh run bash -c \ - 'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \ - -o jsonpath="{.items[0].spec.containers[0].env[*].name}"' - ``` + ``` + infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \ + -o jsonpath="{.items[0].spec.containers[0].env[*].name}"' + ``` ## Likely causes, most likely first When the claim does not take, in order of how often it is the answer here: -1. **The token expired** — you spent more than ~4 minutes, usually in step 3 or waiting on step 4. - Symptom: the "Attempting…" line with no success line. Remedy: a new token, and force the reconcile - rather than waiting for it. +1. **The token expired** — you exceeded the countdown, usually by reviewing the PR instead of + merging it the moment the gate went green. Symptom: the "Attempting…" line with no success line. + Remedy: a new token, and let `claim-ship.sh` drive the loop rather than pasting steps. 2. **The pod was never restarted** (step 5 skipped). Symptom: no claim log lines at all from a pod whose `AGE` predates your push. The Secret is correct in git and in the cluster and the running container still holds the placeholder. diff --git a/runbooks/pvc-capacity.md b/runbooks/pvc-capacity.md index 241d109..35c0c81 100644 --- a/runbooks/pvc-capacity.md +++ b/runbooks/pvc-capacity.md @@ -58,8 +58,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/targets-down.md b/runbooks/targets-down.md index 1094441..03e384d 100644 --- a/runbooks/targets-down.md +++ b/runbooks/targets-down.md @@ -60,8 +60,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/update-rollback.md b/runbooks/update-rollback.md index 2f49e77..a9cc851 100644 --- a/runbooks/update-rollback.md +++ b/runbooks/update-rollback.md @@ -54,8 +54,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands diff --git a/runbooks/vpn-down.md b/runbooks/vpn-down.md index caa7765..00d5495 100644 --- a/runbooks/vpn-down.md +++ b/runbooks/vpn-down.md @@ -66,8 +66,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for -agents here: every durable change goes through git (`git push origin main` **is** the deploy). -Where this runbook gives a mutating command it is for an operator, and it says so. +agents here: every durable change goes through git. **`main` is a protected branch — a direct push +is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is +green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, +and it says so. ## First commands From 761e25cc3f8bcc05dccd553dbe59f7bfe5af64d2 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 29 Sep 2026 01:57:47 -0500 Subject: [PATCH 2/2] docs(runbooks): two more uncited "4-minute" claims the first pass missed A systematic grep for the TTL assertion found it still in runbooks/README.md and in runbooks/kustomization.yaml, outside the page being rewritten. Same correction: the countdown on plex.tv/claim is the authority, not a number in our tree. README.md also now records that the server is CLAIMED and points at claim-ship.sh with the measured ~2m45s loop, rather than describing a push-based sequence that no longer exists. Co-Authored-By: Claude Opus 5 (1M context) --- runbooks/README.md | 8 +++++--- runbooks/kustomization.yaml | 2 +- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/runbooks/README.md b/runbooks/README.md index a2f9db8..c393ee8 100644 --- a/runbooks/README.md +++ b/runbooks/README.md @@ -110,9 +110,11 @@ Each affected runbook says so on its own page and gives the by-hand equivalent. every green signal missed. ### Operator procedures (no alert links here) -- **`plex-claim`** — claiming the Plex Media Server. The plex.tv claim token lives ~4 minutes, so this - is a timed sequence: token → `sops` → push → **forced** `flux reconcile` → pod roll. Includes why a - changed Secret does not restart anything, and why an expired token is harmless. +- **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only + applies after a `/config` wipe. The claim token is short-lived — read the countdown on + plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`: + token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured). + Includes why a changed Secret does not restart anything, and why an expired token is harmless. ### Virtualisation - **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running. diff --git a/runbooks/kustomization.yaml b/runbooks/kustomization.yaml index a0bd184..626ae5f 100644 --- a/runbooks/kustomization.yaml +++ b/runbooks/kustomization.yaml @@ -83,7 +83,7 @@ configMapGenerator: - node-memory-swap=node-memory-swap.md # Operator procedure, not an alert page (no alert links to it — the # alert-runbooks-resolvable check treats that as advisory). It is served - # because the 4-minute plex.tv claim window is not a thing to go looking + # because the plex.tv claim window is short and is not a thing to go looking # for in git: /runbooks/plex-claim has to resolve the moment it is needed. - plex-claim=plex-claim.md - pvc-capacity=pvc-capacity.md