Compare commits

..
Author SHA1 Message Date
jakman85andClaude Opus 5 e72050c690 docs: correct CI/platform documentation that was stale, wrong, or load-bearing
gate / gate (push) Successful in 1m37s
gate / gate (pull_request) Successful in 1m36s
gate / conformance (push) Has been skipped
gate / conformance (pull_request) Has been skipped
The 2026-09-29 session found the merge gate did not exist -- no branch protection, no
runner -- while the tree asserted both. This corrects the documents that carried, and in
several cases CAUSED, that belief.

CLAUDE.md gains section 2b: never infer platform state from this repository. It carries
the specific API and kubectl checks for protection, runners, CI status, workloads and
Flux revisions, and the rule that a green tick is not proof anything ran (a skipped Gitea
job reports success). This is the rule whose absence produced every other finding.

.gitea/workflows/README.md and merge-gate.yaml both said "there is no remote git host,
this file is dormant". Gitea had been live for weeks. Both were written as plans and never
revisited once the plan completed. The Transition section is now marked COMPLETE and says
what is genuinely still outstanding (CLUSTER_AVAILABLE unset, so L2/L3 never runs in CI).

DECISIONS-LOG 2026-08-31 is annotated IN PLACE. Anyone reading chronologically previously
hit "required status check; runner incluster-dind online" with nothing marking it false
until a month later. The correction now sits on the bullet itself.

ci/README.md, clusters/README.md and both conformance READMEs carry a status banner: they
speak as though no cluster exists, which makes a live L2/L3 coverage gap read as an
expected phase. That framing is why the gap went unexamined.

apps/wave-1/gitea-actions-runner-secret.sops.yaml: fixed a wrong path, replaced advice to
`sops -d` into a terminal (which CLAUDE.md forbids) with the rotation script, and recorded
that a fresh token alone does not re-register a runner holding a stale /data/.runner.

apps/wave-0/plex-claim-secret.sops.yaml: the header claimed the file ships the placeholder.
It has not since the claim -- it holds the spent token, which is inert and deliberately
left in place.

catalog/plex.yaml and the plex secret both asserted a "~4-minute TTL" as fact. It was
uncited, upstream documents no expiry, and it was the basis for a since-disproven claim
that the pipeline was too slow to fit inside it. Measured: ~2m45s.

Both .sops.yaml files were verified to still decrypt after the comment edits
(`sops -d > /dev/null`), and render==golden confirms the catalog comment does not
propagate into the rendered manifest.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 02:14:26 -05:00
50 changed files with 291 additions and 414 deletions

No files matched your search

+28 -14
View File
@@ -40,9 +40,15 @@ and ADR-0011: **CI holds no broad kubeconfig.** L2/L3 run as an in-cluster **Cha
*collects* its artifacts, using a narrow trigger credential referenced **by secret name only**
(`secrets.CONFORMANCE_TRIGGER_KUBECONFIG`) — never a plaintext secret, never a cluster-admin kubeconfig.
## Honest SKIP — never fake green (Phase 0)
## Honest SKIP — never fake green
**No cluster exists in Phase 0.** The gate is deliberately built so unavailable checks **SKIP loudly,
**NOTE (2026-09-29):** this section is written in the past's voice — "Phase 0", "no cluster
exists". **A cluster has existed for weeks** and Flux has been reconciling against it. The
skip logic below is still correct and still honest, but read it as describing a **live
coverage gap** (L2/L3 is not wired into CI) rather than an expected phase. Its being
phrased as a phase is part of why the gap went unexamined.
Originally: The gate is deliberately built so unavailable checks **SKIP loudly,
they do not pass**:
- The `conformance` job is `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}` — with no cluster it is
@@ -77,21 +83,29 @@ Both dirs are git-ignored (see [`.gitignore`](../../.gitignore)).
## Triggers & branch model (CONTRIBUTING.md)
- `pull_request` → `main`: **PRs are the enforced merge gate into `main`.**
- `push` → `feature/**`, `fix/**`, `docs/**`, `chore/**`: fast feedback on the branch. **No work on `main`.**
- `push` → `main` only. The per-branch prefix list was removed: `act_runner` runs at capacity 1,
so a branch push fires a redundant run that serialises *ahead* of the PR run for the same tree.
**No work on `main`** — and since 2026-09-29 that is enforced, not conventional: `main` is a
protected branch and a direct push is rejected by the pre-receive hook (verified, not assumed).
## Transition: generic-CI-now → Gitea-Actions-after-Gitea-is-up
## Transition: COMPLETE (2026-09-29)
Per [`CONTRIBUTING.md`](../../../home-network-k3s/CONTRIBUTING.md) and
[`docs/05` §"local bootstrap → self-hosted Gitea"](../../../home-network-k3s/docs/05-gitops-and-delivery.md):
This section used to describe a migration as pending. It has happened. Gitea is live at
`gitea.home.keithjackson.dev`, the runner `incluster-dind` is registered and online, and
`gate.yml` runs on every PR.
1. **Now (interim):** there is **no remote git host** on this network. This workflow file is dormant —
but the **identical gate runs locally today** via `make -C ci gate` (R-TEST-02). Development is
local feature-branch → `main`, the gate enforced by convention + the local runner.
2. **Gitea comes online early** (one of the first services onboarded) at `gitea.home.keithjackson.dev`.
3. **CI moves to Gitea Actions:** this workflow activates unchanged. The only operational steps are
enabling a Gitea Actions runner and setting `vars.CLUSTER_AVAILABLE=true` +
`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` once a dev/test cluster exists. PRs then become the
**platform-enforced** merge gate — the same no-work-in-main rule, now enforced by machine.
**What took a month to notice:** the migration was recorded as done on 2026-08-31 without
anyone checking the platform. In reality `main` had **no branch protection** and **zero
runners were registered** until 2026-09-29. Every "the gate enforces this" statement in
between rested on an assertion nobody verified. PR #1 — the first pull request this
repository ever had — sat `pending` for 30.6 minutes because nothing existed to run it.
So: **do not read this file, or the DECISIONS-LOG, as evidence about the live platform.**
Query the API. `infra/gitea/` holds the scripts and the measured state.
Still genuinely outstanding: `vars.CLUSTER_AVAILABLE` and
`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` are unset, so L2/L3 conformance does not run in
CI. That is a real coverage gap, not a phase.
## Configuration (set in Gitea; referenced by name only)
+8 -5
View File
@@ -11,11 +11,14 @@
# v
# merge-gate decision [R-TEST-03: touched subset green]
#
# Transition (CONTRIBUTING.md / docs/05 §"local bootstrap -> self-hosted Gitea"):
# Today there is no Gitea host, so this file is dormant. The SAME gate is runnable
# now via `make -C ci gate` (R-TEST-02). Once Gitea is self-hosted and Gitea
# Actions is enabled, this workflow becomes the ENFORCED PR merge gate into main —
# no code change required, only turning on the runner and setting CLUSTER_AVAILABLE.
# STATUS (corrected 2026-09-29): Gitea IS hosted and Gitea Actions IS enabled; the runner
# `incluster-dind` is online and gate.yml is the live PR gate. The text that used to sit here
# ("Today there is no Gitea host, so this file is dormant") was stale by weeks -- it was written
# as a plan and never revisited once the plan completed.
#
# THIS file is still dormant, but for the reason in the DORMANT block below -- its spec checkout
# and L2/L3 seams are unwired -- NOT because there is no host. Do not read a stale plan as a
# statement about the live platform; query the API instead (see infra/gitea/).
#
# CI holds NO broad kubeconfig. L2/L3 run as an in-cluster Chainsaw Job whose own
# scoped ServiceAccount holds cluster access (docs/14, R-SEC-02); CI uses only a
+29
View File
@@ -45,6 +45,35 @@ the toolbox gate yourself and quote its numbers. Never describe a PR as "green"
strength of a check you have not seen report — and note that a **skipped** Gitea job
reports as `success`, so a green tick is not by itself evidence that anything ran.
## 2b. Never infer platform state from this repository
**This repo is not evidence about the live platform.** Neither is `DECISIONS-LOG.md`.
Both describe what someone believed at the time of writing, and both have been
confidently wrong for a month at a stretch.
On 2026-09-29 the log stated the merge gate was a required status check with a runner
online. The API said `branch_protections: []` and **zero** runners. Everything downstream
of that belief — "the gate enforces this", "your push would be rejected", "CI covers it" —
was wrong, and nothing in the tree could have revealed it, because the tree was the source
of the error.
**So, before asserting that any gate, protection, runner, secret, or workload is in a
given state: query the thing itself.**
| Claim about... | Check |
|---|---|
| branch protection / required checks | `GET /api/v1/repos/{owner}/{repo}/branches/main` |
| a runner exists | `GET /api/v1/admin/actions/runners` |
| CI actually ran | the commit's `/statuses` — and read the **description**; a *skipped* Gitea job reports `success` with "Has been skipped" |
| a workload / Secret / env | `kubectl --context k3s-prod ...` (read-only is fine for agents) |
| Flux applied your commit | compare `.status.lastAppliedRevision` against the GitRepository's artifact revision |
Quote the measured field, not the prose. If you cannot measure it, say so explicitly —
"unverified" is a legitimate report, and a far better one than a confident guess that
matches the docs.
**A green tick is not proof anything ran.** Read what actually executed.
## 3. Gate before you push
./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
+7 -82
View File
@@ -1192,6 +1192,13 @@ fresh agent should read this entry + `infra/PHASE1-STATUS.md` + `docs/plans/` an
Platform-admins require TOTP MFA.
- **Gitea Actions CI is LIVE + enforcing**: `.gitea/workflows/gate.yml` runs `make -C ci gate` on push/PR to main;
required status check; runner `incluster-dind` online (ns gitea-actions).
> **[CORRECTION, 2026-09-29 — THIS BULLET WAS FALSE WHEN WRITTEN AND STAYED FALSE FOR A MONTH.]**
> Measured against the live Gitea API on 2026-09-29: `branch_protections` returned `[]` (`main`
> `protected:false`, `enable_status_check:false`, `user_can_push:true`) and **zero** Actions runners
> were registered at repo, org or site level. The gate was neither enforced nor able to run. PR #1
> — the first pull request this repo ever had — was opened on 2026-09-29 and sat `pending` for
> 30.6 minutes with nothing to execute it. Both halves of this bullet were asserted, not verified.
> See the 2026-09-29 entries for the root cause and the fix.
### What changed this session (durable)
1. **SSO cluster-wide break→fix — root cause Cilium CVE-2025-30162.** The 1.16→1.20 security fix makes the shared
@@ -4781,85 +4788,3 @@ accurate before the PR flow and are stale now. Correcting them is tracked, not d
3. Land `feat/plex-claim-sops-secret` (PR #1) once the gate can actually run.
4. Claim plex, after checking it is not already claimed.
5. Correct the 37 runbooks' `git push origin main` wording.
## 2026-09-29 — Plex is claimed; the CI/CD gate that "enforced" it never existed. **PM HANDOFF.**
Began as "inject a plex.tv claim token". Ended by discovering the delivery pipeline was fiction,
rebuilding it, and shipping the token through it. Plex is **claimed** — `PlexOnlineToken` is
present in `/config` on the `plex-config` PVC, so the exchange is now a permanent no-op and the
library scan is running over the 8.33T NFS share.
### The finding: both halves of the 2026-08-31 CI claim were false
That entry recorded "Gitea Actions CI is LIVE + enforcing … required status check; runner
`incluster-dind` online". Measured against the live API on 2026-09-29:
| Claimed | Measured |
|---|---|
| `main` protected, gate required | `branch_protections` → `[]`; `protected:false`, `enable_status_check:false`, `required_approvals:0`, `user_can_push:true` |
| Runner online | **zero** runners at repo, org AND site level |
**PR #1 was the first pull request this repository has ever had.** Its contexts sat `pending`
for 30.6 minutes across 122 polls and never reported. For a month, every "the gate enforces this"
statement rested on an assertion nobody had checked — the same fail-open class as 2026-09-18,
and the reason it hid so long is that the log itself was the evidence.
### Root cause of the dead runner
`act_runner` registers **once** and persists its identity in `/data/.runner` on the
`act-runner-data` PVC. Gitea no longer held a matching row, so the pod booted, saw `.runner`,
**skipped registration**, and polled with an identity Gitea did not recognise. Symptom: zero
runners, pod Running, nothing crashlooping, no error anyone reads. Deleting the PVC forced a
genuine re-registration (`.runner is missing` → `Runner registered successfully`).
It then did not come back for several minutes, because Flux could not recreate the PVC:
`apps-wave-1` was blocked four levels deep on a **stale** `DependencyNotReady` for
`platform-config`, which itself read `Ready=True`. Each wave re-evaluates only on its own
`interval: 10m`, so a resource deleted at the bottom of the chain can take ~40 min to return.
**Force it** with `flux reconcile kustomization apps-wave-1 --with-source` rather than waiting.
### Corrections to things this log and the runbooks asserted
- **A skipped Gitea job reports `success`, not pending.** Measured: `gate / conformance` returned
`success` / "Has been skipped". An earlier entry (#2) said requiring it would deadlock every PR.
Wrong — it would pass **vacuously**, forever, while nothing ran. Under HARD RULE 4 that is worse
than a deadlock, because a deadlock is loud. Never require it; require
`gate / gate (pull_request)` alone. Corrected in #3.
- **`gate.yml`'s push trigger omits `feat/**`** (it lists `feature/** fix/** docs/** chore/**`).
With `act_runner` capacity 1, a branch prefix that IS in that list fires a second redundant run
which serialises ahead of the PR run and doubles wall-clock. Exploited deliberately in
`infra/plex/claim-ship.sh`; the honest fix is to align the trigger list.
- **The "~4-minute TTL"** asserted in four places was **uncited**, and upstream `pms-docker`
documents no expiry at all. The authoritative reading is the countdown plex.tv/claim renders at
issue time. Removed from `runbooks/plex-claim.md`; the same claim still sits in
`catalog/plex.yaml` and the Secret's header comment (left alone — editing a `.sops.yaml`
comment risks the MAC, and touching `catalog/plex.yaml` re-renders a golden).
### Measured, once the pipeline actually worked
`gate / gate (pull_request)` 1m34s–1m58s. Full claim loop end to end **~2m45s** — comfortably
inside the window, which retires the theory that a short-lived token cannot survive a PR flow.
That theory was mine and it was wrong: the pipeline was never too slow, it simply did not exist.
### State now
- `main` **protected**, `user_can_push:false`. Runner `incluster-dind` online.
- `CLAUDE.md` is the agent delivery contract; `infra/gitea/` holds the protection + runner scripts.
- **All 37 runbooks** that instructed `git push origin main` now describe the PR flow — that
instruction had become actively blocked, not merely stale.
- `infra/plex/claim-ship.sh` drives the claim loop end to end.
### Next steps
1. **`enable_status_check: true`** with `gate / gate (pull_request)` as the only context. The
prerequisite is met: the gate has been observed green on real PRs.
2. Add `jakman85` as a repo **collaborator**, then `required_approvals: 1`. (Currently the only
org member is `gitea-admin`, which agents authenticate as — requiring approval before that is
fixed would leave nobody able to approve.)
3. Run the **push probe** against protection: `gitea-admin` is a site admin and Gitea may let it
bypass. `user_can_push:false` is the API's opinion, not proof.
4. Align `gate.yml`'s push trigger with this repo's `feat/**` prefix.
5. Convert the `dind` container to a **native sidecar** with a `startupProbe`, per the standing
FOLLOW-UP note in `apps/wave-1/gitea-actions.yaml`. It deletes both hand-rolled wait loops and
the envsubst hazard. Land it ALONE — it is the merge gate for the whole repo.
6. Create the Plex libraries over `/media/movies`, `/media/tv`, `/media/photos`.
+14 -6
View File
@@ -7,12 +7,20 @@
# substitution). A braced token quoted from the upstream shell script would be a live hazard here,
# not prose, so the upstream conditions below are described in words.
#
# THE VALUE SHIPPED WITH THIS FILE IS A PLACEHOLDER, DELIBERATELY — the literal string
# `replace-me-with-a-fresh-plex-tv-claim-token`, not a credential. A https://plex.tv/claim token has
# a ~4-minute TTL, far shorter than commit -> push -> Flux reconcile -> pod-roll, so no token
# committed here is ever still valid when the pod starts. This file exists to make the Secret — and
# therefore the env reference — ALWAYS RESOLVABLE IN THE FLUX GRAPH, not to carry a working token.
# The real token is injected by the operator inside that 4-minute window: runbooks/plex-claim.md.
# THIS FILE NO LONGER HOLDS THE PLACEHOLDER. Corrected 2026-09-29: the server was CLAIMED on that
# date, and the value here is the SPENT token from that claim. It is inert and safe to leave --
# upstream gates the exchange on PlexOnlineToken being ABSENT from /config, and it is now present on
# the plex-config PVC, so this value is a NO-OP on every start whatever it holds. Deleting the Secret
# would NOT be safe (see below), which is why a spent token stays here rather than being blanked.
#
# The claim token is SHORT-LIVED -- minutes. Read the countdown that plex.tv/claim renders when it
# issues one; do NOT budget against a number written in this tree. Earlier revisions of this comment
# asserted a "~4-minute TTL" as established fact with no citation, and upstream plexinc/pms-docker
# documents no expiry at all. That uncited number was load-bearing for the whole procedure design.
#
# This file exists to make the Secret — and therefore the env reference — ALWAYS RESOLVABLE IN THE
# FLUX GRAPH, not to carry a working token. Re-claiming (only needed after a /config wipe) is
# scripted: infra/plex/claim-ship.sh, measured at ~2m45s end to end.
#
# WHY SHIPPING A PLACEHOLDER IS SAFE (verified against plexinc/pms-docker branch `master`, the
# s6-overlay v2 script root/etc/cont-init.d/40-plex-first-run, cross-checked byte-identical against
@@ -4,7 +4,13 @@
# against Gitea. Flux decrypts it via the sops-age key at reconcile (apps Kustomization
# decryption is wired in clusters/dev/kustomization.yaml). Rotate by resetting the token
# in Gitea (admin -> actions -> runners) and re-encrypting a fresh value here.
# Recover/inspect: infra/devcontainer/dev.sh run sops -d apps/gitea-actions-runner-secret.sops.yaml
# ROTATE: ./infra/gitea/runner-token-rotate.sh -- it mints a fresh instance-level token from the
# Gitea API and writes it here via `sops set`, without the value ever being printed.
# Do NOT `sops -d` this file into a terminal, a log, or an agent transcript. To check only that
# it still decrypts, redirect: `sops -d <file> > /dev/null`.
# NOTE: a fresh token alone may not re-register the runner. act_runner registers ONCE and keeps
# its identity in /data/.runner on the act-runner-data PVC; if Gitea no longer holds a matching
# row it skips registration silently. Clearing that PVC is what forces a real re-registration.
apiVersion: v1
kind: Secret
metadata:
+11 -5
View File
@@ -38,11 +38,17 @@ env:
# UNCLAIMED, so this is wired; it was previously left out on the reasoning below, and only one
# half of that reasoning survived contact with the upstream source.
#
# THE 4-MINUTE TTL IS STILL TRUE, and it is now the OPERATIONAL constraint rather than a reason to
# abstain. A https://plex.tv/claim token expires in ~4 minutes — far shorter than commit -> push ->
# Flux reconcile -> pod-roll if that cycle is left to its own timers. So the committed value is a
# documented PLACEHOLDER and the live token is injected by the operator inside that window, with an
# explicit `flux reconcile` to collapse the wait: runbooks/plex-claim.md is the timed procedure.
# A SHORT TTL IS THE OPERATIONAL CONSTRAINT, not a reason to abstain. A https://plex.tv/claim token
# expires in MINUTES — read the countdown the claim page renders; do not budget against a number
# written in this tree. (This comment previously asserted "~4 minutes" as established fact. It was
# UNCITED, upstream plexinc/pms-docker documents no expiry at all, and that number was load-bearing
# for the whole procedure design — including a since-disproven claim that the delivery pipeline was
# too slow to fit inside it. Measured 2026-09-29: the full loop is ~2m45s and fits.)
#
# The live token is injected by the operator inside that window, with an explicit `flux reconcile`
# to collapse the wait. Scripted end to end as infra/plex/claim-ship.sh; runbooks/plex-claim.md is
# the procedure. SERVER CLAIMED 2026-09-29 — this env is now a permanent no-op (see below), and the
# committed value is the spent token from that claim rather than the original placeholder.
#
# ONE-SHOT, AND SAFE TO SIT HERE EXPIRED. Verified from plexinc/pms-docker's own first-run script
# (branch `master`, root/etc/cont-init.d/40-plex-first-run, confirmed byte-identical to the file in
+14
View File
@@ -1,5 +1,19 @@
# `ci/` — portable merge gate (runnable today, without Gitea)
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
> This document speaks as though no cluster exists and the platform is scaffolding. **A
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
> coverage hole, not a phase.
>
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
> documentation rather than measurement, and `main` was unprotected with zero runners
> registered the whole time. **Query the live API before asserting platform state.**
**Task T-0.2 · R-TEST-02 (reproducible, single-command) · Owner WS-GITOPS (CI).**
The same merge gate the Gitea Actions workflow runs
+14
View File
@@ -1,5 +1,19 @@
# clusters/
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
> This document speaks as though no cluster exists and the platform is scaffolding. **A
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
> coverage hole, not a phase.
>
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
> documentation rather than measurement, and `main` was unprotected with zero runners
> registered the whole time. **Query the live API before asserting platform state.**
Flux entrypoints and the environment-overlay model (**R-ENV-02**, **R-GITOPS-01/02/03**).
Governing spec: [`docs/05-gitops-and-delivery.md`](../../home-network-k3s/docs/05-gitops-and-delivery.md),
+14
View File
@@ -1,5 +1,19 @@
# conformance/ — the home-k3s conformance harness (WS-TEST)
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
> This document speaks as though no cluster exists and the platform is scaffolding. **A
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
> coverage hole, not a phase.
>
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
> documentation rather than measurement, and `main` was unprotected with zero runners
> registered the whole time. **Query the live API before asserting platform state.**
This subtree is the **definition of "done"** for the whole platform. Nothing is accepted
on a demo, a screenshot, or "it reconciled": every requirement is proven by an
**automated, reproducible, behavior-based test** tagged with the requirement ID(s) it
+14
View File
@@ -1,5 +1,19 @@
# In-cluster conformance Job (L2/L3)
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
> This document speaks as though no cluster exists and the platform is scaffolding. **A
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
> coverage hole, not a phase.
>
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
> documentation rather than measurement, and `main` was unprotected with zero runners
> registered the whole time. **Query the live API before asserting platform state.**
Per **ADR-0011** and **docs/14 "Where each layer runs"**: L0/L1 run in CI without a
cluster, but **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job**
on the dev/test cluster that **CI triggers and waits for**. CI holds **no** cluster
-93
View File
@@ -1,93 +0,0 @@
#!/usr/bin/env bash
# plex-claim-ship.sh — run this IMMEDIATELY AFTER editing the SOPS secret with a fresh
# plex.tv claim token. It drives the whole CI/CD loop end to end and prints a running
# stopwatch so you can see the budget being spent.
#
# branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verify
#
# It never sees, prints, or handles the token: the file is already encrypted by sops.
set -euo pipefail
HOST=gitea.home.keithjackson.dev
OWNER=Home-Network
REPO=home-network-k3s-impl
SECRET=apps/wave-0/plex-claim-secret.sops.yaml
T0=$(date +%s)
el() { printf '\n[T+%02d:%02d] %s\n' $(( ($(date +%s)-T0)/60 )) $(( ($(date +%s)-T0)%60 )) "$1"; }
cd "$(git rev-parse --show-toplevel)"
# --- preflight: refuse to ship a file that is not what we think it is -------------
git diff --quiet -- "$SECRET" && { echo "ERROR: $SECRET is unchanged. Edit it first:"; \
echo " infra/devcontainer/dev.sh run sops $SECRET"; exit 1; }
grep -q 'PLEX_CLAIM: ENC\[' "$SECRET" || { echo "ERROR: $SECRET is not encrypted. REFUSING."; exit 1; }
[ -z "$(git status --porcelain | grep -v " $SECRET\$")" ] || echo "NOTE: other files are dirty; only $SECRET will be committed."
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
# feat/ prefix ON PURPOSE. act_runner capacity is 1, and gate.yml's PUSH trigger covers
# feature/** fix/** docs/** chore/** but NOT feat/** — so a chore/ branch would fire a
# second, redundant run that SERIALISES ahead of the PR run and doubles the wall clock.
BRANCH="feat/plex-claim-token-$(date +%H%M%S)"
el "committing on $BRANCH"
git checkout -q -b "$BRANCH"
git add "$SECRET"
git commit -q -m 'chore(plex): fresh plex.tv claim token
Injected inside the plex.tv claim window per runbooks/plex-claim.md.'
SHA=$(git rev-parse HEAD)
el "pushing"
git push -q -u origin "$BRANCH"
el "opening PR"
PR=$(jq -n --arg h "$BRANCH" --arg b main --arg t 'chore(plex): fresh plex.tv claim token' \
'{head:$h,base:$b,title:$t,body:"Token injection inside the claim window. Wiring already reviewed in #1."}' \
| api -X POST -H 'Content-Type: application/json' --data @- \
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls" | jq -r '.number')
[ -n "$PR" ] && [ "$PR" != "null" ] || { echo "PR creation failed"; exit 1; }
echo " PR #$PR"
el "waiting for the gate (expect ~1m40s)"
for i in $(seq 1 60); do
st=$(api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/commits/${SHA}/status" | jq -r '.state')
printf '\r gate=%s [T+%ss] ' "$st" "$(( $(date +%s)-T0 ))"
case "$st" in success) echo; break;; failure|error) echo; echo "GATE FAILED — stopping."; exit 1;; esac
sleep 5
done
[ "$st" = "success" ] || { echo; echo "gate did not finish in time"; exit 1; }
el "merging PR #$PR"
api -X POST -H 'Content-Type: application/json' -d '{"Do":"merge"}' \
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls/${PR}/merge" >/dev/null
git checkout -q main && git pull -q --ff-only origin main
el "forcing the Flux reconcile (--with-source, or it applies the OLD revision)"
./infra/devcontainer/dev.sh run bash -c \
'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-0 --with-source'
# MANDATORY here. Only the Secret ciphertext changed, so the Deployment pod template is
# byte-identical and NOTHING restarts on its own. PLEX_CLAIM is read once, at container
# start. Skipping this is failure mode #2 in runbooks/plex-claim.md.
el "rolling the pod (mandatory — a changed Secret restarts nothing)"
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex rollout restart deployment/plex && \
kubectl --context k3s-prod -n plex rollout status deployment/plex --timeout=120s'
el "verdict"
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex logs deploy/plex --since=3m | grep -i -e claim -e token || true'
cat <<'NOTE'
"Attempting to obtain server token..." + "Token obtained successfully" -> CLAIMED
"Attempting..." with no success line -> token expired; retry, nothing is broken
neither line -> already claimed, or env did not reach the container
Confirm it persisted (prints the attribute NAME only, never its value):
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c \
"grep -o PlexOnlineToken= \"/config/Library/Application Support/Plex Media Server/Preferences.xml\" | head -1"'
NOTE
+4 -7
View File
@@ -33,8 +33,7 @@ negative is how a CLI quirk becomes real data loss. Write `backups.velero.io`,
`backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`,
`volumes.longhorn.io`, `nodes.longhorn.io`.
Merging a PR into `main` **is** the deploy; there is no other apply step. `main` is protected, so a
direct push is rejected — durable changes go through a branch, a PR, and a green gate.
`git push origin main` **is** the deploy; there is no other apply step. Durable changes go through git.
## Alerts you should not trust the silence of (measured 2026-09-13)
@@ -110,11 +109,9 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
every green signal missed.
### Operator procedures (no alert links here)
- **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only
applies after a `/config` wipe. The claim token is short-lived — read the countdown on
plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
Includes why a changed Secret does not restart anything, and why an expired token is harmless.
- **`plex-claim`** — claiming the Plex Media Server. The plex.tv claim token lives ~4 minutes, so this
is a timed sequence: token → `sops` → push → **forced** `flux reconcile` → pod roll. Includes why a
changed Secret does not restart anything, and why an expired token is harmless.
### Virtualisation
- **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.
+2 -4
View File
@@ -62,10 +62,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -61,10 +61,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -78,10 +78,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -55,10 +55,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -62,10 +62,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -63,10 +63,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -87,10 +87,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -69,10 +69,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -71,10 +71,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -68,10 +68,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -61,10 +61,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -62,10 +62,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+3 -5
View File
@@ -7,7 +7,7 @@
## What this means here
Something in the GitOps chain is stuck at Ready=False. On this cluster that is more consequential than
it sounds, because **merging a PR into `main` IS the deploy** — there is no other apply step. A stuck
it sounds, because **`git push origin main` IS the deploy** — there is no other apply step. A stuck
Kustomization means git and the cluster have diverged and nothing will close the gap.
The chain has **14 tiers**: `flux-system` → `cilium` → `platform` → `platform-config` →
@@ -61,10 +61,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -58,10 +58,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -53,10 +53,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -53,10 +53,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -87,10 +87,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -67,10 +67,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -59,10 +59,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -57,10 +57,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -53,10 +53,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -62,10 +62,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+1 -1
View File
@@ -83,7 +83,7 @@ configMapGenerator:
- node-memory-swap=node-memory-swap.md
# Operator procedure, not an alert page (no alert links to it — the
# alert-runbooks-resolvable check treats that as advisory). It is served
# because the plex.tv claim window is short and is not a thing to go looking
# because the 4-minute plex.tv claim window is not a thing to go looking
# for in git: /runbooks/plex-claim has to resolve the moment it is needed.
- plex-claim=plex-claim.md
- pvc-capacity=pvc-capacity.md
+2 -4
View File
@@ -83,10 +83,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -74,10 +74,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -75,10 +75,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -52,10 +52,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -49,10 +49,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -56,10 +56,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -56,10 +56,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -227,10 +227,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -52,10 +52,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+55 -59
View File
@@ -1,4 +1,4 @@
# plex-claim — claiming the Plex Media Server inside the token window
# plex-claim — claiming the Plex Media Server inside the 4-minute token window
**Alert:** none. This is an **operator procedure**, not an incident page — it is indexed here because
it is the page you need at the moment you need it, with a stopwatch running.
@@ -14,11 +14,10 @@ server to a plex.tv account. It is a **one-shot** operation: the token is exchan
the `PLEX_CLAIM` env is **ignored on every start** (upstream gate:
`[ ! -z "${PLEX_CLAIM}" ] && [ -z "${token}" ]` in `root/etc/cont-init.d/40-plex-first-run`).
**A `https://plex.tv/claim` token is short-lived — minutes, not hours.** Read the countdown the
claim page shows you; do not budget against a number written here. That is shorter than PR → gate →
merge → Flux's 10-minute reconcile interval → pod roll if you let those run on their own timers,
which is why this page exists: the loop is scripted and the reconcile is **forced** rather than
waited for. Measured end to end on 2026-09-29: **~2m45s**.
**A `https://plex.tv/claim` token lives ~4 minutes.** That is shorter than commit → push → Flux's
10-minute reconcile interval → pod roll if you let those run on their own timers, which is why this
page exists: every step below is written to be run back-to-back, with the reconcile **forced** rather
than waited for. Budget: token in hand at T+0, pod restarted by roughly T+2:00.
**An expired or wrong token does not break anything.** Verified against the image's own first-run
script (and the copy inside `plexinc/pms-docker:latest`): a failed exchange gets a 403 with an empty
@@ -67,84 +66,81 @@ infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0'
```
## The procedure
## The 4-minute procedure
> **STATUS: this server was CLAIMED on 2026-09-29** and `PlexOnlineToken` is present in
> `/config` on the `plex-config` PVC. The upstream exchange is gated on that pref being
> *absent*, so `PLEX_CLAIM` is now a **no-op on every start** whatever it holds. You only need
> this page again after a `/config` wipe or a rebuild onto an empty PVC.
**On the clock.** The token is short-lived — **read the countdown that plex.tv/claim shows you**
rather than trusting a number written here. (Earlier revisions of this page asserted a "~4-minute
TTL" in four places with no citation; upstream `pms-docker` documents no expiry at all. The page
is the only authority, and it is on screen at T+0.)
**A burnt token costs nothing.** An expired token gets a 403 with an empty body, `curl`'s status is
never checked, the `if` does not fire, there is no `else` and no `exit 1`, and the script (no
`set -e`) exits 0. The pod comes up unclaimed and serving. So retry freely.
### One command
Have this page open and do not stop between steps.
1. **T+0 — get the token.** Sign in to plex.tv as the account that should OWN this server, open
<https://plex.tv/claim>, copy the `claim-…` string.
<https://plex.tv/claim>, copy the `claim-…` string. The clock starts now.
2. **Put it in the Secret** — in place, encrypted, never touching a plaintext file:
2. **T+0:15 — put it in the Secret**, in place, encrypted, without it ever touching a plaintext file:
```
infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
```
3. **Ship it.** This drives the whole delivery loop and prints a running stopwatch:
Replace the `PLEX_CLAIM:` value (the placeholder `replace-me-with-a-fresh-plex-tv-claim-token`, or
a previous expired token) and save. `sops` re-encrypts on write to the committed age recipient.
3. **T+0:45 — commit and push.** `git push origin main` **is** the deploy; there is no other apply
step.
```
./infra/plex/claim-ship.sh
git add apps/wave-0/plex-claim-secret.sops.yaml
git commit -m 'chore(plex): fresh plex.tv claim token'
git push origin main
```
branch → commit → push → PR → gate → merge → `flux reconcile --with-source` → `rollout restart`
→ verdict. It refuses to run if the file is unchanged or not encrypted.
4. **T+1:00 — force the reconcile immediately. This is the one command that saves the window** — it
pulls the new commit and re-applies wave 0 instead of waiting up to 10 minutes for the interval:
**Measured end to end, 2026-09-29: ~2m45s.** Gate ~1m40s, merge + reconcile ~25s, pod roll and
container start ~30s.
```
infra/devcontainer/dev.sh run bash -c \
'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-0 --with-source'
```
### Why it is a script and not six pasted commands
`--with-source` is the load-bearing flag: it reconciles the `flux-system` GitRepository first, so
the Kustomization applies **your commit** and not the previously-fetched revision.
`main` is a **protected branch** — the direct push this page used to instruct is now rejected — so
the deploy is a PR merged after a green gate. Three of those steps are easy to get wrong under time
pressure, and two are silent when skipped:
5. **T+1:30 — roll the pod.** This step is **mandatory and easy to forget**: a changed Secret does
**not** restart anything. `PLEX_CLAIM` is injected by `secretKeyRef` and read only at container
start, and the Deployment's pod template is byte-identical across this change, so Flux applies the
new Secret and the running pod keeps the old value forever.
- **`--with-source` is load-bearing.** Without it the Kustomization applies the previously-fetched
revision, not your commit.
- **The pod roll is mandatory.** Only the Secret's ciphertext changes, so the Deployment's pod
template is byte-identical and *nothing restarts on its own*. `PLEX_CLAIM` is read once, at
container start. Skip this and the new Secret sits in the cluster while the running pod keeps the
old value forever.
- **The branch is named `feat/…` on purpose.** `act_runner` capacity is 1, and `gate.yml`'s push
trigger covers `feature/** fix/** docs/** chore/**` but *not* `feat/**`. A `chore/` branch fires a
second redundant run that serialises ahead of the PR run and roughly doubles the wall clock.
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex rollout restart deployment/plex'
```
### Reading the verdict
6. **T+2:00 — confirm the exchange happened**, in the new pod's init log:
* `Attempting to obtain server token from claim token` **and** `Token obtained successfully`
⇒ **claimed.** Re-run the First-commands `PlexOnlineToken=` check to confirm it persisted, then
create the libraries over `/media/movies`, `/media/tv`, `/media/photos`.
* `Attempting…` with **no** success line ⇒ the token was expired or invalid. Nothing is broken; go
back to step 1 with a new one.
* **Neither line** ⇒ the exchange was skipped: `/config` already had a `PlexOnlineToken` (already
claimed — the expected case now), or the env did not reach the container. Check by NAME only:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex logs deploy/plex --since=5m | grep -i -e claim -e token'
```
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
-o jsonpath="{.items[0].spec.containers[0].env[*].name}"'
```
* `Attempting to obtain server token from claim token` **and** `Token obtained successfully`
⇒ **claimed.** Re-run the First-commands `PlexOnlineToken=` check to confirm it persisted, then
create the libraries over `/media/movies`, `/media/tv`, `/media/photos`.
* `Attempting to obtain server token from claim token` with **no** success line ⇒ the token was
expired or invalid. Nothing is broken; go back to step 1 with a new token.
* **Neither line** ⇒ the exchange was skipped, which means `/config` already had a
`PlexOnlineToken` (already claimed) or the env did not reach the container. Check the pod actually
has the variable, by NAME only:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
-o jsonpath="{.items[0].spec.containers[0].env[*].name}"'
```
## Likely causes, most likely first
When the claim does not take, in order of how often it is the answer here:
1. **The token expired** — you exceeded the countdown, usually by reviewing the PR instead of
merging it the moment the gate went green. Symptom: the "Attempting…" line with no success line.
Remedy: a new token, and let `claim-ship.sh` drive the loop rather than pasting steps.
1. **The token expired** — you spent more than ~4 minutes, usually in step 3 or waiting on step 4.
Symptom: the "Attempting…" line with no success line. Remedy: a new token, and force the reconcile
rather than waiting for it.
2. **The pod was never restarted** (step 5 skipped). Symptom: no claim log lines at all from a pod
whose `AGE` predates your push. The Secret is correct in git and in the cluster and the running
container still holds the placeholder.
+2 -4
View File
@@ -58,10 +58,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -60,10 +60,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -54,10 +54,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
+2 -4
View File
@@ -66,10 +66,8 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands