Compare commits
12
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
761e25cc3f | ||
|
|
8e017fad91 | ||
|
|
27d0ef3324 | ||
|
|
bbd4f682a6 | ||
|
|
0391139240 | ||
|
|
e9f441b5ce | ||
|
|
234cc6a71a | ||
|
|
69a0d1a1c7 | ||
|
|
d3a0d587a4 | ||
|
|
fe862ab527 | ||
|
|
a2665c0511 | ||
|
|
5a9e455149 |
No files matched your search
@@ -0,0 +1,117 @@
|
||||
# CLAUDE.md — how work lands in this repo
|
||||
|
||||
Read this before doing anything durable. It is the delivery contract for AI coding
|
||||
sessions and agents. `DECISIONS-LOG.md`'s last entry is the current PM handoff and
|
||||
takes precedence on *what* to work on; this file governs *how* it lands.
|
||||
|
||||
## 1. Never push to `main`. Ever.
|
||||
|
||||
Every durable change reaches `main` exactly one way:
|
||||
|
||||
branch -> push the branch -> open a PR -> CI gate green -> human approval -> merge
|
||||
|
||||
There is no other path. Specifically:
|
||||
|
||||
- **Do not `git push origin main`.**
|
||||
- **Do not `git push origin HEAD:main`.** Same thing wearing a hat.
|
||||
- **Do not merge your own PR.** A human approves and clicks merge.
|
||||
- **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
|
||||
anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
|
||||
|
||||
> **Many runbooks under `runbooks/` still say `git push origin main` IS the deploy.**
|
||||
> That wording is **stale** — it predates the PR flow. It describes what Flux does
|
||||
> (main is what the cluster follows), not what you are permitted to do. Honour this
|
||||
> file, not that line. Fixing those runbooks is tracked work.
|
||||
|
||||
## 2. Enforcement is currently WEAKER than this rule. Follow the rule anyway.
|
||||
|
||||
Verified against the live Gitea on 2026-09-29:
|
||||
|
||||
| Expected | Actual |
|
||||
|---|---|
|
||||
| `main` protected, direct push rejected | **No branch protection.** `protected: false`, `user_can_push: true` |
|
||||
| PR blocked until the gate passes | **`enable_status_check: false`**, `status_check_contexts: []` |
|
||||
| Gate runs on every PR | **Fixed 2026-09-29.** Runner `incluster-dind` is online; `gate / gate (pull_request)` runs and passes in ~1m30s–2m |
|
||||
| Approval required before merge | **`required_approvals: 0`** |
|
||||
|
||||
So nothing mechanically stops you from pushing to `main` right now. **That is a gap
|
||||
being closed, not permission.** `DECISIONS-LOG.md` (2026-08-31) claimed the gate was a
|
||||
"required status check" with a runner online; neither was true, which is how this went
|
||||
unnoticed for a month. The runner half is now genuinely fixed and observed green. The
|
||||
protection half is not — see `infra/gitea/`.
|
||||
|
||||
**Consequence for you:** CI now runs, but it does not yet *block* anything. Always run
|
||||
the toolbox gate yourself and quote its numbers. Never describe a PR as "green" on the
|
||||
strength of a check you have not seen report — and note that a **skipped** Gitea job
|
||||
reports as `success`, so a green tick is not by itself evidence that anything ran.
|
||||
|
||||
## 3. Gate before you push
|
||||
|
||||
./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
|
||||
|
||||
Exit codes are a contract — do not flatten them:
|
||||
|
||||
| rc | Meaning |
|
||||
|---|---|
|
||||
| 0 | PASS |
|
||||
| 1 | BLOCKED — something failed |
|
||||
| 3 | UNDETERMINED — a stage did not run. **Never report this as green.** |
|
||||
|
||||
`spec-immutability` reaches rc 3 whenever the sibling spec repo is not checked out
|
||||
beside this one; that single skip is expected and tolerated. Any *other* skip is a
|
||||
real gap in your run.
|
||||
|
||||
**Never gate on the host.** `promtool` and `kustomize` are not on the host `$PATH`,
|
||||
so checks SKIP and you get a false rc 3. Only the pinned toolbox result counts.
|
||||
|
||||
### Running the gate from a git worktree
|
||||
|
||||
Two unrelated failures, both with known workarounds — put both in any agent brief:
|
||||
|
||||
1. **`dev.sh` refuses to start:** `infra/ansible/.cache/prod.kubeconfig` is untracked
|
||||
and exists only in the main checkout. Do **not** copy it (it is a credential).
|
||||
Use `DEV_KUBECONFIG=/dev/null` — the gate contacts no cluster, so L2/L3 scaffold-skip
|
||||
normally.
|
||||
2. **Every `git` call inside the container fails:** a worktree's `.git` is a pointer to a
|
||||
gitdir outside the mounted root. `dev.sh` takes no extra-mount flag, so inject one via
|
||||
the `ENGINE` hook — a shim that intercepts `run` and re-execs
|
||||
`podman run -v /path/to/main/.git:/path/to/main/.git:ro "$@"`. A traceback in
|
||||
`tracked_files()` is this bug, not a broken tree.
|
||||
|
||||
## 4. Branch naming
|
||||
|
||||
Use `feat/`, `fix/`, `docs/`, `chore/`. Note that `.gitea/workflows/gate.yml`'s **push**
|
||||
trigger lists `feature/**` — not `feat/**` — so branch pushes get no fast-feedback run.
|
||||
The `pull_request` trigger fires regardless of branch name, so the PR gate is unaffected.
|
||||
|
||||
## 5. Cluster access
|
||||
|
||||
`kubectl` does not work on the operator workstation; the toolchain lives only in the
|
||||
pinned container. Always name the context explicitly:
|
||||
|
||||
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
||||
|
||||
`dev.sh` merges both clusters and picks a current-context **per invocation**; that
|
||||
default has already changed once. Never rely on it. Do not run two `dev.sh`
|
||||
invocations concurrently.
|
||||
|
||||
**Mutating `kubectl` and `flux reconcile` are operator actions, not agent actions.**
|
||||
|
||||
## 6. Secrets
|
||||
|
||||
- **Never `sops -d` a file into a terminal, a log, or a transcript.**
|
||||
- **Never print a Secret's `.data`/`.stringData`** when checking whether it exists —
|
||||
list key names or object names only. A password has been leaked into a transcript
|
||||
this way before.
|
||||
- Encrypted files are `*.sops.yaml` or live under `secrets/`. The root `.sops.yaml` is
|
||||
the single creation-rules file; do not add a second one in a subtree.
|
||||
- A short-lived credential (e.g. a plex.tv claim token, ~minutes) cannot survive the PR
|
||||
flow. Do not design around committing one — read the expiry from the issuing page and
|
||||
treat a burnt token as free to retry.
|
||||
|
||||
## 7. Before you finish
|
||||
|
||||
Append a handoff entry to `DECISIONS-LOG.md` describing what you changed, what you
|
||||
verified, and what you did **not** verify. State gaps plainly — an honest
|
||||
"UNVERIFIED" is worth more than an optimistic "green", and this repo has already been
|
||||
burned by the difference.
|
||||
@@ -4716,3 +4716,150 @@ asleep. `git worktree remove` preserves branches and their commits, so only that
|
||||
6. **Five `public` services have no working public path** — waiting on WAN forwarding, deliberately.
|
||||
7. Backlog cherry-picks: `feat/egress-gateway-staleness-alert`, two `rescued/*` fixes, two rendered
|
||||
manifests nothing applies (the keycloak one would stand up a conflicting instance).
|
||||
|
||||
## 2026-09-29 — The CI/CD merge gate does not exist: no runner, no branch protection. **PM HANDOFF.**
|
||||
|
||||
Went to land the plex claim wiring through the documented PR flow and discovered the flow has
|
||||
never run. **PR #1 is the first pull request this repository has ever had.** It is still queued.
|
||||
|
||||
### What is actually true of the live Gitea (verified via API, 2026-09-29)
|
||||
|
||||
| Claimed (DECISIONS-LOG 2026-08-31) | Measured |
|
||||
|---|---|
|
||||
| "required status check" on `main` | `branch_protections` returns `[]`. `protected: false`, `enable_status_check: false`, `status_check_contexts: []`, `required_approvals: 0`, `user_can_push: true` |
|
||||
| "runner `incluster-dind` online (ns gitea-actions)" | **Zero runners registered** at repo, org and site level |
|
||||
|
||||
Both halves of that entry are false. Nothing enforces the gate, and the gate cannot run.
|
||||
PR #1's two contexts (`gate / gate`, `gate / conformance`) sat `pending` for 14+ minutes with
|
||||
nothing to execute them, and will sit there indefinitely.
|
||||
|
||||
This is a **fail-open governance hole**, the same class as the two found on 2026-09-18 — and it
|
||||
was invisible precisely because the log asserted the opposite. Every "the gate is enforcing"
|
||||
statement since 2026-08-31 rested on an unverified claim.
|
||||
|
||||
### Ordering trap — read before fixing
|
||||
|
||||
**Do not enable required status checks until a runner is live.** Requiring
|
||||
`gate / gate (pull_request)` with no runner deadlocks every PR permanently, including the PR
|
||||
that would fix it. Sequence: runner first, verify the gate actually runs and passes on a real
|
||||
PR, *then* turn on enforcement with the context name CI actually reported.
|
||||
|
||||
Related: `gate.yml`'s `conformance` job is `if: false` by design (honest skip, never fake green),
|
||||
but it still publishes a `gate / conformance (pull_request)` context. If that context is ever
|
||||
listed as required, it deadlocks every PR for the same reason. Wire it or exclude it.
|
||||
|
||||
### Delivery contract written down
|
||||
|
||||
Added `CLAUDE.md` at the repo root — the first agent-facing delivery doc here. It states the
|
||||
branch -> PR -> gate -> approval -> merge rule, records that enforcement is currently weaker than
|
||||
the rule (so agents follow the rule rather than what they can get away with), and carries the
|
||||
toolbox-gate invocation, the two git-worktree workarounds, the rc 0/1/3 contract, and the secret
|
||||
handling rules.
|
||||
|
||||
It also flags that **37 of 39 runbooks still instruct `git push origin main`**. Those were
|
||||
accurate before the PR flow and are stale now. Correcting them is tracked, not done.
|
||||
|
||||
### Also found, not fixed
|
||||
|
||||
- `gate.yml`'s push trigger lists `feature/**`; this repo's branches use `feat/**`, so branch
|
||||
pushes get no fast-feedback run. PR trigger is unaffected.
|
||||
- The gitea-actions runner is GitOps-managed (`apps/wave-1/gitea-actions.yaml` +
|
||||
`gitea-actions-runner-secret.sops.yaml`), so it is deployed but not registering — most likely a
|
||||
stale registration token in that SOPS secret. Needs in-cluster diagnosis (operator).
|
||||
- **Plex is still unclaimed and it was never checked** whether it is already claimed. Do that
|
||||
before spending a token — `runbooks/plex-claim.md` §First commands.
|
||||
- `runbooks/plex-claim.md` asserts a "~4-minute TTL" for a plex.tv claim token in four places,
|
||||
**uncited**, and the whole stopwatch procedure is built on it. Upstream `pms-docker`'s README
|
||||
says nothing about expiry. The number is plausible community folklore, not a measured fact.
|
||||
The authoritative reading is the countdown rendered on plex.tv/claim at issue time.
|
||||
|
||||
### Next steps
|
||||
|
||||
1. **Operator:** diagnose the gitea-actions runner; get one registered.
|
||||
2. **Operator:** enable branch protection on `main` — push blocked, approvals required — and only
|
||||
then add the gate as a required status check.
|
||||
3. Land `feat/plex-claim-sops-secret` (PR #1) once the gate can actually run.
|
||||
4. Claim plex, after checking it is not already claimed.
|
||||
5. Correct the 37 runbooks' `git push origin main` wording.
|
||||
|
||||
## 2026-09-29 — Plex is claimed; the CI/CD gate that "enforced" it never existed. **PM HANDOFF.**
|
||||
|
||||
Began as "inject a plex.tv claim token". Ended by discovering the delivery pipeline was fiction,
|
||||
rebuilding it, and shipping the token through it. Plex is **claimed** — `PlexOnlineToken` is
|
||||
present in `/config` on the `plex-config` PVC, so the exchange is now a permanent no-op and the
|
||||
library scan is running over the 8.33T NFS share.
|
||||
|
||||
### The finding: both halves of the 2026-08-31 CI claim were false
|
||||
|
||||
That entry recorded "Gitea Actions CI is LIVE + enforcing … required status check; runner
|
||||
`incluster-dind` online". Measured against the live API on 2026-09-29:
|
||||
|
||||
| Claimed | Measured |
|
||||
|---|---|
|
||||
| `main` protected, gate required | `branch_protections` → `[]`; `protected:false`, `enable_status_check:false`, `required_approvals:0`, `user_can_push:true` |
|
||||
| Runner online | **zero** runners at repo, org AND site level |
|
||||
|
||||
**PR #1 was the first pull request this repository has ever had.** Its contexts sat `pending`
|
||||
for 30.6 minutes across 122 polls and never reported. For a month, every "the gate enforces this"
|
||||
statement rested on an assertion nobody had checked — the same fail-open class as 2026-09-18,
|
||||
and the reason it hid so long is that the log itself was the evidence.
|
||||
|
||||
### Root cause of the dead runner
|
||||
|
||||
`act_runner` registers **once** and persists its identity in `/data/.runner` on the
|
||||
`act-runner-data` PVC. Gitea no longer held a matching row, so the pod booted, saw `.runner`,
|
||||
**skipped registration**, and polled with an identity Gitea did not recognise. Symptom: zero
|
||||
runners, pod Running, nothing crashlooping, no error anyone reads. Deleting the PVC forced a
|
||||
genuine re-registration (`.runner is missing` → `Runner registered successfully`).
|
||||
|
||||
It then did not come back for several minutes, because Flux could not recreate the PVC:
|
||||
`apps-wave-1` was blocked four levels deep on a **stale** `DependencyNotReady` for
|
||||
`platform-config`, which itself read `Ready=True`. Each wave re-evaluates only on its own
|
||||
`interval: 10m`, so a resource deleted at the bottom of the chain can take ~40 min to return.
|
||||
**Force it** with `flux reconcile kustomization apps-wave-1 --with-source` rather than waiting.
|
||||
|
||||
### Corrections to things this log and the runbooks asserted
|
||||
|
||||
- **A skipped Gitea job reports `success`, not pending.** Measured: `gate / conformance` returned
|
||||
`success` / "Has been skipped". An earlier entry (#2) said requiring it would deadlock every PR.
|
||||
Wrong — it would pass **vacuously**, forever, while nothing ran. Under HARD RULE 4 that is worse
|
||||
than a deadlock, because a deadlock is loud. Never require it; require
|
||||
`gate / gate (pull_request)` alone. Corrected in #3.
|
||||
- **`gate.yml`'s push trigger omits `feat/**`** (it lists `feature/** fix/** docs/** chore/**`).
|
||||
With `act_runner` capacity 1, a branch prefix that IS in that list fires a second redundant run
|
||||
which serialises ahead of the PR run and doubles wall-clock. Exploited deliberately in
|
||||
`infra/plex/claim-ship.sh`; the honest fix is to align the trigger list.
|
||||
- **The "~4-minute TTL"** asserted in four places was **uncited**, and upstream `pms-docker`
|
||||
documents no expiry at all. The authoritative reading is the countdown plex.tv/claim renders at
|
||||
issue time. Removed from `runbooks/plex-claim.md`; the same claim still sits in
|
||||
`catalog/plex.yaml` and the Secret's header comment (left alone — editing a `.sops.yaml`
|
||||
comment risks the MAC, and touching `catalog/plex.yaml` re-renders a golden).
|
||||
|
||||
### Measured, once the pipeline actually worked
|
||||
|
||||
`gate / gate (pull_request)` 1m34s–1m58s. Full claim loop end to end **~2m45s** — comfortably
|
||||
inside the window, which retires the theory that a short-lived token cannot survive a PR flow.
|
||||
That theory was mine and it was wrong: the pipeline was never too slow, it simply did not exist.
|
||||
|
||||
### State now
|
||||
|
||||
- `main` **protected**, `user_can_push:false`. Runner `incluster-dind` online.
|
||||
- `CLAUDE.md` is the agent delivery contract; `infra/gitea/` holds the protection + runner scripts.
|
||||
- **All 37 runbooks** that instructed `git push origin main` now describe the PR flow — that
|
||||
instruction had become actively blocked, not merely stale.
|
||||
- `infra/plex/claim-ship.sh` drives the claim loop end to end.
|
||||
|
||||
### Next steps
|
||||
|
||||
1. **`enable_status_check: true`** with `gate / gate (pull_request)` as the only context. The
|
||||
prerequisite is met: the gate has been observed green on real PRs.
|
||||
2. Add `jakman85` as a repo **collaborator**, then `required_approvals: 1`. (Currently the only
|
||||
org member is `gitea-admin`, which agents authenticate as — requiring approval before that is
|
||||
fixed would leave nobody able to approve.)
|
||||
3. Run the **push probe** against protection: `gitea-admin` is a site admin and Gitea may let it
|
||||
bypass. `user_can_push:false` is the API's opinion, not proof.
|
||||
4. Align `gate.yml`'s push trigger with this repo's `feat/**` prefix.
|
||||
5. Convert the `dind` container to a **native sidecar** with a `startupProbe`, per the standing
|
||||
FOLLOW-UP note in `apps/wave-1/gitea-actions.yaml`. It deletes both hand-rolled wait loops and
|
||||
the envsubst hazard. Land it ALONE — it is the merge gate for the whole repo.
|
||||
6. Create the Plex libraries over `/media/movies`, `/media/tv`, `/media/photos`.
|
||||
@@ -46,7 +46,7 @@ metadata:
|
||||
app.kubernetes.io/name: plex
|
||||
type: Opaque
|
||||
stringData:
|
||||
PLEX_CLAIM: ENC[AES256_GCM,data:8O9nvFw+6OR+53mXvHfe321aSmYUQcZuAqLKZpcSlUwNyeU+Nbp/tRPBlg==,iv:EbvvOsXlXjAFOewnWbK28ErmUdWcJJN0Q5Gm53El3ZM=,tag:wXaInGFQ+wqUt765vvjCDA==,type:str]
|
||||
PLEX_CLAIM: ENC[AES256_GCM,data:nLmzeUBWRhRjUakrGiGzQpQuqZNcIYvRH6M=,iv:z0on/dArsNejNLrp3V1nSiNLFzKcREhPuMO1634CwQw=,tag:Ih23CT1JYna/5lfYQUb1Dw==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
@@ -62,8 +62,8 @@ sops:
|
||||
QUtYdHEvN042ajNoNktOLzlYQmQrc1kK4NHozMbXVzNGYbUBRbK6a9Uw/sVPAfe6
|
||||
RW5KWzZ4I/hloqjN3zzVISoSMieMkBGDnF5qZS5N6FqIc6/HrsDD8A==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-09-29T03:25:20Z"
|
||||
mac: ENC[AES256_GCM,data:u5uVmOdXN78UaKUMkPZzqeW84222+tcG57rFKIKcvRKwJkzMTOuSFOeQpKh4eK4KZXVKK7+Q3mFf5G8a+jjuKjYbjSCpR38AF3TJv3xrj8fhnqR+8QHRp8tsgA0jJZBS55N7KKDyOztLFo49sI7mNnlrUwhWTQUkbLgN1LxF5MA=,iv:P4W9G2mI/3VTkvSrDZvnkJ0Qxb1nFmXcF35CrSNhako=,tag:yVsg7xsUALUoCJH0cHfQjw==,type:str]
|
||||
lastmodified: "2026-09-29T06:31:16Z"
|
||||
mac: ENC[AES256_GCM,data:62BmBAq45CD2g6LDyx+Ho4bTfrhJcaro/RB/iM+OaRc4a/D83MhjbcBL11Uz/O726inqhkvCKDJ/fFemoc50JDbtC79cYMaxrGJNjFM0sdckWrowPvh6ie7HHHUBRBaefl+gNfnc+pbvcncvRzwA4RyzcZse1wbxKfHN8FjEopE=,iv:h0cMShs3tIgnY6eruUYymYrcLFzQZLk/uXQJ/Jsd5ek=,tag:nZ9Pau7p4xWT1QCN1vv2vA==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
@@ -14,7 +14,7 @@ metadata:
|
||||
app.kubernetes.io/name: act-runner
|
||||
type: Opaque
|
||||
stringData:
|
||||
token: ENC[AES256_GCM,data:W48mEtBqGeukX13lMhrhrer+zB21cLVp0acy3m1Xy/oo89DqO8FX/Q==,iv:wfTUU7RiFLdEc1fmEFS9DfE2SmGCFLPK17m0IKspft0=,tag:UhoXA47p7386KHcYNbPzCg==,type:str]
|
||||
token: ENC[AES256_GCM,data:f574mAUkLvMGq0TYycwCd4tduMuBgQLiTg/140kuRDgFUYYm1ZoSjQ==,iv:Tn7gFWBLm2hHk/BbfVX7qIZegwZp9ah1cMou5i2O91E=,tag:TkdSiMI4sdCNn2JuN++Pig==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
@@ -30,8 +30,8 @@ sops:
|
||||
MDZLUE52QzJXbFlrNWxOcVhJbDNnY00KxCpFkbhhCuk4Fa8CC3C4+R9AcsUnaXQR
|
||||
UjAfwKFoXAneXUih7TTXMbR/6+VgQBww5Bmwpn7jVLnzP0N0pIZVTA==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-08-26T04:48:14Z"
|
||||
mac: ENC[AES256_GCM,data:b99/zkL332RIT3FYBa1sR3o9+pwlmrUgKvVHawbjWYSJZ7ERUfO+LXuhKJ0HKrp6i1JfkurZlRO27iBcfrBisAXS9KLaqmsCJf4kKfFc4skANrmKCzprrXUmZexr1uiE8eDUzjNCLNxSCZfnFR6d/AkMXJqi6Hsj9/O+wH2lWsc=,iv:3DIPszSdxK81ERO3o/nTs33qQYHaAKLNcGP4qlZViFc=,tag:/B6VKkzMUwh0byNUicS5wQ==,type:str]
|
||||
lastmodified: "2026-09-29T05:48:35Z"
|
||||
mac: ENC[AES256_GCM,data:nobqr794nrdRYFrd8Ct8bw9M8NtftJXawRGk9zrL+pTWahwlWrwXVtOAxMGEPVojz3WemsxTJESzmzy0gcT1QcrjpYTsF2MVHcr6+umk8q9gAh2enyxy/0QkYiSt6XjvRVqfqleLanEkeIq1FPf2zhSLSJaiw4/oZTfkCHD1/Zg=,iv:JxaP0htmGcvE4ZU1YAmGF4A6JkejV29NNygyMl1OKF8=,tag:G70NvcPyVuagXhugWoUfTw==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
@@ -0,0 +1,82 @@
|
||||
# infra/gitea — platform settings that are NOT GitOps-managed
|
||||
|
||||
Gitea's own repo settings (branch protection, required status checks, approvals) live
|
||||
in Gitea's database, not in a manifest Flux reconciles. They are therefore **drift-prone
|
||||
and invisible to the gate** — exactly how the state below went unnoticed.
|
||||
|
||||
## The gap this closes
|
||||
|
||||
Verified against the live API on 2026-09-29, against a `DECISIONS-LOG.md` entry
|
||||
(2026-08-31) asserting the opposite:
|
||||
|
||||
| Claimed | Measured |
|
||||
|---|---|
|
||||
| `main` is a protected branch, gate is a required status check | `branch_protections` → `[]`; `protected:false`, `enable_status_check:false`, `required_approvals:0`, `user_can_push:true` |
|
||||
| Runner `incluster-dind` online | **zero** runners registered at repo, org and site level |
|
||||
|
||||
So `main` accepted direct pushes, and the gate could not run at all. `CLAUDE.md` is the
|
||||
rule; this directory is how the rule gets enforced by the machine.
|
||||
|
||||
## `branch-protection.sh`
|
||||
|
||||
Blocks direct pushes to `main`. Run it, then run the push probe it prints — **verify,
|
||||
do not trust the API response**, since Gitea may let site admins bypass protection.
|
||||
|
||||
./infra/gitea/branch-protection.sh
|
||||
|
||||
It authenticates with the credential `git` already has stored for this host, and never
|
||||
prints the secret.
|
||||
|
||||
## `runner-token-rotate.sh`
|
||||
|
||||
Mints a fresh instance-level registration token from the Gitea API and writes it into
|
||||
`apps/wave-1/gitea-actions-runner-secret.sops.yaml` via `sops set`. **The token value is
|
||||
never printed.**
|
||||
|
||||
./infra/gitea/runner-token-rotate.sh
|
||||
|
||||
A fresh token is usually **not sufficient on its own**. `act_runner` registers exactly once
|
||||
and persists its identity in `/data/.runner` on the `act-runner-data` PVC. If Gitea no
|
||||
longer holds a matching runner row, the pod starts, sees `.runner`, **skips registration**,
|
||||
and polls with an identity Gitea does not recognise — so it never appears in the runner
|
||||
list. That is precisely the observed symptom: zero runners, pod running, no error anyone
|
||||
looks at. Clearing the PVC is what forces a real re-registration; the script prints the
|
||||
commands.
|
||||
|
||||
## Sequence — order matters
|
||||
|
||||
1. **Get an Actions runner registered** — `runner-token-rotate.sh`, then clear the PVC so
|
||||
the runner actually re-registers, then confirm it shows `online` in the API. Do not
|
||||
skip the confirmation: the whole reason this went unnoticed is that a runner was
|
||||
*recorded* as online without anyone checking.
|
||||
2. **Watch the gate actually report on a real PR**, and note the context name it produced.
|
||||
3. **Only then set `enable_status_check: true`** with that exact context.
|
||||
4. Add the human reviewer as a repo collaborator, then raise `required_approvals` to 1.
|
||||
|
||||
> **Do not do step 3 before step 1.** A required status check with no runner leaves every
|
||||
> PR permanently pending — including the PR that would fix it. That half is measured: PR #1
|
||||
> sat `pending` for 30.6 minutes across 122 polls with no runner registered, and never
|
||||
> reported.
|
||||
|
||||
### Measured, once the runner came up (2026-09-29)
|
||||
|
||||
The first CI jobs this repo ever ran, on PR #1 and PR #2:
|
||||
|
||||
| Context | Result | Time |
|
||||
|---|---|---|
|
||||
| `gate / gate (pull_request)` | success | 1m58s (PR #1), 1m34s (PR #2) |
|
||||
| `gate / gate (push)` | success | 1m32s (PR #2 only) |
|
||||
| `gate / conformance (pull_request)` | **success — "Has been skipped"** | — |
|
||||
|
||||
**Require `gate / gate (pull_request)`. Do NOT require either `conformance` context —
|
||||
but not for the reason an earlier draft of this file gave.** A skipped Gitea Actions job
|
||||
does not hang a status check: Gitea reports it as **`success`** with the description
|
||||
"Has been skipped". So requiring it would not deadlock anything — it would pass
|
||||
*vacuously*, every time, forever, while nothing ran. Under HARD RULE 4 that is worse than
|
||||
a deadlock: a deadlock is loud, a permanently-green check that verifies nothing is
|
||||
exactly the fail-open shape this repo keeps finding.
|
||||
|
||||
The `(push)` contexts appear only for branches matching `gate.yml`'s push trigger
|
||||
(`feature/**`, `fix/**`, `docs/**`, `chore/**`). `feat/**` is **not** in that list, which
|
||||
is why PR #1's `feat/` branch produced only `pull_request` contexts. Require only the
|
||||
`pull_request` context, or PRs from non-matching prefixes will never satisfy the rule.
|
||||
Executable
+57
@@ -0,0 +1,57 @@
|
||||
#!/usr/bin/env bash
|
||||
# Enable branch protection on main for home-network-k3s-impl.
|
||||
#
|
||||
# Sets enable_push:false -> no direct pushes to main; changes land only by merging a PR.
|
||||
#
|
||||
# enable_status_check is deliberately FALSE. There is currently NO Gitea Actions runner
|
||||
# registered (verified 2026-09-29: zero at repo, org and site level), so requiring the
|
||||
# gate context would deadlock EVERY PR permanently -- including the PR that fixes it.
|
||||
# Turn it on only after a runner is live and you have seen the gate actually report on
|
||||
# a real PR, using the context name CI itself produced.
|
||||
#
|
||||
# required_approvals is 0 for the same class of reason: jakman85 is a site admin but NOT
|
||||
# a repo collaborator, so requiring an approval now could leave nobody able to approve.
|
||||
# Add the collaborator first, then raise this to 1.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev
|
||||
OWNER=Home-Network
|
||||
REPO=home-network-k3s-impl
|
||||
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"
|
||||
P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
[ -n "$U" ] && [ -n "$P" ] || { echo "no stored credential for $HOST" >&2; exit 1; }
|
||||
echo "authenticating as '${U}' (secret not shown)"
|
||||
|
||||
echo "==> creating branch protection on 'main'"
|
||||
jq -n '{
|
||||
branch_name: "main",
|
||||
enable_push: false,
|
||||
enable_status_check: false,
|
||||
status_check_contexts: [],
|
||||
required_approvals: 0,
|
||||
block_on_rejected_reviews: true,
|
||||
block_on_outdated_branch: false,
|
||||
dismiss_stale_approvals: true
|
||||
}' | curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
|
||||
-X POST -H 'Content-Type: application/json' --data @- \
|
||||
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections" \
|
||||
| jq -r 'if .branch_name then "created" else "API said: \(.message // tostring)" end'
|
||||
|
||||
echo "==> verifying (this is the part that matters)"
|
||||
curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
|
||||
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \
|
||||
| jq -r '{protected, user_can_push, required_approvals, enable_status_check, status_check_contexts}'
|
||||
|
||||
cat <<'NOTE'
|
||||
|
||||
Now prove it actually blocks you -- do not take the API's word for it. From a clean
|
||||
checkout on main, with nothing to lose:
|
||||
|
||||
git commit --allow-empty -m 'protection probe'
|
||||
git push origin main # EXPECT: rejected by the remote
|
||||
git reset --hard HEAD~1 # tidy up either way
|
||||
|
||||
If that push SUCCEEDS, Gitea is letting site admins bypass protection and this is not
|
||||
yet doing what you want.
|
||||
NOTE
|
||||
Executable
+61
@@ -0,0 +1,61 @@
|
||||
#!/usr/bin/env bash
|
||||
# Mint a fresh Gitea Actions registration token and write it into the SOPS secret.
|
||||
#
|
||||
# The token value is NEVER printed. It goes from the Gitea API into `sops set` and
|
||||
# nowhere else. (It is briefly visible in this host's process list as a sops argument;
|
||||
# acceptable on a single-operator workstation, and far better than echoing it.)
|
||||
#
|
||||
# This ALONE may not bring the runner back -- see "IMPORTANT" at the end of the output.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev
|
||||
SECRET=apps/wave-1/gitea-actions-runner-secret.sops.yaml
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||
cd "$REPO_ROOT"
|
||||
[ -f "$SECRET" ] || { echo "not in the repo root, or $SECRET is missing" >&2; exit 1; }
|
||||
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"
|
||||
P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
[ -n "$U" ] && [ -n "$P" ] || { echo "no stored git credential for $HOST" >&2; exit 1; }
|
||||
echo "==> minting an instance-level registration token as '${U}'"
|
||||
|
||||
TOKEN="$(curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
|
||||
"https://${HOST}/api/v1/admin/runners/registration-token" | jq -r '.token // empty')"
|
||||
[ -n "$TOKEN" ] || { echo "Gitea did not return a token" >&2; exit 1; }
|
||||
echo "==> got a ${#TOKEN}-character token (value not shown)"
|
||||
|
||||
echo "==> writing it into ${SECRET} via sops"
|
||||
DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run \
|
||||
sops set "$SECRET" '["stringData"]["token"]' "\"${TOKEN}\""
|
||||
unset TOKEN
|
||||
|
||||
echo "==> done. Verify the file is still encrypted (no plaintext token):"
|
||||
grep -q 'ENC\[' "$SECRET" && echo " OK - stringData is still SOPS-encrypted" \
|
||||
|| { echo " REFUSING TO CONTINUE: $SECRET is not encrypted" >&2; exit 1; }
|
||||
|
||||
cat <<'NOTE'
|
||||
|
||||
Next: commit this on a branch, open a PR, and merge it so Flux applies the new Secret.
|
||||
|
||||
IMPORTANT -- a new token is usually NOT sufficient on its own.
|
||||
act_runner registers exactly ONCE and then persists its identity in /data/.runner on the
|
||||
act-runner-data PVC. If Gitea no longer has a matching runner row (DB restore, the row was
|
||||
deleted, the instance was rebuilt), the pod starts, sees .runner, SKIPS registration, and
|
||||
polls with an identity Gitea does not recognise -- so it never appears in the runner list,
|
||||
which is exactly the observed symptom (zero runners, pod running).
|
||||
|
||||
So after the Secret lands, force a genuine re-registration by clearing that state:
|
||||
|
||||
kubectl --context k3s-prod -n gitea-actions scale deploy/act-runner --replicas=0
|
||||
kubectl --context k3s-prod -n gitea-actions delete pvc act-runner-data
|
||||
kubectl --context k3s-prod -n gitea-actions scale deploy/act-runner --replicas=1
|
||||
|
||||
(The PVC holds only runner registration state -- no build artifacts, nothing durable.
|
||||
Deleting it is the supported way to re-register. Flux recreates it from the manifest.)
|
||||
|
||||
Then confirm it actually registered -- do not assume:
|
||||
|
||||
curl -s -u '<user>' https://gitea.home.keithjackson.dev/api/v1/admin/actions/runners | jq
|
||||
|
||||
A runner must appear with status "online" and the labels ubuntu-latest / homelab.
|
||||
NOTE
|
||||
Executable
+93
@@ -0,0 +1,93 @@
|
||||
#!/usr/bin/env bash
|
||||
# plex-claim-ship.sh — run this IMMEDIATELY AFTER editing the SOPS secret with a fresh
|
||||
# plex.tv claim token. It drives the whole CI/CD loop end to end and prints a running
|
||||
# stopwatch so you can see the budget being spent.
|
||||
#
|
||||
# branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verify
|
||||
#
|
||||
# It never sees, prints, or handles the token: the file is already encrypted by sops.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev
|
||||
OWNER=Home-Network
|
||||
REPO=home-network-k3s-impl
|
||||
SECRET=apps/wave-0/plex-claim-secret.sops.yaml
|
||||
T0=$(date +%s)
|
||||
el() { printf '\n[T+%02d:%02d] %s\n' $(( ($(date +%s)-T0)/60 )) $(( ($(date +%s)-T0)%60 )) "$1"; }
|
||||
|
||||
cd "$(git rev-parse --show-toplevel)"
|
||||
|
||||
# --- preflight: refuse to ship a file that is not what we think it is -------------
|
||||
git diff --quiet -- "$SECRET" && { echo "ERROR: $SECRET is unchanged. Edit it first:"; \
|
||||
echo " infra/devcontainer/dev.sh run sops $SECRET"; exit 1; }
|
||||
grep -q 'PLEX_CLAIM: ENC\[' "$SECRET" || { echo "ERROR: $SECRET is not encrypted. REFUSING."; exit 1; }
|
||||
[ -z "$(git status --porcelain | grep -v " $SECRET\$")" ] || echo "NOTE: other files are dirty; only $SECRET will be committed."
|
||||
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
||||
|
||||
# feat/ prefix ON PURPOSE. act_runner capacity is 1, and gate.yml's PUSH trigger covers
|
||||
# feature/** fix/** docs/** chore/** but NOT feat/** — so a chore/ branch would fire a
|
||||
# second, redundant run that SERIALISES ahead of the PR run and doubles the wall clock.
|
||||
BRANCH="feat/plex-claim-token-$(date +%H%M%S)"
|
||||
|
||||
el "committing on $BRANCH"
|
||||
git checkout -q -b "$BRANCH"
|
||||
git add "$SECRET"
|
||||
git commit -q -m 'chore(plex): fresh plex.tv claim token
|
||||
|
||||
Injected inside the plex.tv claim window per runbooks/plex-claim.md.'
|
||||
SHA=$(git rev-parse HEAD)
|
||||
|
||||
el "pushing"
|
||||
git push -q -u origin "$BRANCH"
|
||||
|
||||
el "opening PR"
|
||||
PR=$(jq -n --arg h "$BRANCH" --arg b main --arg t 'chore(plex): fresh plex.tv claim token' \
|
||||
'{head:$h,base:$b,title:$t,body:"Token injection inside the claim window. Wiring already reviewed in #1."}' \
|
||||
| api -X POST -H 'Content-Type: application/json' --data @- \
|
||||
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls" | jq -r '.number')
|
||||
[ -n "$PR" ] && [ "$PR" != "null" ] || { echo "PR creation failed"; exit 1; }
|
||||
echo " PR #$PR"
|
||||
|
||||
el "waiting for the gate (expect ~1m40s)"
|
||||
for i in $(seq 1 60); do
|
||||
st=$(api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/commits/${SHA}/status" | jq -r '.state')
|
||||
printf '\r gate=%s [T+%ss] ' "$st" "$(( $(date +%s)-T0 ))"
|
||||
case "$st" in success) echo; break;; failure|error) echo; echo "GATE FAILED — stopping."; exit 1;; esac
|
||||
sleep 5
|
||||
done
|
||||
[ "$st" = "success" ] || { echo; echo "gate did not finish in time"; exit 1; }
|
||||
|
||||
el "merging PR #$PR"
|
||||
api -X POST -H 'Content-Type: application/json' -d '{"Do":"merge"}' \
|
||||
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls/${PR}/merge" >/dev/null
|
||||
git checkout -q main && git pull -q --ff-only origin main
|
||||
|
||||
el "forcing the Flux reconcile (--with-source, or it applies the OLD revision)"
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-0 --with-source'
|
||||
|
||||
# MANDATORY here. Only the Secret ciphertext changed, so the Deployment pod template is
|
||||
# byte-identical and NOTHING restarts on its own. PLEX_CLAIM is read once, at container
|
||||
# start. Skipping this is failure mode #2 in runbooks/plex-claim.md.
|
||||
el "rolling the pod (mandatory — a changed Secret restarts nothing)"
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex rollout restart deployment/plex && \
|
||||
kubectl --context k3s-prod -n plex rollout status deployment/plex --timeout=120s'
|
||||
|
||||
el "verdict"
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex logs deploy/plex --since=3m | grep -i -e claim -e token || true'
|
||||
cat <<'NOTE'
|
||||
|
||||
"Attempting to obtain server token..." + "Token obtained successfully" -> CLAIMED
|
||||
"Attempting..." with no success line -> token expired; retry, nothing is broken
|
||||
neither line -> already claimed, or env did not reach the container
|
||||
|
||||
Confirm it persisted (prints the attribute NAME only, never its value):
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c \
|
||||
"grep -o PlexOnlineToken= \"/config/Library/Application Support/Plex Media Server/Preferences.xml\" | head -1"'
|
||||
NOTE
|
||||
+7
-4
@@ -33,7 +33,8 @@ negative is how a CLI quirk becomes real data loss. Write `backups.velero.io`,
|
||||
`backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`,
|
||||
`volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
`git push origin main` **is** the deploy; there is no other apply step. Durable changes go through git.
|
||||
Merging a PR into `main` **is** the deploy; there is no other apply step. `main` is protected, so a
|
||||
direct push is rejected — durable changes go through a branch, a PR, and a green gate.
|
||||
|
||||
## Alerts you should not trust the silence of (measured 2026-09-13)
|
||||
|
||||
@@ -109,9 +110,11 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
|
||||
every green signal missed.
|
||||
|
||||
### Operator procedures (no alert links here)
|
||||
- **`plex-claim`** — claiming the Plex Media Server. The plex.tv claim token lives ~4 minutes, so this
|
||||
is a timed sequence: token → `sops` → push → **forced** `flux reconcile` → pod roll. Includes why a
|
||||
changed Secret does not restart anything, and why an expired token is harmless.
|
||||
- **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only
|
||||
applies after a `/config` wipe. The claim token is short-lived — read the countdown on
|
||||
plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
|
||||
token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
|
||||
Includes why a changed Secret does not restart anything, and why an expired token is harmless.
|
||||
|
||||
### Virtualisation
|
||||
- **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.
|
||||
|
||||
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -78,8 +78,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -55,8 +55,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -63,8 +63,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -87,8 +87,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -69,8 +69,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -71,8 +71,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -68,8 +68,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
## What this means here
|
||||
|
||||
Something in the GitOps chain is stuck at Ready=False. On this cluster that is more consequential than
|
||||
it sounds, because **`git push origin main` IS the deploy** — there is no other apply step. A stuck
|
||||
it sounds, because **merging a PR into `main` IS the deploy** — there is no other apply step. A stuck
|
||||
Kustomization means git and the cluster have diverged and nothing will close the gap.
|
||||
|
||||
The chain has **14 tiers**: `flux-system` → `cilium` → `platform` → `platform-config` →
|
||||
@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -58,8 +58,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -87,8 +87,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -67,8 +67,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -59,8 +59,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -57,8 +57,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -83,7 +83,7 @@ configMapGenerator:
|
||||
- node-memory-swap=node-memory-swap.md
|
||||
# Operator procedure, not an alert page (no alert links to it — the
|
||||
# alert-runbooks-resolvable check treats that as advisory). It is served
|
||||
# because the 4-minute plex.tv claim window is not a thing to go looking
|
||||
# because the plex.tv claim window is short and is not a thing to go looking
|
||||
# for in git: /runbooks/plex-claim has to resolve the moment it is needed.
|
||||
- plex-claim=plex-claim.md
|
||||
- pvc-capacity=pvc-capacity.md
|
||||
|
||||
@@ -83,8 +83,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -74,8 +74,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -75,8 +75,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -52,8 +52,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -49,8 +49,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -56,8 +56,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -56,8 +56,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
+4
-2
@@ -227,8 +227,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -52,8 +52,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
+59
-55
@@ -1,4 +1,4 @@
|
||||
# plex-claim — claiming the Plex Media Server inside the 4-minute token window
|
||||
# plex-claim — claiming the Plex Media Server inside the token window
|
||||
|
||||
**Alert:** none. This is an **operator procedure**, not an incident page — it is indexed here because
|
||||
it is the page you need at the moment you need it, with a stopwatch running.
|
||||
@@ -14,10 +14,11 @@ server to a plex.tv account. It is a **one-shot** operation: the token is exchan
|
||||
the `PLEX_CLAIM` env is **ignored on every start** (upstream gate:
|
||||
`[ ! -z "${PLEX_CLAIM}" ] && [ -z "${token}" ]` in `root/etc/cont-init.d/40-plex-first-run`).
|
||||
|
||||
**A `https://plex.tv/claim` token lives ~4 minutes.** That is shorter than commit → push → Flux's
|
||||
10-minute reconcile interval → pod roll if you let those run on their own timers, which is why this
|
||||
page exists: every step below is written to be run back-to-back, with the reconcile **forced** rather
|
||||
than waited for. Budget: token in hand at T+0, pod restarted by roughly T+2:00.
|
||||
**A `https://plex.tv/claim` token is short-lived — minutes, not hours.** Read the countdown the
|
||||
claim page shows you; do not budget against a number written here. That is shorter than PR → gate →
|
||||
merge → Flux's 10-minute reconcile interval → pod roll if you let those run on their own timers,
|
||||
which is why this page exists: the loop is scripted and the reconcile is **forced** rather than
|
||||
waited for. Measured end to end on 2026-09-29: **~2m45s**.
|
||||
|
||||
**An expired or wrong token does not break anything.** Verified against the image's own first-run
|
||||
script (and the copy inside `plexinc/pms-docker:latest`): a failed exchange gets a 403 with an empty
|
||||
@@ -66,81 +67,84 @@ infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0'
|
||||
```
|
||||
|
||||
## The 4-minute procedure
|
||||
## The procedure
|
||||
|
||||
Have this page open and do not stop between steps.
|
||||
> **STATUS: this server was CLAIMED on 2026-09-29** and `PlexOnlineToken` is present in
|
||||
> `/config` on the `plex-config` PVC. The upstream exchange is gated on that pref being
|
||||
> *absent*, so `PLEX_CLAIM` is now a **no-op on every start** whatever it holds. You only need
|
||||
> this page again after a `/config` wipe or a rebuild onto an empty PVC.
|
||||
|
||||
**On the clock.** The token is short-lived — **read the countdown that plex.tv/claim shows you**
|
||||
rather than trusting a number written here. (Earlier revisions of this page asserted a "~4-minute
|
||||
TTL" in four places with no citation; upstream `pms-docker` documents no expiry at all. The page
|
||||
is the only authority, and it is on screen at T+0.)
|
||||
|
||||
**A burnt token costs nothing.** An expired token gets a 403 with an empty body, `curl`'s status is
|
||||
never checked, the `if` does not fire, there is no `else` and no `exit 1`, and the script (no
|
||||
`set -e`) exits 0. The pod comes up unclaimed and serving. So retry freely.
|
||||
|
||||
### One command
|
||||
|
||||
1. **T+0 — get the token.** Sign in to plex.tv as the account that should OWN this server, open
|
||||
<https://plex.tv/claim>, copy the `claim-…` string. The clock starts now.
|
||||
<https://plex.tv/claim>, copy the `claim-…` string.
|
||||
|
||||
2. **T+0:15 — put it in the Secret**, in place, encrypted, without it ever touching a plaintext file:
|
||||
2. **Put it in the Secret** — in place, encrypted, never touching a plaintext file:
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
|
||||
```
|
||||
|
||||
Replace the `PLEX_CLAIM:` value (the placeholder `replace-me-with-a-fresh-plex-tv-claim-token`, or
|
||||
a previous expired token) and save. `sops` re-encrypts on write to the committed age recipient.
|
||||
|
||||
3. **T+0:45 — commit and push.** `git push origin main` **is** the deploy; there is no other apply
|
||||
step.
|
||||
3. **Ship it.** This drives the whole delivery loop and prints a running stopwatch:
|
||||
|
||||
```
|
||||
git add apps/wave-0/plex-claim-secret.sops.yaml
|
||||
git commit -m 'chore(plex): fresh plex.tv claim token'
|
||||
git push origin main
|
||||
./infra/plex/claim-ship.sh
|
||||
```
|
||||
|
||||
4. **T+1:00 — force the reconcile immediately. This is the one command that saves the window** — it
|
||||
pulls the new commit and re-applies wave 0 instead of waiting up to 10 minutes for the interval:
|
||||
branch → commit → push → PR → gate → merge → `flux reconcile --with-source` → `rollout restart`
|
||||
→ verdict. It refuses to run if the file is unchanged or not encrypted.
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-0 --with-source'
|
||||
```
|
||||
**Measured end to end, 2026-09-29: ~2m45s.** Gate ~1m40s, merge + reconcile ~25s, pod roll and
|
||||
container start ~30s.
|
||||
|
||||
`--with-source` is the load-bearing flag: it reconciles the `flux-system` GitRepository first, so
|
||||
the Kustomization applies **your commit** and not the previously-fetched revision.
|
||||
### Why it is a script and not six pasted commands
|
||||
|
||||
5. **T+1:30 — roll the pod.** This step is **mandatory and easy to forget**: a changed Secret does
|
||||
**not** restart anything. `PLEX_CLAIM` is injected by `secretKeyRef` and read only at container
|
||||
start, and the Deployment's pod template is byte-identical across this change, so Flux applies the
|
||||
new Secret and the running pod keeps the old value forever.
|
||||
`main` is a **protected branch** — the direct push this page used to instruct is now rejected — so
|
||||
the deploy is a PR merged after a green gate. Three of those steps are easy to get wrong under time
|
||||
pressure, and two are silent when skipped:
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex rollout restart deployment/plex'
|
||||
```
|
||||
- **`--with-source` is load-bearing.** Without it the Kustomization applies the previously-fetched
|
||||
revision, not your commit.
|
||||
- **The pod roll is mandatory.** Only the Secret's ciphertext changes, so the Deployment's pod
|
||||
template is byte-identical and *nothing restarts on its own*. `PLEX_CLAIM` is read once, at
|
||||
container start. Skip this and the new Secret sits in the cluster while the running pod keeps the
|
||||
old value forever.
|
||||
- **The branch is named `feat/…` on purpose.** `act_runner` capacity is 1, and `gate.yml`'s push
|
||||
trigger covers `feature/** fix/** docs/** chore/**` but *not* `feat/**`. A `chore/` branch fires a
|
||||
second redundant run that serialises ahead of the PR run and roughly doubles the wall clock.
|
||||
|
||||
6. **T+2:00 — confirm the exchange happened**, in the new pod's init log:
|
||||
### Reading the verdict
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex logs deploy/plex --since=5m | grep -i -e claim -e token'
|
||||
```
|
||||
* `Attempting to obtain server token from claim token` **and** `Token obtained successfully`
|
||||
⇒ **claimed.** Re-run the First-commands `PlexOnlineToken=` check to confirm it persisted, then
|
||||
create the libraries over `/media/movies`, `/media/tv`, `/media/photos`.
|
||||
* `Attempting…` with **no** success line ⇒ the token was expired or invalid. Nothing is broken; go
|
||||
back to step 1 with a new one.
|
||||
* **Neither line** ⇒ the exchange was skipped: `/config` already had a `PlexOnlineToken` (already
|
||||
claimed — the expected case now), or the env did not reach the container. Check by NAME only:
|
||||
|
||||
* `Attempting to obtain server token from claim token` **and** `Token obtained successfully`
|
||||
⇒ **claimed.** Re-run the First-commands `PlexOnlineToken=` check to confirm it persisted, then
|
||||
create the libraries over `/media/movies`, `/media/tv`, `/media/photos`.
|
||||
* `Attempting to obtain server token from claim token` with **no** success line ⇒ the token was
|
||||
expired or invalid. Nothing is broken; go back to step 1 with a new token.
|
||||
* **Neither line** ⇒ the exchange was skipped, which means `/config` already had a
|
||||
`PlexOnlineToken` (already claimed) or the env did not reach the container. Check the pod actually
|
||||
has the variable, by NAME only:
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
|
||||
-o jsonpath="{.items[0].spec.containers[0].env[*].name}"'
|
||||
```
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
|
||||
-o jsonpath="{.items[0].spec.containers[0].env[*].name}"'
|
||||
```
|
||||
|
||||
## Likely causes, most likely first
|
||||
|
||||
When the claim does not take, in order of how often it is the answer here:
|
||||
|
||||
1. **The token expired** — you spent more than ~4 minutes, usually in step 3 or waiting on step 4.
|
||||
Symptom: the "Attempting…" line with no success line. Remedy: a new token, and force the reconcile
|
||||
rather than waiting for it.
|
||||
1. **The token expired** — you exceeded the countdown, usually by reviewing the PR instead of
|
||||
merging it the moment the gate went green. Symptom: the "Attempting…" line with no success line.
|
||||
Remedy: a new token, and let `claim-ship.sh` drive the loop rather than pasting steps.
|
||||
2. **The pod was never restarted** (step 5 skipped). Symptom: no claim log lines at all from a pod
|
||||
whose `AGE` predates your push. The Secret is correct in git and in the cluster and the running
|
||||
container still holds the placeholder.
|
||||
|
||||
@@ -58,8 +58,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -60,8 +60,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -54,8 +54,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -66,8 +66,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
Reference in new issue
Block a user