Compare commits
9
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
761e25cc3f | ||
|
|
8e017fad91 | ||
|
|
27d0ef3324 | ||
|
|
bbd4f682a6 | ||
|
|
0391139240 | ||
|
|
e9f441b5ce | ||
|
|
234cc6a71a | ||
|
|
69a0d1a1c7 | ||
|
|
4fd75ccdaa |
No files matched your search
@@ -31,17 +31,19 @@ Verified against the live Gitea on 2026-09-29:
|
|||||||
|---|---|
|
|---|---|
|
||||||
| `main` protected, direct push rejected | **No branch protection.** `protected: false`, `user_can_push: true` |
|
| `main` protected, direct push rejected | **No branch protection.** `protected: false`, `user_can_push: true` |
|
||||||
| PR blocked until the gate passes | **`enable_status_check: false`**, `status_check_contexts: []` |
|
| PR blocked until the gate passes | **`enable_status_check: false`**, `status_check_contexts: []` |
|
||||||
| Gate runs on every PR | **No Actions runner is registered** (repo, org and site level all report zero) |
|
| Gate runs on every PR | **Fixed 2026-09-29.** Runner `incluster-dind` is online; `gate / gate (pull_request)` runs and passes in ~1m30s–2m |
|
||||||
| Approval required before merge | **`required_approvals: 0`** |
|
| Approval required before merge | **`required_approvals: 0`** |
|
||||||
|
|
||||||
So nothing mechanically stops you from pushing to `main` right now. **That is a gap
|
So nothing mechanically stops you from pushing to `main` right now. **That is a gap
|
||||||
being closed, not permission.** `DECISIONS-LOG.md` (2026-08-31) claims the gate is a
|
being closed, not permission.** `DECISIONS-LOG.md` (2026-08-31) claimed the gate was a
|
||||||
"required status check" with a runner online; that claim is not true of the live
|
"required status check" with a runner online; neither was true, which is how this went
|
||||||
system, which is how this went unnoticed.
|
unnoticed for a month. The runner half is now genuinely fixed and observed green. The
|
||||||
|
protection half is not — see `infra/gitea/`.
|
||||||
|
|
||||||
**Consequence for you:** because CI cannot run, the **local toolbox gate is the only
|
**Consequence for you:** CI now runs, but it does not yet *block* anything. Always run
|
||||||
real gate**. Run it and quote its numbers in your report. Never describe a PR as
|
the toolbox gate yourself and quote its numbers. Never describe a PR as "green" on the
|
||||||
"green" on the strength of CI that did not execute.
|
strength of a check you have not seen report — and note that a **skipped** Gitea job
|
||||||
|
reports as `success`, so a green tick is not by itself evidence that anything ran.
|
||||||
|
|
||||||
## 3. Gate before you push
|
## 3. Gate before you push
|
||||||
|
|
||||||
|
|||||||
@@ -4781,3 +4781,85 @@ accurate before the PR flow and are stale now. Correcting them is tracked, not d
|
|||||||
3. Land `feat/plex-claim-sops-secret` (PR #1) once the gate can actually run.
|
3. Land `feat/plex-claim-sops-secret` (PR #1) once the gate can actually run.
|
||||||
4. Claim plex, after checking it is not already claimed.
|
4. Claim plex, after checking it is not already claimed.
|
||||||
5. Correct the 37 runbooks' `git push origin main` wording.
|
5. Correct the 37 runbooks' `git push origin main` wording.
|
||||||
|
|
||||||
|
## 2026-09-29 — Plex is claimed; the CI/CD gate that "enforced" it never existed. **PM HANDOFF.**
|
||||||
|
|
||||||
|
Began as "inject a plex.tv claim token". Ended by discovering the delivery pipeline was fiction,
|
||||||
|
rebuilding it, and shipping the token through it. Plex is **claimed** — `PlexOnlineToken` is
|
||||||
|
present in `/config` on the `plex-config` PVC, so the exchange is now a permanent no-op and the
|
||||||
|
library scan is running over the 8.33T NFS share.
|
||||||
|
|
||||||
|
### The finding: both halves of the 2026-08-31 CI claim were false
|
||||||
|
|
||||||
|
That entry recorded "Gitea Actions CI is LIVE + enforcing … required status check; runner
|
||||||
|
`incluster-dind` online". Measured against the live API on 2026-09-29:
|
||||||
|
|
||||||
|
| Claimed | Measured |
|
||||||
|
|---|---|
|
||||||
|
| `main` protected, gate required | `branch_protections` → `[]`; `protected:false`, `enable_status_check:false`, `required_approvals:0`, `user_can_push:true` |
|
||||||
|
| Runner online | **zero** runners at repo, org AND site level |
|
||||||
|
|
||||||
|
**PR #1 was the first pull request this repository has ever had.** Its contexts sat `pending`
|
||||||
|
for 30.6 minutes across 122 polls and never reported. For a month, every "the gate enforces this"
|
||||||
|
statement rested on an assertion nobody had checked — the same fail-open class as 2026-09-18,
|
||||||
|
and the reason it hid so long is that the log itself was the evidence.
|
||||||
|
|
||||||
|
### Root cause of the dead runner
|
||||||
|
|
||||||
|
`act_runner` registers **once** and persists its identity in `/data/.runner` on the
|
||||||
|
`act-runner-data` PVC. Gitea no longer held a matching row, so the pod booted, saw `.runner`,
|
||||||
|
**skipped registration**, and polled with an identity Gitea did not recognise. Symptom: zero
|
||||||
|
runners, pod Running, nothing crashlooping, no error anyone reads. Deleting the PVC forced a
|
||||||
|
genuine re-registration (`.runner is missing` → `Runner registered successfully`).
|
||||||
|
|
||||||
|
It then did not come back for several minutes, because Flux could not recreate the PVC:
|
||||||
|
`apps-wave-1` was blocked four levels deep on a **stale** `DependencyNotReady` for
|
||||||
|
`platform-config`, which itself read `Ready=True`. Each wave re-evaluates only on its own
|
||||||
|
`interval: 10m`, so a resource deleted at the bottom of the chain can take ~40 min to return.
|
||||||
|
**Force it** with `flux reconcile kustomization apps-wave-1 --with-source` rather than waiting.
|
||||||
|
|
||||||
|
### Corrections to things this log and the runbooks asserted
|
||||||
|
|
||||||
|
- **A skipped Gitea job reports `success`, not pending.** Measured: `gate / conformance` returned
|
||||||
|
`success` / "Has been skipped". An earlier entry (#2) said requiring it would deadlock every PR.
|
||||||
|
Wrong — it would pass **vacuously**, forever, while nothing ran. Under HARD RULE 4 that is worse
|
||||||
|
than a deadlock, because a deadlock is loud. Never require it; require
|
||||||
|
`gate / gate (pull_request)` alone. Corrected in #3.
|
||||||
|
- **`gate.yml`'s push trigger omits `feat/**`** (it lists `feature/** fix/** docs/** chore/**`).
|
||||||
|
With `act_runner` capacity 1, a branch prefix that IS in that list fires a second redundant run
|
||||||
|
which serialises ahead of the PR run and doubles wall-clock. Exploited deliberately in
|
||||||
|
`infra/plex/claim-ship.sh`; the honest fix is to align the trigger list.
|
||||||
|
- **The "~4-minute TTL"** asserted in four places was **uncited**, and upstream `pms-docker`
|
||||||
|
documents no expiry at all. The authoritative reading is the countdown plex.tv/claim renders at
|
||||||
|
issue time. Removed from `runbooks/plex-claim.md`; the same claim still sits in
|
||||||
|
`catalog/plex.yaml` and the Secret's header comment (left alone — editing a `.sops.yaml`
|
||||||
|
comment risks the MAC, and touching `catalog/plex.yaml` re-renders a golden).
|
||||||
|
|
||||||
|
### Measured, once the pipeline actually worked
|
||||||
|
|
||||||
|
`gate / gate (pull_request)` 1m34s–1m58s. Full claim loop end to end **~2m45s** — comfortably
|
||||||
|
inside the window, which retires the theory that a short-lived token cannot survive a PR flow.
|
||||||
|
That theory was mine and it was wrong: the pipeline was never too slow, it simply did not exist.
|
||||||
|
|
||||||
|
### State now
|
||||||
|
|
||||||
|
- `main` **protected**, `user_can_push:false`. Runner `incluster-dind` online.
|
||||||
|
- `CLAUDE.md` is the agent delivery contract; `infra/gitea/` holds the protection + runner scripts.
|
||||||
|
- **All 37 runbooks** that instructed `git push origin main` now describe the PR flow — that
|
||||||
|
instruction had become actively blocked, not merely stale.
|
||||||
|
- `infra/plex/claim-ship.sh` drives the claim loop end to end.
|
||||||
|
|
||||||
|
### Next steps
|
||||||
|
|
||||||
|
1. **`enable_status_check: true`** with `gate / gate (pull_request)` as the only context. The
|
||||||
|
prerequisite is met: the gate has been observed green on real PRs.
|
||||||
|
2. Add `jakman85` as a repo **collaborator**, then `required_approvals: 1`. (Currently the only
|
||||||
|
org member is `gitea-admin`, which agents authenticate as — requiring approval before that is
|
||||||
|
fixed would leave nobody able to approve.)
|
||||||
|
3. Run the **push probe** against protection: `gitea-admin` is a site admin and Gitea may let it
|
||||||
|
bypass. `user_can_push:false` is the API's opinion, not proof.
|
||||||
|
4. Align `gate.yml`'s push trigger with this repo's `feat/**` prefix.
|
||||||
|
5. Convert the `dind` container to a **native sidecar** with a `startupProbe`, per the standing
|
||||||
|
FOLLOW-UP note in `apps/wave-1/gitea-actions.yaml`. It deletes both hand-rolled wait loops and
|
||||||
|
the envsubst hazard. Land it ALONE — it is the merge gate for the whole repo.
|
||||||
|
6. Create the Plex libraries over `/media/movies`, `/media/tv`, `/media/photos`.
|
||||||
@@ -17,6 +17,7 @@ resources:
|
|||||||
- pihole-dns.yaml
|
- pihole-dns.yaml
|
||||||
- pihole-oauth2-proxy-credentials.sops.yaml
|
- pihole-oauth2-proxy-credentials.sops.yaml
|
||||||
- pihole.yaml
|
- pihole.yaml
|
||||||
|
- plex-claim-secret.sops.yaml
|
||||||
- plex.yaml
|
- plex.yaml
|
||||||
- qbittorrent-bootstrap-secret.sops.yaml
|
- qbittorrent-bootstrap-secret.sops.yaml
|
||||||
- qbittorrent-oauth2-proxy-credentials.sops.yaml
|
- qbittorrent-oauth2-proxy-credentials.sops.yaml
|
||||||
|
|||||||
@@ -0,0 +1,69 @@
|
|||||||
|
# Plex plex.tv server-claim token (WS-APPS). SOPS-encrypted (age).
|
||||||
|
# Consumed by catalog/plex.yaml as env PLEX_CLAIM via valueFrom.secretKeyRef.
|
||||||
|
#
|
||||||
|
# NOTE ON THESE COMMENTS: they carry NO dollar-brace expressions, deliberately. This file is applied
|
||||||
|
# by the apps-wave-0 Flux Kustomization, which runs envsubst over the built output, and kustomize
|
||||||
|
# preserves body comments into that stream (see conformance/tests/l0-static/flux-postbuild-
|
||||||
|
# substitution). A braced token quoted from the upstream shell script would be a live hazard here,
|
||||||
|
# not prose, so the upstream conditions below are described in words.
|
||||||
|
#
|
||||||
|
# THE VALUE SHIPPED WITH THIS FILE IS A PLACEHOLDER, DELIBERATELY — the literal string
|
||||||
|
# `replace-me-with-a-fresh-plex-tv-claim-token`, not a credential. A https://plex.tv/claim token has
|
||||||
|
# a ~4-minute TTL, far shorter than commit -> push -> Flux reconcile -> pod-roll, so no token
|
||||||
|
# committed here is ever still valid when the pod starts. This file exists to make the Secret — and
|
||||||
|
# therefore the env reference — ALWAYS RESOLVABLE IN THE FLUX GRAPH, not to carry a working token.
|
||||||
|
# The real token is injected by the operator inside that 4-minute window: runbooks/plex-claim.md.
|
||||||
|
#
|
||||||
|
# WHY SHIPPING A PLACEHOLDER IS SAFE (verified against plexinc/pms-docker branch `master`, the
|
||||||
|
# s6-overlay v2 script root/etc/cont-init.d/40-plex-first-run, cross-checked byte-identical against
|
||||||
|
# the same file extracted from the running `plexinc/pms-docker:latest` image):
|
||||||
|
# - line 106 guards the claim exchange on TWO conditions: PLEX_CLAIM non-empty AND the
|
||||||
|
# PlexOnlineToken pref read out of /config empty. Once the server is claimed that block is
|
||||||
|
# skipped entirely, so whatever this Secret holds is a NO-OP on every later start. Upstream
|
||||||
|
# README: "If server is already logged in, this parameter is ignored."
|
||||||
|
# - on a still-UNCLAIMED /config, an invalid or expired token makes plex.tv answer 403 with an
|
||||||
|
# empty body. curl's exit status is never checked, the sed for the authentication-token element
|
||||||
|
# yields the empty string, and the success branch at line 120 simply does not fire. There is no
|
||||||
|
# else branch and no non-zero exit; the script has no `set -e` and ends by touching
|
||||||
|
# /.firstRunComplete and echoing, so it exits 0. The server comes up UNCLAIMED and serving.
|
||||||
|
# It does NOT crashloop.
|
||||||
|
# - the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS, whose default is 0 ("continue silently even
|
||||||
|
# if a cont-init script failed"), so even a hypothetical non-zero exit would not halt startup.
|
||||||
|
# A MISSING Secret would NOT be safe: the ServiceDefinition schema's valueFrom.secretKeyRef shape
|
||||||
|
# has no `optional` field, so an absent Secret is a hard CreateContainerConfigError. That is exactly
|
||||||
|
# why this file is committed and listed in apps/wave-0/kustomization.yaml instead of being created
|
||||||
|
# by hand at claim time.
|
||||||
|
#
|
||||||
|
# INJECTION / ROTATION (operator, not agents) — runbooks/plex-claim.md has the timed procedure:
|
||||||
|
# infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
|
||||||
|
# Never decrypt this file into a shell, a log, or a shared terminal.
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Secret
|
||||||
|
metadata:
|
||||||
|
name: plex-claim
|
||||||
|
namespace: plex
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/name: plex
|
||||||
|
type: Opaque
|
||||||
|
stringData:
|
||||||
|
PLEX_CLAIM: ENC[AES256_GCM,data:nLmzeUBWRhRjUakrGiGzQpQuqZNcIYvRH6M=,iv:z0on/dArsNejNLrp3V1nSiNLFzKcREhPuMO1634CwQw=,tag:Ih23CT1JYna/5lfYQUb1Dw==,type:str]
|
||||||
|
sops:
|
||||||
|
kms: []
|
||||||
|
gcp_kms: []
|
||||||
|
azure_kv: []
|
||||||
|
hc_vault: []
|
||||||
|
age:
|
||||||
|
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||||
|
enc: |
|
||||||
|
-----BEGIN AGE ENCRYPTED FILE-----
|
||||||
|
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2dWtqOVNQalZ2QlRxOXJa
|
||||||
|
cDc1MTdtWENkdWIxS1JId3dIU1JDSmk0ODNrCk1YUzhFWXR3bUxyV2NNcmFOakdz
|
||||||
|
THl1Q284dFJxTy9Rd1h0eXJJaVZtdmcKLS0tIERuN0ZFeW9zdi9aNFRYNWxmUS9s
|
||||||
|
QUtYdHEvN042ajNoNktOLzlYQmQrc1kK4NHozMbXVzNGYbUBRbK6a9Uw/sVPAfe6
|
||||||
|
RW5KWzZ4I/hloqjN3zzVISoSMieMkBGDnF5qZS5N6FqIc6/HrsDD8A==
|
||||||
|
-----END AGE ENCRYPTED FILE-----
|
||||||
|
lastmodified: "2026-09-29T06:31:16Z"
|
||||||
|
mac: ENC[AES256_GCM,data:62BmBAq45CD2g6LDyx+Ho4bTfrhJcaro/RB/iM+OaRc4a/D83MhjbcBL11Uz/O726inqhkvCKDJ/fFemoc50JDbtC79cYMaxrGJNjFM0sdckWrowPvh6ie7HHHUBRBaefl+gNfnc+pbvcncvRzwA4RyzcZse1wbxKfHN8FjEopE=,iv:h0cMShs3tIgnY6eruUYymYrcLFzQZLk/uXQJ/Jsd5ek=,tag:nZ9Pau7p4xWT1QCN1vv2vA==,type:str]
|
||||||
|
pgp: []
|
||||||
|
encrypted_regex: ^(data|stringData)$
|
||||||
|
version: 3.9.4
|
||||||
@@ -132,6 +132,11 @@ spec:
|
|||||||
value: Etc/UTC
|
value: Etc/UTC
|
||||||
- name: ALLOWED_NETWORKS
|
- name: ALLOWED_NETWORKS
|
||||||
value: 10.0.0.0/8,192.168.0.0/16
|
value: 10.0.0.0/8,192.168.0.0/16
|
||||||
|
- name: PLEX_CLAIM
|
||||||
|
valueFrom:
|
||||||
|
secretKeyRef:
|
||||||
|
name: plex-claim
|
||||||
|
key: PLEX_CLAIM
|
||||||
volumeMounts:
|
volumeMounts:
|
||||||
- name: config
|
- name: config
|
||||||
mountPath: /config
|
mountPath: /config
|
||||||
|
|||||||
+40
-9
@@ -32,11 +32,40 @@ env:
|
|||||||
# (10.43.0.0/16); 192.168.0.0/16 covers the home LAN.
|
# (10.43.0.0/16); 192.168.0.0/16 covers the home LAN.
|
||||||
- name: ALLOWED_NETWORKS
|
- name: ALLOWED_NETWORKS
|
||||||
value: 10.0.0.0/8,192.168.0.0/16
|
value: 10.0.0.0/8,192.168.0.0/16
|
||||||
# PLEX_CLAIM is deliberately NOT set here: a https://plex.tv/claim token has a ~4-min TTL, far
|
# PLEX_CLAIM — the ONE-SHOT plex.tv server claim, sourced from the SOPS Secret `plex-claim`
|
||||||
# shorter than the commit -> push -> Flux reconcile -> pod-roll cycle, so any token committed to git
|
# (apps/wave-0/plex-claim-secret.sops.yaml, same wave => applied in the same reconcile as this
|
||||||
# is already expired by the time the pod starts. Claiming is a one-time LIVE step (UI sign-in, or a
|
# workload). Plex cannot create libraries over the mounted bulk share while the server is
|
||||||
# single API POST) against the running server; the claim then persists on the /config PVC and
|
# UNCLAIMED, so this is wired; it was previously left out on the reasoning below, and only one
|
||||||
# survives pod rolls (including the roll this env change causes), so it never belongs in git.
|
# half of that reasoning survived contact with the upstream source.
|
||||||
|
#
|
||||||
|
# THE 4-MINUTE TTL IS STILL TRUE, and it is now the OPERATIONAL constraint rather than a reason to
|
||||||
|
# abstain. A https://plex.tv/claim token expires in ~4 minutes — far shorter than commit -> push ->
|
||||||
|
# Flux reconcile -> pod-roll if that cycle is left to its own timers. So the committed value is a
|
||||||
|
# documented PLACEHOLDER and the live token is injected by the operator inside that window, with an
|
||||||
|
# explicit `flux reconcile` to collapse the wait: runbooks/plex-claim.md is the timed procedure.
|
||||||
|
#
|
||||||
|
# ONE-SHOT, AND SAFE TO SIT HERE EXPIRED. Verified from plexinc/pms-docker's own first-run script
|
||||||
|
# (branch `master`, root/etc/cont-init.d/40-plex-first-run, confirmed byte-identical to the file in
|
||||||
|
# the running `:latest` image), NOT assumed:
|
||||||
|
# - the exchange (line 106) is guarded on TWO conditions: PLEX_CLAIM non-empty AND the
|
||||||
|
# PlexOnlineToken pref in /config empty. Once claimed, the block is SKIPPED — this env is a
|
||||||
|
# no-op on every subsequent start, whatever it holds. (Upstream README: "If server is already
|
||||||
|
# logged in, this parameter is ignored.") The claim itself persists on the /config PVC as before.
|
||||||
|
# - an invalid/expired token on a still-unclaimed /config gets a 403 with an empty body; curl's
|
||||||
|
# status is unchecked, the token `sed` yields "", the success branch at line 120 does not fire,
|
||||||
|
# there is no `else` and no `exit 1`, and the script (no `set -e`) exits 0. It comes up UNCLAIMED and
|
||||||
|
# SERVING — it does NOT crashloop. The image also leaves S6_BEHAVIOUR_IF_STAGE2_FAILS at its
|
||||||
|
# default 0, which continues startup even when a cont-init script fails.
|
||||||
|
# So the earlier "a placeholder claim Secret would crashloop the pod" note was WRONG about the
|
||||||
|
# placeholder; it was right only about an ABSENT Secret. secretKeyRef here has no `optional` field,
|
||||||
|
# so the Secret must always exist in the Flux graph — which is why it is committed, encrypted, and
|
||||||
|
# listed in apps/wave-0/kustomization.yaml rather than hand-created at claim time. No schema change
|
||||||
|
# is needed: `optional` would only matter if a missing Secret had to be tolerated, and it doesn't.
|
||||||
|
- name: PLEX_CLAIM
|
||||||
|
valueFrom:
|
||||||
|
secretKeyRef:
|
||||||
|
name: plex-claim
|
||||||
|
key: PLEX_CLAIM
|
||||||
# ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex
|
# ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex
|
||||||
# which URL to advertise to plex.tv for REMOTE clients. The only candidate is
|
# which URL to advertise to plex.tv for REMOTE clients. The only candidate is
|
||||||
# https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL
|
# https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL
|
||||||
@@ -56,10 +85,12 @@ storage:
|
|||||||
# container's EPHEMERAL layer -> unclaimed + all state lost on restart. The image inits as
|
# container's EPHEMERAL layer -> unclaimed + all state lost on restart. The image inits as
|
||||||
# root (s6-svscan PID 1), chowns /config to the plex user, then drops -> no fsGroup needed
|
# root (s6-svscan PID 1), chowns /config to the plex user, then drops -> no fsGroup needed
|
||||||
# (verified in-cluster: PID1=root, "Plex Media Server"=plex). Persisting /config here also
|
# (verified in-cluster: PID1=root, "Plex Media Server"=plex). Persisting /config here also
|
||||||
# makes the claim survive restarts once claimed. To auto-claim on first boot instead of the
|
# makes the claim survive restarts once claimed — which is what makes the PLEX_CLAIM env
|
||||||
# UI sign-in, the operator adds a PLEX_CLAIM env from a fresh https://plex.tv/claim token
|
# above a ONE-SHOT: the token is exchanged for a PlexOnlineToken written here, and the
|
||||||
# (4-min TTL) — deliberately NOT wired here: the env secretKeyRef pattern has no `optional`,
|
# exchange is then skipped on every later start. (The older note here claimed a
|
||||||
# so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
|
# placeholder claim Secret "would crashloop the pod"; upstream's first-run script says
|
||||||
|
# otherwise — see the PLEX_CLAIM comment in `env` above. An ABSENT Secret is the hard
|
||||||
|
# failure, and that is why one is committed.)
|
||||||
# THE LIBRARY (docs/04 bulk tier). One READ-ONLY mount of the export ROOT, giving /media/movies,
|
# THE LIBRARY (docs/04 bulk tier). One READ-ONLY mount of the export ROOT, giving /media/movies,
|
||||||
# /media/tv and /media/photos — the whole tree the layout actually has (`ls /mnt/Storage/plex`:
|
# /media/tv and /media/photos — the whole tree the layout actually has (`ls /mnt/Storage/plex`:
|
||||||
# downloads/ movies/ photos/ tv/).
|
# downloads/ movies/ photos/ tv/).
|
||||||
|
|||||||
+15
-1
@@ -1844,6 +1844,10 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
|
|||||||
convention:
|
convention:
|
||||||
- `<svc>.yaml` -> wave(svc)
|
- `<svc>.yaml` -> wave(svc)
|
||||||
- `<svc>-bootstrap-secret.sops.yaml` -> wave(svc) (consumed by svc's env, same apply)
|
- `<svc>-bootstrap-secret.sops.yaml` -> wave(svc) (consumed by svc's env, same apply)
|
||||||
|
- `<svc>-claim-secret.sops.yaml` -> wave(svc) (a one-shot upstream-service claim token
|
||||||
|
consumed by svc's env — plex's plex.tv claim. Same wave as the app for the same reason the
|
||||||
|
bootstrap secret is: an env `secretKeyRef` has no `optional`, so the Secret must be applied
|
||||||
|
in the SAME reconcile as the workload or the pod is CreateContainerConfigError.)
|
||||||
- `<svc>-oauth2-proxy-credentials.sops.yaml` -> wave(svc) (forward-auth sidecar OIDC creds)
|
- `<svc>-oauth2-proxy-credentials.sops.yaml` -> wave(svc) (forward-auth sidecar OIDC creds)
|
||||||
- `<svc>-db.yaml` -> wave(svc-db) (a strictly-lower wave; CNPG DB)
|
- `<svc>-db.yaml` -> wave(svc-db) (a strictly-lower wave; CNPG DB)
|
||||||
- `<svc>-db-app.sops.yaml` -> wave(svc-db) (CNPG initdb owner secret)
|
- `<svc>-db-app.sops.yaml` -> wave(svc-db) (CNPG initdb owner secret)
|
||||||
@@ -1870,6 +1874,11 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
|
|||||||
w = waves[name]
|
w = waves[name]
|
||||||
add(w, f"{name}.yaml")
|
add(w, f"{name}.yaml")
|
||||||
add(w, f"{name}-bootstrap-secret.sops.yaml")
|
add(w, f"{name}-bootstrap-secret.sops.yaml")
|
||||||
|
# A one-shot claim token for an UPSTREAM service (plex's plex.tv claim). Same wave as the
|
||||||
|
# app: it is read by the app's env at pod start, and an env secretKeyRef has no `optional`,
|
||||||
|
# so a Secret applied in a later wave would leave the pod in CreateContainerConfigError.
|
||||||
|
# Placed only when the file exists, so services without one are unaffected.
|
||||||
|
add(w, f"{name}-claim-secret.sops.yaml")
|
||||||
# forward-auth (Option B): the per-app oauth2-proxy sidecar's OIDC client + cookie secret.
|
# forward-auth (Option B): the per-app oauth2-proxy sidecar's OIDC client + cookie secret.
|
||||||
# Same wave as the app (the sidecar reads it at pod start); placed only when the file exists,
|
# Same wave as the app (the sidecar reads it at pod start); placed only when the file exists,
|
||||||
# so non-forward-auth apps are unaffected. Mirrors the bootstrap-secret naming convention.
|
# so non-forward-auth apps are unaffected. Mirrors the bootstrap-secret naming convention.
|
||||||
@@ -2548,7 +2557,8 @@ def _selftest() -> int:
|
|||||||
# 7a) plan_apps_waves: sidecars inherit their unit's wave; not-onboarded SDs are skipped;
|
# 7a) plan_apps_waves: sidecars inherit their unit's wave; not-onboarded SDs are skipped;
|
||||||
# EXTRA_UNITS anchor strictly above; a file absent from the tree is never placed.
|
# EXTRA_UNITS anchor strictly above; a file absent from the tree is never placed.
|
||||||
present = {
|
present = {
|
||||||
"a.yaml", "svc.yaml", "svc-bootstrap-secret.sops.yaml",
|
"a.yaml", "a-claim-secret.sops.yaml",
|
||||||
|
"svc.yaml", "svc-bootstrap-secret.sops.yaml",
|
||||||
"svc-db.yaml", "svc-db-app.sops.yaml", "svc-db-backup-credentials.sops.yaml",
|
"svc-db.yaml", "svc-db-app.sops.yaml", "svc-db-backup-credentials.sops.yaml",
|
||||||
"svc-db-backup-credentials-prod.sops.yaml",
|
"svc-db-backup-credentials-prod.sops.yaml",
|
||||||
"gitea.yaml",
|
"gitea.yaml",
|
||||||
@@ -2559,6 +2569,10 @@ def _selftest() -> int:
|
|||||||
plan = plan_apps_waves(plan_sds, present)
|
plan = plan_apps_waves(plan_sds, present)
|
||||||
flat = {f: wv for wv, fs in plan.items() for f in fs}
|
flat = {f: wv for wv, fs in plan.items() for f in fs}
|
||||||
check(flat.get("a.yaml") == 0, "leaf app must land in wave 0")
|
check(flat.get("a.yaml") == 0, "leaf app must land in wave 0")
|
||||||
|
check(flat.get("a-claim-secret.sops.yaml") == 0,
|
||||||
|
"a `<svc>-claim-secret.sops.yaml` (plex's plex.tv claim) must be placed in the SAME wave "
|
||||||
|
"as its app — an env secretKeyRef has no `optional`, so a later wave means "
|
||||||
|
"CreateContainerConfigError")
|
||||||
check(flat.get("svc-db.yaml") == 0 and flat.get("svc-db-app.sops.yaml") == 0
|
check(flat.get("svc-db.yaml") == 0 and flat.get("svc-db-app.sops.yaml") == 0
|
||||||
and flat.get("svc-db-backup-credentials.sops.yaml") == 0
|
and flat.get("svc-db-backup-credentials.sops.yaml") == 0
|
||||||
and flat.get("svc-db-backup-credentials-prod.sops.yaml") == 0,
|
and flat.get("svc-db-backup-credentials-prod.sops.yaml") == 0,
|
||||||
|
|||||||
+26
-3
@@ -54,6 +54,29 @@ commands.
|
|||||||
4. Add the human reviewer as a repo collaborator, then raise `required_approvals` to 1.
|
4. Add the human reviewer as a repo collaborator, then raise `required_approvals` to 1.
|
||||||
|
|
||||||
> **Do not do step 3 before step 1.** A required status check with no runner leaves every
|
> **Do not do step 3 before step 1.** A required status check with no runner leaves every
|
||||||
> PR permanently pending — including the PR that would fix it. The same applies to
|
> PR permanently pending — including the PR that would fix it. That half is measured: PR #1
|
||||||
> `gate.yml`'s `conformance` job: it is `if: false` and never runs, yet still publishes a
|
> sat `pending` for 30.6 minutes across 122 polls with no runner registered, and never
|
||||||
> `gate / conformance (pull_request)` context. Never list that one as required.
|
> reported.
|
||||||
|
|
||||||
|
### Measured, once the runner came up (2026-09-29)
|
||||||
|
|
||||||
|
The first CI jobs this repo ever ran, on PR #1 and PR #2:
|
||||||
|
|
||||||
|
| Context | Result | Time |
|
||||||
|
|---|---|---|
|
||||||
|
| `gate / gate (pull_request)` | success | 1m58s (PR #1), 1m34s (PR #2) |
|
||||||
|
| `gate / gate (push)` | success | 1m32s (PR #2 only) |
|
||||||
|
| `gate / conformance (pull_request)` | **success — "Has been skipped"** | — |
|
||||||
|
|
||||||
|
**Require `gate / gate (pull_request)`. Do NOT require either `conformance` context —
|
||||||
|
but not for the reason an earlier draft of this file gave.** A skipped Gitea Actions job
|
||||||
|
does not hang a status check: Gitea reports it as **`success`** with the description
|
||||||
|
"Has been skipped". So requiring it would not deadlock anything — it would pass
|
||||||
|
*vacuously*, every time, forever, while nothing ran. Under HARD RULE 4 that is worse than
|
||||||
|
a deadlock: a deadlock is loud, a permanently-green check that verifies nothing is
|
||||||
|
exactly the fail-open shape this repo keeps finding.
|
||||||
|
|
||||||
|
The `(push)` contexts appear only for branches matching `gate.yml`'s push trigger
|
||||||
|
(`feature/**`, `fix/**`, `docs/**`, `chore/**`). `feat/**` is **not** in that list, which
|
||||||
|
is why PR #1's `feat/` branch produced only `pull_request` contexts. Require only the
|
||||||
|
`pull_request` context, or PRs from non-matching prefixes will never satisfy the rule.
|
||||||
Executable
+93
@@ -0,0 +1,93 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# plex-claim-ship.sh — run this IMMEDIATELY AFTER editing the SOPS secret with a fresh
|
||||||
|
# plex.tv claim token. It drives the whole CI/CD loop end to end and prints a running
|
||||||
|
# stopwatch so you can see the budget being spent.
|
||||||
|
#
|
||||||
|
# branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verify
|
||||||
|
#
|
||||||
|
# It never sees, prints, or handles the token: the file is already encrypted by sops.
|
||||||
|
set -euo pipefail
|
||||||
|
HOST=gitea.home.keithjackson.dev
|
||||||
|
OWNER=Home-Network
|
||||||
|
REPO=home-network-k3s-impl
|
||||||
|
SECRET=apps/wave-0/plex-claim-secret.sops.yaml
|
||||||
|
T0=$(date +%s)
|
||||||
|
el() { printf '\n[T+%02d:%02d] %s\n' $(( ($(date +%s)-T0)/60 )) $(( ($(date +%s)-T0)%60 )) "$1"; }
|
||||||
|
|
||||||
|
cd "$(git rev-parse --show-toplevel)"
|
||||||
|
|
||||||
|
# --- preflight: refuse to ship a file that is not what we think it is -------------
|
||||||
|
git diff --quiet -- "$SECRET" && { echo "ERROR: $SECRET is unchanged. Edit it first:"; \
|
||||||
|
echo " infra/devcontainer/dev.sh run sops $SECRET"; exit 1; }
|
||||||
|
grep -q 'PLEX_CLAIM: ENC\[' "$SECRET" || { echo "ERROR: $SECRET is not encrypted. REFUSING."; exit 1; }
|
||||||
|
[ -z "$(git status --porcelain | grep -v " $SECRET\$")" ] || echo "NOTE: other files are dirty; only $SECRET will be committed."
|
||||||
|
|
||||||
|
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||||
|
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||||
|
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
||||||
|
|
||||||
|
# feat/ prefix ON PURPOSE. act_runner capacity is 1, and gate.yml's PUSH trigger covers
|
||||||
|
# feature/** fix/** docs/** chore/** but NOT feat/** — so a chore/ branch would fire a
|
||||||
|
# second, redundant run that SERIALISES ahead of the PR run and doubles the wall clock.
|
||||||
|
BRANCH="feat/plex-claim-token-$(date +%H%M%S)"
|
||||||
|
|
||||||
|
el "committing on $BRANCH"
|
||||||
|
git checkout -q -b "$BRANCH"
|
||||||
|
git add "$SECRET"
|
||||||
|
git commit -q -m 'chore(plex): fresh plex.tv claim token
|
||||||
|
|
||||||
|
Injected inside the plex.tv claim window per runbooks/plex-claim.md.'
|
||||||
|
SHA=$(git rev-parse HEAD)
|
||||||
|
|
||||||
|
el "pushing"
|
||||||
|
git push -q -u origin "$BRANCH"
|
||||||
|
|
||||||
|
el "opening PR"
|
||||||
|
PR=$(jq -n --arg h "$BRANCH" --arg b main --arg t 'chore(plex): fresh plex.tv claim token' \
|
||||||
|
'{head:$h,base:$b,title:$t,body:"Token injection inside the claim window. Wiring already reviewed in #1."}' \
|
||||||
|
| api -X POST -H 'Content-Type: application/json' --data @- \
|
||||||
|
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls" | jq -r '.number')
|
||||||
|
[ -n "$PR" ] && [ "$PR" != "null" ] || { echo "PR creation failed"; exit 1; }
|
||||||
|
echo " PR #$PR"
|
||||||
|
|
||||||
|
el "waiting for the gate (expect ~1m40s)"
|
||||||
|
for i in $(seq 1 60); do
|
||||||
|
st=$(api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/commits/${SHA}/status" | jq -r '.state')
|
||||||
|
printf '\r gate=%s [T+%ss] ' "$st" "$(( $(date +%s)-T0 ))"
|
||||||
|
case "$st" in success) echo; break;; failure|error) echo; echo "GATE FAILED — stopping."; exit 1;; esac
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
|
[ "$st" = "success" ] || { echo; echo "gate did not finish in time"; exit 1; }
|
||||||
|
|
||||||
|
el "merging PR #$PR"
|
||||||
|
api -X POST -H 'Content-Type: application/json' -d '{"Do":"merge"}' \
|
||||||
|
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls/${PR}/merge" >/dev/null
|
||||||
|
git checkout -q main && git pull -q --ff-only origin main
|
||||||
|
|
||||||
|
el "forcing the Flux reconcile (--with-source, or it applies the OLD revision)"
|
||||||
|
./infra/devcontainer/dev.sh run bash -c \
|
||||||
|
'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-0 --with-source'
|
||||||
|
|
||||||
|
# MANDATORY here. Only the Secret ciphertext changed, so the Deployment pod template is
|
||||||
|
# byte-identical and NOTHING restarts on its own. PLEX_CLAIM is read once, at container
|
||||||
|
# start. Skipping this is failure mode #2 in runbooks/plex-claim.md.
|
||||||
|
el "rolling the pod (mandatory — a changed Secret restarts nothing)"
|
||||||
|
./infra/devcontainer/dev.sh run bash -c \
|
||||||
|
'kubectl --context k3s-prod -n plex rollout restart deployment/plex && \
|
||||||
|
kubectl --context k3s-prod -n plex rollout status deployment/plex --timeout=120s'
|
||||||
|
|
||||||
|
el "verdict"
|
||||||
|
./infra/devcontainer/dev.sh run bash -c \
|
||||||
|
'kubectl --context k3s-prod -n plex logs deploy/plex --since=3m | grep -i -e claim -e token || true'
|
||||||
|
cat <<'NOTE'
|
||||||
|
|
||||||
|
"Attempting to obtain server token..." + "Token obtained successfully" -> CLAIMED
|
||||||
|
"Attempting..." with no success line -> token expired; retry, nothing is broken
|
||||||
|
neither line -> already claimed, or env did not reach the container
|
||||||
|
|
||||||
|
Confirm it persisted (prints the attribute NAME only, never its value):
|
||||||
|
|
||||||
|
infra/devcontainer/dev.sh run bash -c \
|
||||||
|
'kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c \
|
||||||
|
"grep -o PlexOnlineToken= \"/config/Library/Application Support/Plex Media Server/Preferences.xml\" | head -1"'
|
||||||
|
NOTE
|
||||||
+10
-2
@@ -6,7 +6,7 @@ Every alert rule in this platform carries an annotation of the form
|
|||||||
|
|
||||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"
|
||||||
|
|
||||||
46 of them, over the 35 names below. Until this directory existed, that path returned **HTTP 404** on the
|
46 of them, over 35 names. Until this directory existed, that path returned **HTTP 404** on the
|
||||||
live cluster: every alert that reached a human handed them a dead link at the exact moment they needed
|
live cluster: every alert that reached a human handed them a dead link at the exact moment they needed
|
||||||
help. These pages are what makes the link resolve.
|
help. These pages are what makes the link resolve.
|
||||||
|
|
||||||
@@ -33,7 +33,8 @@ negative is how a CLI quirk becomes real data loss. Write `backups.velero.io`,
|
|||||||
`backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`,
|
`backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`,
|
||||||
`volumes.longhorn.io`, `nodes.longhorn.io`.
|
`volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
`git push origin main` **is** the deploy; there is no other apply step. Durable changes go through git.
|
Merging a PR into `main` **is** the deploy; there is no other apply step. `main` is protected, so a
|
||||||
|
direct push is rejected — durable changes go through a branch, a PR, and a green gate.
|
||||||
|
|
||||||
## Alerts you should not trust the silence of (measured 2026-09-13)
|
## Alerts you should not trust the silence of (measured 2026-09-13)
|
||||||
|
|
||||||
@@ -108,6 +109,13 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
|
|||||||
- **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
|
- **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
|
||||||
every green signal missed.
|
every green signal missed.
|
||||||
|
|
||||||
|
### Operator procedures (no alert links here)
|
||||||
|
- **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only
|
||||||
|
applies after a `/config` wipe. The claim token is short-lived — read the countdown on
|
||||||
|
plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
|
||||||
|
token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
|
||||||
|
Includes why a changed Secret does not restart anything, and why an expired token is harmless.
|
||||||
|
|
||||||
### Virtualisation
|
### Virtualisation
|
||||||
- **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.
|
- **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.
|
||||||
- **`kubevirt-launch-failed`** — a VMI reached the Failed phase. The three k3s-specific launch traps.
|
- **`kubevirt-launch-failed`** — a VMI reached the Failed phase. The three k3s-specific launch traps.
|
||||||
|
|||||||
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -78,8 +78,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -55,8 +55,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -63,8 +63,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -87,8 +87,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -69,8 +69,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -71,8 +71,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -68,8 +68,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -7,7 +7,7 @@
|
|||||||
## What this means here
|
## What this means here
|
||||||
|
|
||||||
Something in the GitOps chain is stuck at Ready=False. On this cluster that is more consequential than
|
Something in the GitOps chain is stuck at Ready=False. On this cluster that is more consequential than
|
||||||
it sounds, because **`git push origin main` IS the deploy** — there is no other apply step. A stuck
|
it sounds, because **merging a PR into `main` IS the deploy** — there is no other apply step. A stuck
|
||||||
Kustomization means git and the cluster have diverged and nothing will close the gap.
|
Kustomization means git and the cluster have diverged and nothing will close the gap.
|
||||||
|
|
||||||
The chain has **14 tiers**: `flux-system` → `cilium` → `platform` → `platform-config` →
|
The chain has **14 tiers**: `flux-system` → `cilium` → `platform` → `platform-config` →
|
||||||
@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -58,8 +58,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -87,8 +87,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -67,8 +67,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -59,8 +59,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -57,8 +57,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -81,6 +81,11 @@ configMapGenerator:
|
|||||||
- node-io=node-io.md
|
- node-io=node-io.md
|
||||||
- node-memory=node-memory.md
|
- node-memory=node-memory.md
|
||||||
- node-memory-swap=node-memory-swap.md
|
- node-memory-swap=node-memory-swap.md
|
||||||
|
# Operator procedure, not an alert page (no alert links to it — the
|
||||||
|
# alert-runbooks-resolvable check treats that as advisory). It is served
|
||||||
|
# because the plex.tv claim window is short and is not a thing to go looking
|
||||||
|
# for in git: /runbooks/plex-claim has to resolve the moment it is needed.
|
||||||
|
- plex-claim=plex-claim.md
|
||||||
- pvc-capacity=pvc-capacity.md
|
- pvc-capacity=pvc-capacity.md
|
||||||
- targets-down=targets-down.md
|
- targets-down=targets-down.md
|
||||||
- update-rollback=update-rollback.md
|
- update-rollback=update-rollback.md
|
||||||
|
|||||||
@@ -83,8 +83,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -74,8 +74,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -75,8 +75,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -52,8 +52,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -49,8 +49,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -56,8 +56,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -56,8 +56,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
+4
-2
@@ -227,8 +227,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -52,8 +52,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,189 @@
|
|||||||
|
# plex-claim — claiming the Plex Media Server inside the token window
|
||||||
|
|
||||||
|
**Alert:** none. This is an **operator procedure**, not an incident page — it is indexed here because
|
||||||
|
it is the page you need at the moment you need it, with a stopwatch running.
|
||||||
|
**Files:** `catalog/plex.yaml` (env `PLEX_CLAIM`), `apps/wave-0/plex-claim-secret.sops.yaml`
|
||||||
|
(the SOPS Secret `plex-claim`, key `PLEX_CLAIM`), `apps/wave-0/plex.yaml` (the rendered golden).
|
||||||
|
|
||||||
|
## What this means here
|
||||||
|
|
||||||
|
Plex will not let you **create libraries** over the mounted bulk media share while the server is
|
||||||
|
**unclaimed** — the 8.33 T NFS tree at `/media` is there, readable, and unusable. Claiming binds the
|
||||||
|
server to a plex.tv account. It is a **one-shot** operation: the token is exchanged once for a durable
|
||||||
|
`PlexOnlineToken` written into `Preferences.xml` on the `plex-config` Longhorn PVC, and from then on
|
||||||
|
the `PLEX_CLAIM` env is **ignored on every start** (upstream gate:
|
||||||
|
`[ ! -z "${PLEX_CLAIM}" ] && [ -z "${token}" ]` in `root/etc/cont-init.d/40-plex-first-run`).
|
||||||
|
|
||||||
|
**A `https://plex.tv/claim` token is short-lived — minutes, not hours.** Read the countdown the
|
||||||
|
claim page shows you; do not budget against a number written here. That is shorter than PR → gate →
|
||||||
|
merge → Flux's 10-minute reconcile interval → pod roll if you let those run on their own timers,
|
||||||
|
which is why this page exists: the loop is scripted and the reconcile is **forced** rather than
|
||||||
|
waited for. Measured end to end on 2026-09-29: **~2m45s**.
|
||||||
|
|
||||||
|
**An expired or wrong token does not break anything.** Verified against the image's own first-run
|
||||||
|
script (and the copy inside `plexinc/pms-docker:latest`): a failed exchange gets a 403 with an empty
|
||||||
|
body, `curl`'s status is never checked, the token `sed` yields empty, the `if` does not fire, there is
|
||||||
|
no `else` and no `exit 1`, and the script (no `set -e`) exits 0. The pod comes up **unclaimed and
|
||||||
|
serving**. So a burnt token costs you one more attempt, not an outage — and the repo deliberately
|
||||||
|
ships a **placeholder** in that Secret for exactly this reason (the env `secretKeyRef` has no
|
||||||
|
`optional`, so the Secret must always exist even when its value is useless).
|
||||||
|
|
||||||
|
## Running the commands in this runbook
|
||||||
|
|
||||||
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
||||||
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that
|
||||||
|
hang is not the cluster being down. Every command below runs as:
|
||||||
|
|
||||||
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
||||||
|
|
||||||
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
||||||
|
picks a current-context per invocation, and that default has already changed once (k3s-dev until dev
|
||||||
|
was decommissioned on 2026-09-13, k3s-prod now). Do not run two `dev.sh` invocations at once.
|
||||||
|
|
||||||
|
Mutating `kubectl` and `flux reconcile` are blocked for agents here. **This whole page is operator
|
||||||
|
work** — the reconcile and the pod roll below are deliberate operator actions, and the durable change
|
||||||
|
is still a git change.
|
||||||
|
|
||||||
|
## First commands
|
||||||
|
|
||||||
|
**Before spending a token, find out whether the server is already claimed.** This prints only the
|
||||||
|
attribute NAME, never its value — do not widen it to print the token:
|
||||||
|
|
||||||
|
```
|
||||||
|
infra/devcontainer/dev.sh run bash -c \
|
||||||
|
'kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c \
|
||||||
|
"grep -o PlexOnlineToken= \"/config/Library/Application Support/Plex Media Server/Preferences.xml\" | head -1"'
|
||||||
|
```
|
||||||
|
|
||||||
|
`PlexOnlineToken=` printed ⇒ **already claimed**; stop, there is nothing to do and no token to burn.
|
||||||
|
Empty output ⇒ unclaimed, continue.
|
||||||
|
|
||||||
|
Is the workload healthy enough to claim at all, and is the wave reconciling?
|
||||||
|
|
||||||
|
```
|
||||||
|
infra/devcontainer/dev.sh run bash -c \
|
||||||
|
'kubectl --context k3s-prod -n plex get deploy,pod,pvc'
|
||||||
|
infra/devcontainer/dev.sh run bash -c \
|
||||||
|
'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0'
|
||||||
|
```
|
||||||
|
|
||||||
|
## The procedure
|
||||||
|
|
||||||
|
> **STATUS: this server was CLAIMED on 2026-09-29** and `PlexOnlineToken` is present in
|
||||||
|
> `/config` on the `plex-config` PVC. The upstream exchange is gated on that pref being
|
||||||
|
> *absent*, so `PLEX_CLAIM` is now a **no-op on every start** whatever it holds. You only need
|
||||||
|
> this page again after a `/config` wipe or a rebuild onto an empty PVC.
|
||||||
|
|
||||||
|
**On the clock.** The token is short-lived — **read the countdown that plex.tv/claim shows you**
|
||||||
|
rather than trusting a number written here. (Earlier revisions of this page asserted a "~4-minute
|
||||||
|
TTL" in four places with no citation; upstream `pms-docker` documents no expiry at all. The page
|
||||||
|
is the only authority, and it is on screen at T+0.)
|
||||||
|
|
||||||
|
**A burnt token costs nothing.** An expired token gets a 403 with an empty body, `curl`'s status is
|
||||||
|
never checked, the `if` does not fire, there is no `else` and no `exit 1`, and the script (no
|
||||||
|
`set -e`) exits 0. The pod comes up unclaimed and serving. So retry freely.
|
||||||
|
|
||||||
|
### One command
|
||||||
|
|
||||||
|
1. **T+0 — get the token.** Sign in to plex.tv as the account that should OWN this server, open
|
||||||
|
<https://plex.tv/claim>, copy the `claim-…` string.
|
||||||
|
|
||||||
|
2. **Put it in the Secret** — in place, encrypted, never touching a plaintext file:
|
||||||
|
|
||||||
|
```
|
||||||
|
infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Ship it.** This drives the whole delivery loop and prints a running stopwatch:
|
||||||
|
|
||||||
|
```
|
||||||
|
./infra/plex/claim-ship.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
branch → commit → push → PR → gate → merge → `flux reconcile --with-source` → `rollout restart`
|
||||||
|
→ verdict. It refuses to run if the file is unchanged or not encrypted.
|
||||||
|
|
||||||
|
**Measured end to end, 2026-09-29: ~2m45s.** Gate ~1m40s, merge + reconcile ~25s, pod roll and
|
||||||
|
container start ~30s.
|
||||||
|
|
||||||
|
### Why it is a script and not six pasted commands
|
||||||
|
|
||||||
|
`main` is a **protected branch** — the direct push this page used to instruct is now rejected — so
|
||||||
|
the deploy is a PR merged after a green gate. Three of those steps are easy to get wrong under time
|
||||||
|
pressure, and two are silent when skipped:
|
||||||
|
|
||||||
|
- **`--with-source` is load-bearing.** Without it the Kustomization applies the previously-fetched
|
||||||
|
revision, not your commit.
|
||||||
|
- **The pod roll is mandatory.** Only the Secret's ciphertext changes, so the Deployment's pod
|
||||||
|
template is byte-identical and *nothing restarts on its own*. `PLEX_CLAIM` is read once, at
|
||||||
|
container start. Skip this and the new Secret sits in the cluster while the running pod keeps the
|
||||||
|
old value forever.
|
||||||
|
- **The branch is named `feat/…` on purpose.** `act_runner` capacity is 1, and `gate.yml`'s push
|
||||||
|
trigger covers `feature/** fix/** docs/** chore/**` but *not* `feat/**`. A `chore/` branch fires a
|
||||||
|
second redundant run that serialises ahead of the PR run and roughly doubles the wall clock.
|
||||||
|
|
||||||
|
### Reading the verdict
|
||||||
|
|
||||||
|
* `Attempting to obtain server token from claim token` **and** `Token obtained successfully`
|
||||||
|
⇒ **claimed.** Re-run the First-commands `PlexOnlineToken=` check to confirm it persisted, then
|
||||||
|
create the libraries over `/media/movies`, `/media/tv`, `/media/photos`.
|
||||||
|
* `Attempting…` with **no** success line ⇒ the token was expired or invalid. Nothing is broken; go
|
||||||
|
back to step 1 with a new one.
|
||||||
|
* **Neither line** ⇒ the exchange was skipped: `/config` already had a `PlexOnlineToken` (already
|
||||||
|
claimed — the expected case now), or the env did not reach the container. Check by NAME only:
|
||||||
|
|
||||||
|
```
|
||||||
|
infra/devcontainer/dev.sh run bash -c \
|
||||||
|
'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
|
||||||
|
-o jsonpath="{.items[0].spec.containers[0].env[*].name}"'
|
||||||
|
```
|
||||||
|
|
||||||
|
## Likely causes, most likely first
|
||||||
|
|
||||||
|
When the claim does not take, in order of how often it is the answer here:
|
||||||
|
|
||||||
|
1. **The token expired** — you exceeded the countdown, usually by reviewing the PR instead of
|
||||||
|
merging it the moment the gate went green. Symptom: the "Attempting…" line with no success line.
|
||||||
|
Remedy: a new token, and let `claim-ship.sh` drive the loop rather than pasting steps.
|
||||||
|
2. **The pod was never restarted** (step 5 skipped). Symptom: no claim log lines at all from a pod
|
||||||
|
whose `AGE` predates your push. The Secret is correct in git and in the cluster and the running
|
||||||
|
container still holds the placeholder.
|
||||||
|
3. **Flux had not applied the commit yet** when you rolled the pod — step 5 run before step 4
|
||||||
|
finished, so the new pod read the OLD Secret. Check
|
||||||
|
`kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0 -o jsonpath='{.status.lastAppliedRevision}'`
|
||||||
|
against your commit, then roll again.
|
||||||
|
4. **Already claimed** — the exchange is gated on `PlexOnlineToken` being absent from `/config`, so a
|
||||||
|
server claimed earlier ignores every later token silently. This is the benign case.
|
||||||
|
5. **`plex.tv` unreachable.** The upstream `curl` has no `--max-time`/`--connect-timeout`, so a stalled
|
||||||
|
connection can leave the container sitting in cont-init with **nothing listening on 32400** — the
|
||||||
|
one genuine hang risk in this path. Symptom: pod Running but never Ready, log stops after
|
||||||
|
"Attempting…". Check egress (`plex` is `egress: internet`) rather than re-rolling repeatedly.
|
||||||
|
6. **Wrong plex.tv account.** The claim succeeds and binds the server to whoever was signed in at
|
||||||
|
step 1. Unclaiming is a plex.tv-side operation on that account — see "escalate or stop".
|
||||||
|
|
||||||
|
## When to escalate or stop
|
||||||
|
|
||||||
|
- **Stop after two failed token attempts** and fix the timing, not the token. The third rushed attempt
|
||||||
|
is how people start editing live Secrets by hand, which leaves the cluster holding a value that is
|
||||||
|
in no git commit — precisely the drift `qbittorrent-bootstrap` had to be recovered from.
|
||||||
|
- **Never `kubectl edit`/`patch` the `plex-claim` Secret in the cluster, and never
|
||||||
|
`kubectl create secret` it.** Everything durable goes through the SOPS file; a live-only Secret is
|
||||||
|
reverted (or adopted with the wrong value) on the next reconcile.
|
||||||
|
- **Never `sops -d` this file into a terminal, a log, or a shared screen**, and never widen the
|
||||||
|
`PlexOnlineToken=` check into one that prints the value. The token is a plex.tv account credential.
|
||||||
|
- **If the server got claimed to the wrong account, stop and get the operator.** Reversing it means
|
||||||
|
removing the server from that account on plex.tv and clearing `PlexOnlineToken` from
|
||||||
|
`Preferences.xml` on the PVC — a destructive edit of app state on the replicated tier, not a git
|
||||||
|
change.
|
||||||
|
- **Do not delete the `plex-config` PVC to "start clean".** It holds the library database, metadata
|
||||||
|
and watch state (`policy: daily-offcluster`), and it is expensive to rebuild. The claim is a
|
||||||
|
two-line pref, not a reason to lose the DB.
|
||||||
|
- After a successful claim, **leave the expired token in the SOPS file**. It is inert (the exchange is
|
||||||
|
skipped once claimed) and the Secret must keep existing, because the env `secretKeyRef` has no
|
||||||
|
`optional` — deleting it would put the pod into `CreateContainerConfigError`.
|
||||||
|
|
||||||
|
---
|
||||||
|
Source of truth: `runbooks/plex-claim.md`. Index: `/runbooks/`
|
||||||
|
Upstream behaviour cited here is from `plexinc/pms-docker` branch `master`,
|
||||||
|
`root/etc/cont-init.d/40-plex-first-run` (s6-overlay v2), cross-checked against the same file inside
|
||||||
|
the running `plexinc/pms-docker:latest` image.
|
||||||
@@ -58,8 +58,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -60,8 +60,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -54,8 +54,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
@@ -66,8 +66,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
|||||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||||
|
|
||||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||||
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||||
|
and it says so.
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user