Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e72050c690 | ||
|
|
27d0ef3324 | ||
|
|
bbd4f682a6 | ||
|
|
0391139240 | ||
|
|
e9f441b5ce | ||
|
|
4fd75ccdaa |
No files matched your search
+28
-14
@@ -40,9 +40,15 @@ and ADR-0011: **CI holds no broad kubeconfig.** L2/L3 run as an in-cluster **Cha
|
|||||||
*collects* its artifacts, using a narrow trigger credential referenced **by secret name only**
|
*collects* its artifacts, using a narrow trigger credential referenced **by secret name only**
|
||||||
(`secrets.CONFORMANCE_TRIGGER_KUBECONFIG`) — never a plaintext secret, never a cluster-admin kubeconfig.
|
(`secrets.CONFORMANCE_TRIGGER_KUBECONFIG`) — never a plaintext secret, never a cluster-admin kubeconfig.
|
||||||
|
|
||||||
## Honest SKIP — never fake green (Phase 0)
|
## Honest SKIP — never fake green
|
||||||
|
|
||||||
**No cluster exists in Phase 0.** The gate is deliberately built so unavailable checks **SKIP loudly,
|
**NOTE (2026-09-29):** this section is written in the past's voice — "Phase 0", "no cluster
|
||||||
|
exists". **A cluster has existed for weeks** and Flux has been reconciling against it. The
|
||||||
|
skip logic below is still correct and still honest, but read it as describing a **live
|
||||||
|
coverage gap** (L2/L3 is not wired into CI) rather than an expected phase. Its being
|
||||||
|
phrased as a phase is part of why the gap went unexamined.
|
||||||
|
|
||||||
|
Originally: The gate is deliberately built so unavailable checks **SKIP loudly,
|
||||||
they do not pass**:
|
they do not pass**:
|
||||||
|
|
||||||
- The `conformance` job is `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}` — with no cluster it is
|
- The `conformance` job is `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}` — with no cluster it is
|
||||||
@@ -77,21 +83,29 @@ Both dirs are git-ignored (see [`.gitignore`](../../.gitignore)).
|
|||||||
## Triggers & branch model (CONTRIBUTING.md)
|
## Triggers & branch model (CONTRIBUTING.md)
|
||||||
|
|
||||||
- `pull_request` → `main`: **PRs are the enforced merge gate into `main`.**
|
- `pull_request` → `main`: **PRs are the enforced merge gate into `main`.**
|
||||||
- `push` → `feature/**`, `fix/**`, `docs/**`, `chore/**`: fast feedback on the branch. **No work on `main`.**
|
- `push` → `main` only. The per-branch prefix list was removed: `act_runner` runs at capacity 1,
|
||||||
|
so a branch push fires a redundant run that serialises *ahead* of the PR run for the same tree.
|
||||||
|
**No work on `main`** — and since 2026-09-29 that is enforced, not conventional: `main` is a
|
||||||
|
protected branch and a direct push is rejected by the pre-receive hook (verified, not assumed).
|
||||||
|
|
||||||
## Transition: generic-CI-now → Gitea-Actions-after-Gitea-is-up
|
## Transition: COMPLETE (2026-09-29)
|
||||||
|
|
||||||
Per [`CONTRIBUTING.md`](../../../home-network-k3s/CONTRIBUTING.md) and
|
This section used to describe a migration as pending. It has happened. Gitea is live at
|
||||||
[`docs/05` §"local bootstrap → self-hosted Gitea"](../../../home-network-k3s/docs/05-gitops-and-delivery.md):
|
`gitea.home.keithjackson.dev`, the runner `incluster-dind` is registered and online, and
|
||||||
|
`gate.yml` runs on every PR.
|
||||||
|
|
||||||
1. **Now (interim):** there is **no remote git host** on this network. This workflow file is dormant —
|
**What took a month to notice:** the migration was recorded as done on 2026-08-31 without
|
||||||
but the **identical gate runs locally today** via `make -C ci gate` (R-TEST-02). Development is
|
anyone checking the platform. In reality `main` had **no branch protection** and **zero
|
||||||
local feature-branch → `main`, the gate enforced by convention + the local runner.
|
runners were registered** until 2026-09-29. Every "the gate enforces this" statement in
|
||||||
2. **Gitea comes online early** (one of the first services onboarded) at `gitea.home.keithjackson.dev`.
|
between rested on an assertion nobody verified. PR #1 — the first pull request this
|
||||||
3. **CI moves to Gitea Actions:** this workflow activates unchanged. The only operational steps are
|
repository ever had — sat `pending` for 30.6 minutes because nothing existed to run it.
|
||||||
enabling a Gitea Actions runner and setting `vars.CLUSTER_AVAILABLE=true` +
|
|
||||||
`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` once a dev/test cluster exists. PRs then become the
|
So: **do not read this file, or the DECISIONS-LOG, as evidence about the live platform.**
|
||||||
**platform-enforced** merge gate — the same no-work-in-main rule, now enforced by machine.
|
Query the API. `infra/gitea/` holds the scripts and the measured state.
|
||||||
|
|
||||||
|
Still genuinely outstanding: `vars.CLUSTER_AVAILABLE` and
|
||||||
|
`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` are unset, so L2/L3 conformance does not run in
|
||||||
|
CI. That is a real coverage gap, not a phase.
|
||||||
|
|
||||||
## Configuration (set in Gitea; referenced by name only)
|
## Configuration (set in Gitea; referenced by name only)
|
||||||
|
|
||||||
|
|||||||
@@ -11,11 +11,14 @@
|
|||||||
# v
|
# v
|
||||||
# merge-gate decision [R-TEST-03: touched subset green]
|
# merge-gate decision [R-TEST-03: touched subset green]
|
||||||
#
|
#
|
||||||
# Transition (CONTRIBUTING.md / docs/05 §"local bootstrap -> self-hosted Gitea"):
|
# STATUS (corrected 2026-09-29): Gitea IS hosted and Gitea Actions IS enabled; the runner
|
||||||
# Today there is no Gitea host, so this file is dormant. The SAME gate is runnable
|
# `incluster-dind` is online and gate.yml is the live PR gate. The text that used to sit here
|
||||||
# now via `make -C ci gate` (R-TEST-02). Once Gitea is self-hosted and Gitea
|
# ("Today there is no Gitea host, so this file is dormant") was stale by weeks -- it was written
|
||||||
# Actions is enabled, this workflow becomes the ENFORCED PR merge gate into main —
|
# as a plan and never revisited once the plan completed.
|
||||||
# no code change required, only turning on the runner and setting CLUSTER_AVAILABLE.
|
#
|
||||||
|
# THIS file is still dormant, but for the reason in the DORMANT block below -- its spec checkout
|
||||||
|
# and L2/L3 seams are unwired -- NOT because there is no host. Do not read a stale plan as a
|
||||||
|
# statement about the live platform; query the API instead (see infra/gitea/).
|
||||||
#
|
#
|
||||||
# CI holds NO broad kubeconfig. L2/L3 run as an in-cluster Chainsaw Job whose own
|
# CI holds NO broad kubeconfig. L2/L3 run as an in-cluster Chainsaw Job whose own
|
||||||
# scoped ServiceAccount holds cluster access (docs/14, R-SEC-02); CI uses only a
|
# scoped ServiceAccount holds cluster access (docs/14, R-SEC-02); CI uses only a
|
||||||
|
|||||||
@@ -45,6 +45,35 @@ the toolbox gate yourself and quote its numbers. Never describe a PR as "green"
|
|||||||
strength of a check you have not seen report — and note that a **skipped** Gitea job
|
strength of a check you have not seen report — and note that a **skipped** Gitea job
|
||||||
reports as `success`, so a green tick is not by itself evidence that anything ran.
|
reports as `success`, so a green tick is not by itself evidence that anything ran.
|
||||||
|
|
||||||
|
## 2b. Never infer platform state from this repository
|
||||||
|
|
||||||
|
**This repo is not evidence about the live platform.** Neither is `DECISIONS-LOG.md`.
|
||||||
|
Both describe what someone believed at the time of writing, and both have been
|
||||||
|
confidently wrong for a month at a stretch.
|
||||||
|
|
||||||
|
On 2026-09-29 the log stated the merge gate was a required status check with a runner
|
||||||
|
online. The API said `branch_protections: []` and **zero** runners. Everything downstream
|
||||||
|
of that belief — "the gate enforces this", "your push would be rejected", "CI covers it" —
|
||||||
|
was wrong, and nothing in the tree could have revealed it, because the tree was the source
|
||||||
|
of the error.
|
||||||
|
|
||||||
|
**So, before asserting that any gate, protection, runner, secret, or workload is in a
|
||||||
|
given state: query the thing itself.**
|
||||||
|
|
||||||
|
| Claim about... | Check |
|
||||||
|
|---|---|
|
||||||
|
| branch protection / required checks | `GET /api/v1/repos/{owner}/{repo}/branches/main` |
|
||||||
|
| a runner exists | `GET /api/v1/admin/actions/runners` |
|
||||||
|
| CI actually ran | the commit's `/statuses` — and read the **description**; a *skipped* Gitea job reports `success` with "Has been skipped" |
|
||||||
|
| a workload / Secret / env | `kubectl --context k3s-prod ...` (read-only is fine for agents) |
|
||||||
|
| Flux applied your commit | compare `.status.lastAppliedRevision` against the GitRepository's artifact revision |
|
||||||
|
|
||||||
|
Quote the measured field, not the prose. If you cannot measure it, say so explicitly —
|
||||||
|
"unverified" is a legitimate report, and a far better one than a confident guess that
|
||||||
|
matches the docs.
|
||||||
|
|
||||||
|
**A green tick is not proof anything ran.** Read what actually executed.
|
||||||
|
|
||||||
## 3. Gate before you push
|
## 3. Gate before you push
|
||||||
|
|
||||||
./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
|
./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
|
||||||
|
|||||||
@@ -1192,6 +1192,13 @@ fresh agent should read this entry + `infra/PHASE1-STATUS.md` + `docs/plans/` an
|
|||||||
Platform-admins require TOTP MFA.
|
Platform-admins require TOTP MFA.
|
||||||
- **Gitea Actions CI is LIVE + enforcing**: `.gitea/workflows/gate.yml` runs `make -C ci gate` on push/PR to main;
|
- **Gitea Actions CI is LIVE + enforcing**: `.gitea/workflows/gate.yml` runs `make -C ci gate` on push/PR to main;
|
||||||
required status check; runner `incluster-dind` online (ns gitea-actions).
|
required status check; runner `incluster-dind` online (ns gitea-actions).
|
||||||
|
> **[CORRECTION, 2026-09-29 — THIS BULLET WAS FALSE WHEN WRITTEN AND STAYED FALSE FOR A MONTH.]**
|
||||||
|
> Measured against the live Gitea API on 2026-09-29: `branch_protections` returned `[]` (`main`
|
||||||
|
> `protected:false`, `enable_status_check:false`, `user_can_push:true`) and **zero** Actions runners
|
||||||
|
> were registered at repo, org or site level. The gate was neither enforced nor able to run. PR #1
|
||||||
|
> — the first pull request this repo ever had — was opened on 2026-09-29 and sat `pending` for
|
||||||
|
> 30.6 minutes with nothing to execute it. Both halves of this bullet were asserted, not verified.
|
||||||
|
> See the 2026-09-29 entries for the root cause and the fix.
|
||||||
|
|
||||||
### What changed this session (durable)
|
### What changed this session (durable)
|
||||||
1. **SSO cluster-wide break→fix — root cause Cilium CVE-2025-30162.** The 1.16→1.20 security fix makes the shared
|
1. **SSO cluster-wide break→fix — root cause Cilium CVE-2025-30162.** The 1.16→1.20 security fix makes the shared
|
||||||
|
|||||||
@@ -17,6 +17,7 @@ resources:
|
|||||||
- pihole-dns.yaml
|
- pihole-dns.yaml
|
||||||
- pihole-oauth2-proxy-credentials.sops.yaml
|
- pihole-oauth2-proxy-credentials.sops.yaml
|
||||||
- pihole.yaml
|
- pihole.yaml
|
||||||
|
- plex-claim-secret.sops.yaml
|
||||||
- plex.yaml
|
- plex.yaml
|
||||||
- qbittorrent-bootstrap-secret.sops.yaml
|
- qbittorrent-bootstrap-secret.sops.yaml
|
||||||
- qbittorrent-oauth2-proxy-credentials.sops.yaml
|
- qbittorrent-oauth2-proxy-credentials.sops.yaml
|
||||||
|
|||||||
@@ -0,0 +1,77 @@
|
|||||||
|
# Plex plex.tv server-claim token (WS-APPS). SOPS-encrypted (age).
|
||||||
|
# Consumed by catalog/plex.yaml as env PLEX_CLAIM via valueFrom.secretKeyRef.
|
||||||
|
#
|
||||||
|
# NOTE ON THESE COMMENTS: they carry NO dollar-brace expressions, deliberately. This file is applied
|
||||||
|
# by the apps-wave-0 Flux Kustomization, which runs envsubst over the built output, and kustomize
|
||||||
|
# preserves body comments into that stream (see conformance/tests/l0-static/flux-postbuild-
|
||||||
|
# substitution). A braced token quoted from the upstream shell script would be a live hazard here,
|
||||||
|
# not prose, so the upstream conditions below are described in words.
|
||||||
|
#
|
||||||
|
# THIS FILE NO LONGER HOLDS THE PLACEHOLDER. Corrected 2026-09-29: the server was CLAIMED on that
|
||||||
|
# date, and the value here is the SPENT token from that claim. It is inert and safe to leave --
|
||||||
|
# upstream gates the exchange on PlexOnlineToken being ABSENT from /config, and it is now present on
|
||||||
|
# the plex-config PVC, so this value is a NO-OP on every start whatever it holds. Deleting the Secret
|
||||||
|
# would NOT be safe (see below), which is why a spent token stays here rather than being blanked.
|
||||||
|
#
|
||||||
|
# The claim token is SHORT-LIVED -- minutes. Read the countdown that plex.tv/claim renders when it
|
||||||
|
# issues one; do NOT budget against a number written in this tree. Earlier revisions of this comment
|
||||||
|
# asserted a "~4-minute TTL" as established fact with no citation, and upstream plexinc/pms-docker
|
||||||
|
# documents no expiry at all. That uncited number was load-bearing for the whole procedure design.
|
||||||
|
#
|
||||||
|
# This file exists to make the Secret — and therefore the env reference — ALWAYS RESOLVABLE IN THE
|
||||||
|
# FLUX GRAPH, not to carry a working token. Re-claiming (only needed after a /config wipe) is
|
||||||
|
# scripted: infra/plex/claim-ship.sh, measured at ~2m45s end to end.
|
||||||
|
#
|
||||||
|
# WHY SHIPPING A PLACEHOLDER IS SAFE (verified against plexinc/pms-docker branch `master`, the
|
||||||
|
# s6-overlay v2 script root/etc/cont-init.d/40-plex-first-run, cross-checked byte-identical against
|
||||||
|
# the same file extracted from the running `plexinc/pms-docker:latest` image):
|
||||||
|
# - line 106 guards the claim exchange on TWO conditions: PLEX_CLAIM non-empty AND the
|
||||||
|
# PlexOnlineToken pref read out of /config empty. Once the server is claimed that block is
|
||||||
|
# skipped entirely, so whatever this Secret holds is a NO-OP on every later start. Upstream
|
||||||
|
# README: "If server is already logged in, this parameter is ignored."
|
||||||
|
# - on a still-UNCLAIMED /config, an invalid or expired token makes plex.tv answer 403 with an
|
||||||
|
# empty body. curl's exit status is never checked, the sed for the authentication-token element
|
||||||
|
# yields the empty string, and the success branch at line 120 simply does not fire. There is no
|
||||||
|
# else branch and no non-zero exit; the script has no `set -e` and ends by touching
|
||||||
|
# /.firstRunComplete and echoing, so it exits 0. The server comes up UNCLAIMED and serving.
|
||||||
|
# It does NOT crashloop.
|
||||||
|
# - the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS, whose default is 0 ("continue silently even
|
||||||
|
# if a cont-init script failed"), so even a hypothetical non-zero exit would not halt startup.
|
||||||
|
# A MISSING Secret would NOT be safe: the ServiceDefinition schema's valueFrom.secretKeyRef shape
|
||||||
|
# has no `optional` field, so an absent Secret is a hard CreateContainerConfigError. That is exactly
|
||||||
|
# why this file is committed and listed in apps/wave-0/kustomization.yaml instead of being created
|
||||||
|
# by hand at claim time.
|
||||||
|
#
|
||||||
|
# INJECTION / ROTATION (operator, not agents) — runbooks/plex-claim.md has the timed procedure:
|
||||||
|
# infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
|
||||||
|
# Never decrypt this file into a shell, a log, or a shared terminal.
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Secret
|
||||||
|
metadata:
|
||||||
|
name: plex-claim
|
||||||
|
namespace: plex
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/name: plex
|
||||||
|
type: Opaque
|
||||||
|
stringData:
|
||||||
|
PLEX_CLAIM: ENC[AES256_GCM,data:nLmzeUBWRhRjUakrGiGzQpQuqZNcIYvRH6M=,iv:z0on/dArsNejNLrp3V1nSiNLFzKcREhPuMO1634CwQw=,tag:Ih23CT1JYna/5lfYQUb1Dw==,type:str]
|
||||||
|
sops:
|
||||||
|
kms: []
|
||||||
|
gcp_kms: []
|
||||||
|
azure_kv: []
|
||||||
|
hc_vault: []
|
||||||
|
age:
|
||||||
|
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||||
|
enc: |
|
||||||
|
-----BEGIN AGE ENCRYPTED FILE-----
|
||||||
|
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2dWtqOVNQalZ2QlRxOXJa
|
||||||
|
cDc1MTdtWENkdWIxS1JId3dIU1JDSmk0ODNrCk1YUzhFWXR3bUxyV2NNcmFOakdz
|
||||||
|
THl1Q284dFJxTy9Rd1h0eXJJaVZtdmcKLS0tIERuN0ZFeW9zdi9aNFRYNWxmUS9s
|
||||||
|
QUtYdHEvN042ajNoNktOLzlYQmQrc1kK4NHozMbXVzNGYbUBRbK6a9Uw/sVPAfe6
|
||||||
|
RW5KWzZ4I/hloqjN3zzVISoSMieMkBGDnF5qZS5N6FqIc6/HrsDD8A==
|
||||||
|
-----END AGE ENCRYPTED FILE-----
|
||||||
|
lastmodified: "2026-09-29T06:31:16Z"
|
||||||
|
mac: ENC[AES256_GCM,data:62BmBAq45CD2g6LDyx+Ho4bTfrhJcaro/RB/iM+OaRc4a/D83MhjbcBL11Uz/O726inqhkvCKDJ/fFemoc50JDbtC79cYMaxrGJNjFM0sdckWrowPvh6ie7HHHUBRBaefl+gNfnc+pbvcncvRzwA4RyzcZse1wbxKfHN8FjEopE=,iv:h0cMShs3tIgnY6eruUYymYrcLFzQZLk/uXQJ/Jsd5ek=,tag:nZ9Pau7p4xWT1QCN1vv2vA==,type:str]
|
||||||
|
pgp: []
|
||||||
|
encrypted_regex: ^(data|stringData)$
|
||||||
|
version: 3.9.4
|
||||||
@@ -132,6 +132,11 @@ spec:
|
|||||||
value: Etc/UTC
|
value: Etc/UTC
|
||||||
- name: ALLOWED_NETWORKS
|
- name: ALLOWED_NETWORKS
|
||||||
value: 10.0.0.0/8,192.168.0.0/16
|
value: 10.0.0.0/8,192.168.0.0/16
|
||||||
|
- name: PLEX_CLAIM
|
||||||
|
valueFrom:
|
||||||
|
secretKeyRef:
|
||||||
|
name: plex-claim
|
||||||
|
key: PLEX_CLAIM
|
||||||
volumeMounts:
|
volumeMounts:
|
||||||
- name: config
|
- name: config
|
||||||
mountPath: /config
|
mountPath: /config
|
||||||
|
|||||||
@@ -4,7 +4,13 @@
|
|||||||
# against Gitea. Flux decrypts it via the sops-age key at reconcile (apps Kustomization
|
# against Gitea. Flux decrypts it via the sops-age key at reconcile (apps Kustomization
|
||||||
# decryption is wired in clusters/dev/kustomization.yaml). Rotate by resetting the token
|
# decryption is wired in clusters/dev/kustomization.yaml). Rotate by resetting the token
|
||||||
# in Gitea (admin -> actions -> runners) and re-encrypting a fresh value here.
|
# in Gitea (admin -> actions -> runners) and re-encrypting a fresh value here.
|
||||||
# Recover/inspect: infra/devcontainer/dev.sh run sops -d apps/gitea-actions-runner-secret.sops.yaml
|
# ROTATE: ./infra/gitea/runner-token-rotate.sh -- it mints a fresh instance-level token from the
|
||||||
|
# Gitea API and writes it here via `sops set`, without the value ever being printed.
|
||||||
|
# Do NOT `sops -d` this file into a terminal, a log, or an agent transcript. To check only that
|
||||||
|
# it still decrypts, redirect: `sops -d <file> > /dev/null`.
|
||||||
|
# NOTE: a fresh token alone may not re-register the runner. act_runner registers ONCE and keeps
|
||||||
|
# its identity in /data/.runner on the act-runner-data PVC; if Gitea no longer holds a matching
|
||||||
|
# row it skips registration silently. Clearing that PVC is what forces a real re-registration.
|
||||||
apiVersion: v1
|
apiVersion: v1
|
||||||
kind: Secret
|
kind: Secret
|
||||||
metadata:
|
metadata:
|
||||||
|
|||||||
+46
-9
@@ -32,11 +32,46 @@ env:
|
|||||||
# (10.43.0.0/16); 192.168.0.0/16 covers the home LAN.
|
# (10.43.0.0/16); 192.168.0.0/16 covers the home LAN.
|
||||||
- name: ALLOWED_NETWORKS
|
- name: ALLOWED_NETWORKS
|
||||||
value: 10.0.0.0/8,192.168.0.0/16
|
value: 10.0.0.0/8,192.168.0.0/16
|
||||||
# PLEX_CLAIM is deliberately NOT set here: a https://plex.tv/claim token has a ~4-min TTL, far
|
# PLEX_CLAIM — the ONE-SHOT plex.tv server claim, sourced from the SOPS Secret `plex-claim`
|
||||||
# shorter than the commit -> push -> Flux reconcile -> pod-roll cycle, so any token committed to git
|
# (apps/wave-0/plex-claim-secret.sops.yaml, same wave => applied in the same reconcile as this
|
||||||
# is already expired by the time the pod starts. Claiming is a one-time LIVE step (UI sign-in, or a
|
# workload). Plex cannot create libraries over the mounted bulk share while the server is
|
||||||
# single API POST) against the running server; the claim then persists on the /config PVC and
|
# UNCLAIMED, so this is wired; it was previously left out on the reasoning below, and only one
|
||||||
# survives pod rolls (including the roll this env change causes), so it never belongs in git.
|
# half of that reasoning survived contact with the upstream source.
|
||||||
|
#
|
||||||
|
# A SHORT TTL IS THE OPERATIONAL CONSTRAINT, not a reason to abstain. A https://plex.tv/claim token
|
||||||
|
# expires in MINUTES — read the countdown the claim page renders; do not budget against a number
|
||||||
|
# written in this tree. (This comment previously asserted "~4 minutes" as established fact. It was
|
||||||
|
# UNCITED, upstream plexinc/pms-docker documents no expiry at all, and that number was load-bearing
|
||||||
|
# for the whole procedure design — including a since-disproven claim that the delivery pipeline was
|
||||||
|
# too slow to fit inside it. Measured 2026-09-29: the full loop is ~2m45s and fits.)
|
||||||
|
#
|
||||||
|
# The live token is injected by the operator inside that window, with an explicit `flux reconcile`
|
||||||
|
# to collapse the wait. Scripted end to end as infra/plex/claim-ship.sh; runbooks/plex-claim.md is
|
||||||
|
# the procedure. SERVER CLAIMED 2026-09-29 — this env is now a permanent no-op (see below), and the
|
||||||
|
# committed value is the spent token from that claim rather than the original placeholder.
|
||||||
|
#
|
||||||
|
# ONE-SHOT, AND SAFE TO SIT HERE EXPIRED. Verified from plexinc/pms-docker's own first-run script
|
||||||
|
# (branch `master`, root/etc/cont-init.d/40-plex-first-run, confirmed byte-identical to the file in
|
||||||
|
# the running `:latest` image), NOT assumed:
|
||||||
|
# - the exchange (line 106) is guarded on TWO conditions: PLEX_CLAIM non-empty AND the
|
||||||
|
# PlexOnlineToken pref in /config empty. Once claimed, the block is SKIPPED — this env is a
|
||||||
|
# no-op on every subsequent start, whatever it holds. (Upstream README: "If server is already
|
||||||
|
# logged in, this parameter is ignored.") The claim itself persists on the /config PVC as before.
|
||||||
|
# - an invalid/expired token on a still-unclaimed /config gets a 403 with an empty body; curl's
|
||||||
|
# status is unchecked, the token `sed` yields "", the success branch at line 120 does not fire,
|
||||||
|
# there is no `else` and no `exit 1`, and the script (no `set -e`) exits 0. It comes up UNCLAIMED and
|
||||||
|
# SERVING — it does NOT crashloop. The image also leaves S6_BEHAVIOUR_IF_STAGE2_FAILS at its
|
||||||
|
# default 0, which continues startup even when a cont-init script fails.
|
||||||
|
# So the earlier "a placeholder claim Secret would crashloop the pod" note was WRONG about the
|
||||||
|
# placeholder; it was right only about an ABSENT Secret. secretKeyRef here has no `optional` field,
|
||||||
|
# so the Secret must always exist in the Flux graph — which is why it is committed, encrypted, and
|
||||||
|
# listed in apps/wave-0/kustomization.yaml rather than hand-created at claim time. No schema change
|
||||||
|
# is needed: `optional` would only matter if a missing Secret had to be tolerated, and it doesn't.
|
||||||
|
- name: PLEX_CLAIM
|
||||||
|
valueFrom:
|
||||||
|
secretKeyRef:
|
||||||
|
name: plex-claim
|
||||||
|
key: PLEX_CLAIM
|
||||||
# ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex
|
# ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex
|
||||||
# which URL to advertise to plex.tv for REMOTE clients. The only candidate is
|
# which URL to advertise to plex.tv for REMOTE clients. The only candidate is
|
||||||
# https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL
|
# https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL
|
||||||
@@ -56,10 +91,12 @@ storage:
|
|||||||
# container's EPHEMERAL layer -> unclaimed + all state lost on restart. The image inits as
|
# container's EPHEMERAL layer -> unclaimed + all state lost on restart. The image inits as
|
||||||
# root (s6-svscan PID 1), chowns /config to the plex user, then drops -> no fsGroup needed
|
# root (s6-svscan PID 1), chowns /config to the plex user, then drops -> no fsGroup needed
|
||||||
# (verified in-cluster: PID1=root, "Plex Media Server"=plex). Persisting /config here also
|
# (verified in-cluster: PID1=root, "Plex Media Server"=plex). Persisting /config here also
|
||||||
# makes the claim survive restarts once claimed. To auto-claim on first boot instead of the
|
# makes the claim survive restarts once claimed — which is what makes the PLEX_CLAIM env
|
||||||
# UI sign-in, the operator adds a PLEX_CLAIM env from a fresh https://plex.tv/claim token
|
# above a ONE-SHOT: the token is exchanged for a PlexOnlineToken written here, and the
|
||||||
# (4-min TTL) — deliberately NOT wired here: the env secretKeyRef pattern has no `optional`,
|
# exchange is then skipped on every later start. (The older note here claimed a
|
||||||
# so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
|
# placeholder claim Secret "would crashloop the pod"; upstream's first-run script says
|
||||||
|
# otherwise — see the PLEX_CLAIM comment in `env` above. An ABSENT Secret is the hard
|
||||||
|
# failure, and that is why one is committed.)
|
||||||
# THE LIBRARY (docs/04 bulk tier). One READ-ONLY mount of the export ROOT, giving /media/movies,
|
# THE LIBRARY (docs/04 bulk tier). One READ-ONLY mount of the export ROOT, giving /media/movies,
|
||||||
# /media/tv and /media/photos — the whole tree the layout actually has (`ls /mnt/Storage/plex`:
|
# /media/tv and /media/photos — the whole tree the layout actually has (`ls /mnt/Storage/plex`:
|
||||||
# downloads/ movies/ photos/ tv/).
|
# downloads/ movies/ photos/ tv/).
|
||||||
|
|||||||
+15
-1
@@ -1844,6 +1844,10 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
|
|||||||
convention:
|
convention:
|
||||||
- `<svc>.yaml` -> wave(svc)
|
- `<svc>.yaml` -> wave(svc)
|
||||||
- `<svc>-bootstrap-secret.sops.yaml` -> wave(svc) (consumed by svc's env, same apply)
|
- `<svc>-bootstrap-secret.sops.yaml` -> wave(svc) (consumed by svc's env, same apply)
|
||||||
|
- `<svc>-claim-secret.sops.yaml` -> wave(svc) (a one-shot upstream-service claim token
|
||||||
|
consumed by svc's env — plex's plex.tv claim. Same wave as the app for the same reason the
|
||||||
|
bootstrap secret is: an env `secretKeyRef` has no `optional`, so the Secret must be applied
|
||||||
|
in the SAME reconcile as the workload or the pod is CreateContainerConfigError.)
|
||||||
- `<svc>-oauth2-proxy-credentials.sops.yaml` -> wave(svc) (forward-auth sidecar OIDC creds)
|
- `<svc>-oauth2-proxy-credentials.sops.yaml` -> wave(svc) (forward-auth sidecar OIDC creds)
|
||||||
- `<svc>-db.yaml` -> wave(svc-db) (a strictly-lower wave; CNPG DB)
|
- `<svc>-db.yaml` -> wave(svc-db) (a strictly-lower wave; CNPG DB)
|
||||||
- `<svc>-db-app.sops.yaml` -> wave(svc-db) (CNPG initdb owner secret)
|
- `<svc>-db-app.sops.yaml` -> wave(svc-db) (CNPG initdb owner secret)
|
||||||
@@ -1870,6 +1874,11 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
|
|||||||
w = waves[name]
|
w = waves[name]
|
||||||
add(w, f"{name}.yaml")
|
add(w, f"{name}.yaml")
|
||||||
add(w, f"{name}-bootstrap-secret.sops.yaml")
|
add(w, f"{name}-bootstrap-secret.sops.yaml")
|
||||||
|
# A one-shot claim token for an UPSTREAM service (plex's plex.tv claim). Same wave as the
|
||||||
|
# app: it is read by the app's env at pod start, and an env secretKeyRef has no `optional`,
|
||||||
|
# so a Secret applied in a later wave would leave the pod in CreateContainerConfigError.
|
||||||
|
# Placed only when the file exists, so services without one are unaffected.
|
||||||
|
add(w, f"{name}-claim-secret.sops.yaml")
|
||||||
# forward-auth (Option B): the per-app oauth2-proxy sidecar's OIDC client + cookie secret.
|
# forward-auth (Option B): the per-app oauth2-proxy sidecar's OIDC client + cookie secret.
|
||||||
# Same wave as the app (the sidecar reads it at pod start); placed only when the file exists,
|
# Same wave as the app (the sidecar reads it at pod start); placed only when the file exists,
|
||||||
# so non-forward-auth apps are unaffected. Mirrors the bootstrap-secret naming convention.
|
# so non-forward-auth apps are unaffected. Mirrors the bootstrap-secret naming convention.
|
||||||
@@ -2548,7 +2557,8 @@ def _selftest() -> int:
|
|||||||
# 7a) plan_apps_waves: sidecars inherit their unit's wave; not-onboarded SDs are skipped;
|
# 7a) plan_apps_waves: sidecars inherit their unit's wave; not-onboarded SDs are skipped;
|
||||||
# EXTRA_UNITS anchor strictly above; a file absent from the tree is never placed.
|
# EXTRA_UNITS anchor strictly above; a file absent from the tree is never placed.
|
||||||
present = {
|
present = {
|
||||||
"a.yaml", "svc.yaml", "svc-bootstrap-secret.sops.yaml",
|
"a.yaml", "a-claim-secret.sops.yaml",
|
||||||
|
"svc.yaml", "svc-bootstrap-secret.sops.yaml",
|
||||||
"svc-db.yaml", "svc-db-app.sops.yaml", "svc-db-backup-credentials.sops.yaml",
|
"svc-db.yaml", "svc-db-app.sops.yaml", "svc-db-backup-credentials.sops.yaml",
|
||||||
"svc-db-backup-credentials-prod.sops.yaml",
|
"svc-db-backup-credentials-prod.sops.yaml",
|
||||||
"gitea.yaml",
|
"gitea.yaml",
|
||||||
@@ -2559,6 +2569,10 @@ def _selftest() -> int:
|
|||||||
plan = plan_apps_waves(plan_sds, present)
|
plan = plan_apps_waves(plan_sds, present)
|
||||||
flat = {f: wv for wv, fs in plan.items() for f in fs}
|
flat = {f: wv for wv, fs in plan.items() for f in fs}
|
||||||
check(flat.get("a.yaml") == 0, "leaf app must land in wave 0")
|
check(flat.get("a.yaml") == 0, "leaf app must land in wave 0")
|
||||||
|
check(flat.get("a-claim-secret.sops.yaml") == 0,
|
||||||
|
"a `<svc>-claim-secret.sops.yaml` (plex's plex.tv claim) must be placed in the SAME wave "
|
||||||
|
"as its app — an env secretKeyRef has no `optional`, so a later wave means "
|
||||||
|
"CreateContainerConfigError")
|
||||||
check(flat.get("svc-db.yaml") == 0 and flat.get("svc-db-app.sops.yaml") == 0
|
check(flat.get("svc-db.yaml") == 0 and flat.get("svc-db-app.sops.yaml") == 0
|
||||||
and flat.get("svc-db-backup-credentials.sops.yaml") == 0
|
and flat.get("svc-db-backup-credentials.sops.yaml") == 0
|
||||||
and flat.get("svc-db-backup-credentials-prod.sops.yaml") == 0,
|
and flat.get("svc-db-backup-credentials-prod.sops.yaml") == 0,
|
||||||
|
|||||||
@@ -1,5 +1,19 @@
|
|||||||
# `ci/` — portable merge gate (runnable today, without Gitea)
|
# `ci/` — portable merge gate (runnable today, without Gitea)
|
||||||
|
|
||||||
|
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
||||||
|
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
||||||
|
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
||||||
|
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
||||||
|
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
||||||
|
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
||||||
|
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
||||||
|
> coverage hole, not a phase.
|
||||||
|
>
|
||||||
|
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
||||||
|
> documentation rather than measurement, and `main` was unprotected with zero runners
|
||||||
|
> registered the whole time. **Query the live API before asserting platform state.**
|
||||||
|
|
||||||
|
|
||||||
**Task T-0.2 · R-TEST-02 (reproducible, single-command) · Owner WS-GITOPS (CI).**
|
**Task T-0.2 · R-TEST-02 (reproducible, single-command) · Owner WS-GITOPS (CI).**
|
||||||
|
|
||||||
The same merge gate the Gitea Actions workflow runs
|
The same merge gate the Gitea Actions workflow runs
|
||||||
|
|||||||
@@ -1,5 +1,19 @@
|
|||||||
# clusters/
|
# clusters/
|
||||||
|
|
||||||
|
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
||||||
|
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
||||||
|
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
||||||
|
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
||||||
|
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
||||||
|
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
||||||
|
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
||||||
|
> coverage hole, not a phase.
|
||||||
|
>
|
||||||
|
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
||||||
|
> documentation rather than measurement, and `main` was unprotected with zero runners
|
||||||
|
> registered the whole time. **Query the live API before asserting platform state.**
|
||||||
|
|
||||||
|
|
||||||
Flux entrypoints and the environment-overlay model (**R-ENV-02**, **R-GITOPS-01/02/03**).
|
Flux entrypoints and the environment-overlay model (**R-ENV-02**, **R-GITOPS-01/02/03**).
|
||||||
|
|
||||||
Governing spec: [`docs/05-gitops-and-delivery.md`](../../home-network-k3s/docs/05-gitops-and-delivery.md),
|
Governing spec: [`docs/05-gitops-and-delivery.md`](../../home-network-k3s/docs/05-gitops-and-delivery.md),
|
||||||
|
|||||||
@@ -1,5 +1,19 @@
|
|||||||
# conformance/ — the home-k3s conformance harness (WS-TEST)
|
# conformance/ — the home-k3s conformance harness (WS-TEST)
|
||||||
|
|
||||||
|
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
||||||
|
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
||||||
|
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
||||||
|
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
||||||
|
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
||||||
|
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
||||||
|
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
||||||
|
> coverage hole, not a phase.
|
||||||
|
>
|
||||||
|
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
||||||
|
> documentation rather than measurement, and `main` was unprotected with zero runners
|
||||||
|
> registered the whole time. **Query the live API before asserting platform state.**
|
||||||
|
|
||||||
|
|
||||||
This subtree is the **definition of "done"** for the whole platform. Nothing is accepted
|
This subtree is the **definition of "done"** for the whole platform. Nothing is accepted
|
||||||
on a demo, a screenshot, or "it reconciled": every requirement is proven by an
|
on a demo, a screenshot, or "it reconciled": every requirement is proven by an
|
||||||
**automated, reproducible, behavior-based test** tagged with the requirement ID(s) it
|
**automated, reproducible, behavior-based test** tagged with the requirement ID(s) it
|
||||||
|
|||||||
@@ -1,5 +1,19 @@
|
|||||||
# In-cluster conformance Job (L2/L3)
|
# In-cluster conformance Job (L2/L3)
|
||||||
|
|
||||||
|
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
||||||
|
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
||||||
|
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
||||||
|
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
||||||
|
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
||||||
|
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
||||||
|
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
||||||
|
> coverage hole, not a phase.
|
||||||
|
>
|
||||||
|
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
||||||
|
> documentation rather than measurement, and `main` was unprotected with zero runners
|
||||||
|
> registered the whole time. **Query the live API before asserting platform state.**
|
||||||
|
|
||||||
|
|
||||||
Per **ADR-0011** and **docs/14 "Where each layer runs"**: L0/L1 run in CI without a
|
Per **ADR-0011** and **docs/14 "Where each layer runs"**: L0/L1 run in CI without a
|
||||||
cluster, but **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job**
|
cluster, but **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job**
|
||||||
on the dev/test cluster that **CI triggers and waits for**. CI holds **no** cluster
|
on the dev/test cluster that **CI triggers and waits for**. CI holds **no** cluster
|
||||||
|
|||||||
+6
-1
@@ -6,7 +6,7 @@ Every alert rule in this platform carries an annotation of the form
|
|||||||
|
|
||||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"
|
||||||
|
|
||||||
46 of them, over the 35 names below. Until this directory existed, that path returned **HTTP 404** on the
|
46 of them, over 35 names. Until this directory existed, that path returned **HTTP 404** on the
|
||||||
live cluster: every alert that reached a human handed them a dead link at the exact moment they needed
|
live cluster: every alert that reached a human handed them a dead link at the exact moment they needed
|
||||||
help. These pages are what makes the link resolve.
|
help. These pages are what makes the link resolve.
|
||||||
|
|
||||||
@@ -108,6 +108,11 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
|
|||||||
- **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
|
- **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
|
||||||
every green signal missed.
|
every green signal missed.
|
||||||
|
|
||||||
|
### Operator procedures (no alert links here)
|
||||||
|
- **`plex-claim`** — claiming the Plex Media Server. The plex.tv claim token lives ~4 minutes, so this
|
||||||
|
is a timed sequence: token → `sops` → push → **forced** `flux reconcile` → pod roll. Includes why a
|
||||||
|
changed Secret does not restart anything, and why an expired token is harmless.
|
||||||
|
|
||||||
### Virtualisation
|
### Virtualisation
|
||||||
- **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.
|
- **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.
|
||||||
- **`kubevirt-launch-failed`** — a VMI reached the Failed phase. The three k3s-specific launch traps.
|
- **`kubevirt-launch-failed`** — a VMI reached the Failed phase. The three k3s-specific launch traps.
|
||||||
|
|||||||
@@ -81,6 +81,11 @@ configMapGenerator:
|
|||||||
- node-io=node-io.md
|
- node-io=node-io.md
|
||||||
- node-memory=node-memory.md
|
- node-memory=node-memory.md
|
||||||
- node-memory-swap=node-memory-swap.md
|
- node-memory-swap=node-memory-swap.md
|
||||||
|
# Operator procedure, not an alert page (no alert links to it — the
|
||||||
|
# alert-runbooks-resolvable check treats that as advisory). It is served
|
||||||
|
# because the 4-minute plex.tv claim window is not a thing to go looking
|
||||||
|
# for in git: /runbooks/plex-claim has to resolve the moment it is needed.
|
||||||
|
- plex-claim=plex-claim.md
|
||||||
- pvc-capacity=pvc-capacity.md
|
- pvc-capacity=pvc-capacity.md
|
||||||
- targets-down=targets-down.md
|
- targets-down=targets-down.md
|
||||||
- update-rollback=update-rollback.md
|
- update-rollback=update-rollback.md
|
||||||
|
|||||||
@@ -0,0 +1,185 @@
|
|||||||
|
# plex-claim — claiming the Plex Media Server inside the 4-minute token window
|
||||||
|
|
||||||
|
**Alert:** none. This is an **operator procedure**, not an incident page — it is indexed here because
|
||||||
|
it is the page you need at the moment you need it, with a stopwatch running.
|
||||||
|
**Files:** `catalog/plex.yaml` (env `PLEX_CLAIM`), `apps/wave-0/plex-claim-secret.sops.yaml`
|
||||||
|
(the SOPS Secret `plex-claim`, key `PLEX_CLAIM`), `apps/wave-0/plex.yaml` (the rendered golden).
|
||||||
|
|
||||||
|
## What this means here
|
||||||
|
|
||||||
|
Plex will not let you **create libraries** over the mounted bulk media share while the server is
|
||||||
|
**unclaimed** — the 8.33 T NFS tree at `/media` is there, readable, and unusable. Claiming binds the
|
||||||
|
server to a plex.tv account. It is a **one-shot** operation: the token is exchanged once for a durable
|
||||||
|
`PlexOnlineToken` written into `Preferences.xml` on the `plex-config` Longhorn PVC, and from then on
|
||||||
|
the `PLEX_CLAIM` env is **ignored on every start** (upstream gate:
|
||||||
|
`[ ! -z "${PLEX_CLAIM}" ] && [ -z "${token}" ]` in `root/etc/cont-init.d/40-plex-first-run`).
|
||||||
|
|
||||||
|
**A `https://plex.tv/claim` token lives ~4 minutes.** That is shorter than commit → push → Flux's
|
||||||
|
10-minute reconcile interval → pod roll if you let those run on their own timers, which is why this
|
||||||
|
page exists: every step below is written to be run back-to-back, with the reconcile **forced** rather
|
||||||
|
than waited for. Budget: token in hand at T+0, pod restarted by roughly T+2:00.
|
||||||
|
|
||||||
|
**An expired or wrong token does not break anything.** Verified against the image's own first-run
|
||||||
|
script (and the copy inside `plexinc/pms-docker:latest`): a failed exchange gets a 403 with an empty
|
||||||
|
body, `curl`'s status is never checked, the token `sed` yields empty, the `if` does not fire, there is
|
||||||
|
no `else` and no `exit 1`, and the script (no `set -e`) exits 0. The pod comes up **unclaimed and
|
||||||
|
serving**. So a burnt token costs you one more attempt, not an outage — and the repo deliberately
|
||||||
|
ships a **placeholder** in that Secret for exactly this reason (the env `secretKeyRef` has no
|
||||||
|
`optional`, so the Secret must always exist even when its value is useless).
|
||||||
|
|
||||||
|
## Running the commands in this runbook
|
||||||
|
|
||||||
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
||||||
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that
|
||||||
|
hang is not the cluster being down. Every command below runs as:
|
||||||
|
|
||||||
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
||||||
|
|
||||||
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
||||||
|
picks a current-context per invocation, and that default has already changed once (k3s-dev until dev
|
||||||
|
was decommissioned on 2026-09-13, k3s-prod now). Do not run two `dev.sh` invocations at once.
|
||||||
|
|
||||||
|
Mutating `kubectl` and `flux reconcile` are blocked for agents here. **This whole page is operator
|
||||||
|
work** — the reconcile and the pod roll below are deliberate operator actions, and the durable change
|
||||||
|
is still a git change.
|
||||||
|
|
||||||
|
## First commands
|
||||||
|
|
||||||
|
**Before spending a token, find out whether the server is already claimed.** This prints only the
|
||||||
|
attribute NAME, never its value — do not widen it to print the token:
|
||||||
|
|
||||||
|
```
|
||||||
|
infra/devcontainer/dev.sh run bash -c \
|
||||||
|
'kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c \
|
||||||
|
"grep -o PlexOnlineToken= \"/config/Library/Application Support/Plex Media Server/Preferences.xml\" | head -1"'
|
||||||
|
```
|
||||||
|
|
||||||
|
`PlexOnlineToken=` printed ⇒ **already claimed**; stop, there is nothing to do and no token to burn.
|
||||||
|
Empty output ⇒ unclaimed, continue.
|
||||||
|
|
||||||
|
Is the workload healthy enough to claim at all, and is the wave reconciling?
|
||||||
|
|
||||||
|
```
|
||||||
|
infra/devcontainer/dev.sh run bash -c \
|
||||||
|
'kubectl --context k3s-prod -n plex get deploy,pod,pvc'
|
||||||
|
infra/devcontainer/dev.sh run bash -c \
|
||||||
|
'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0'
|
||||||
|
```
|
||||||
|
|
||||||
|
## The 4-minute procedure
|
||||||
|
|
||||||
|
Have this page open and do not stop between steps.
|
||||||
|
|
||||||
|
1. **T+0 — get the token.** Sign in to plex.tv as the account that should OWN this server, open
|
||||||
|
<https://plex.tv/claim>, copy the `claim-…` string. The clock starts now.
|
||||||
|
|
||||||
|
2. **T+0:15 — put it in the Secret**, in place, encrypted, without it ever touching a plaintext file:
|
||||||
|
|
||||||
|
```
|
||||||
|
infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
Replace the `PLEX_CLAIM:` value (the placeholder `replace-me-with-a-fresh-plex-tv-claim-token`, or
|
||||||
|
a previous expired token) and save. `sops` re-encrypts on write to the committed age recipient.
|
||||||
|
|
||||||
|
3. **T+0:45 — commit and push.** `git push origin main` **is** the deploy; there is no other apply
|
||||||
|
step.
|
||||||
|
|
||||||
|
```
|
||||||
|
git add apps/wave-0/plex-claim-secret.sops.yaml
|
||||||
|
git commit -m 'chore(plex): fresh plex.tv claim token'
|
||||||
|
git push origin main
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **T+1:00 — force the reconcile immediately. This is the one command that saves the window** — it
|
||||||
|
pulls the new commit and re-applies wave 0 instead of waiting up to 10 minutes for the interval:
|
||||||
|
|
||||||
|
```
|
||||||
|
infra/devcontainer/dev.sh run bash -c \
|
||||||
|
'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-0 --with-source'
|
||||||
|
```
|
||||||
|
|
||||||
|
`--with-source` is the load-bearing flag: it reconciles the `flux-system` GitRepository first, so
|
||||||
|
the Kustomization applies **your commit** and not the previously-fetched revision.
|
||||||
|
|
||||||
|
5. **T+1:30 — roll the pod.** This step is **mandatory and easy to forget**: a changed Secret does
|
||||||
|
**not** restart anything. `PLEX_CLAIM` is injected by `secretKeyRef` and read only at container
|
||||||
|
start, and the Deployment's pod template is byte-identical across this change, so Flux applies the
|
||||||
|
new Secret and the running pod keeps the old value forever.
|
||||||
|
|
||||||
|
```
|
||||||
|
infra/devcontainer/dev.sh run bash -c \
|
||||||
|
'kubectl --context k3s-prod -n plex rollout restart deployment/plex'
|
||||||
|
```
|
||||||
|
|
||||||
|
6. **T+2:00 — confirm the exchange happened**, in the new pod's init log:
|
||||||
|
|
||||||
|
```
|
||||||
|
infra/devcontainer/dev.sh run bash -c \
|
||||||
|
'kubectl --context k3s-prod -n plex logs deploy/plex --since=5m | grep -i -e claim -e token'
|
||||||
|
```
|
||||||
|
|
||||||
|
* `Attempting to obtain server token from claim token` **and** `Token obtained successfully`
|
||||||
|
⇒ **claimed.** Re-run the First-commands `PlexOnlineToken=` check to confirm it persisted, then
|
||||||
|
create the libraries over `/media/movies`, `/media/tv`, `/media/photos`.
|
||||||
|
* `Attempting to obtain server token from claim token` with **no** success line ⇒ the token was
|
||||||
|
expired or invalid. Nothing is broken; go back to step 1 with a new token.
|
||||||
|
* **Neither line** ⇒ the exchange was skipped, which means `/config` already had a
|
||||||
|
`PlexOnlineToken` (already claimed) or the env did not reach the container. Check the pod actually
|
||||||
|
has the variable, by NAME only:
|
||||||
|
|
||||||
|
```
|
||||||
|
infra/devcontainer/dev.sh run bash -c \
|
||||||
|
'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
|
||||||
|
-o jsonpath="{.items[0].spec.containers[0].env[*].name}"'
|
||||||
|
```
|
||||||
|
|
||||||
|
## Likely causes, most likely first
|
||||||
|
|
||||||
|
When the claim does not take, in order of how often it is the answer here:
|
||||||
|
|
||||||
|
1. **The token expired** — you spent more than ~4 minutes, usually in step 3 or waiting on step 4.
|
||||||
|
Symptom: the "Attempting…" line with no success line. Remedy: a new token, and force the reconcile
|
||||||
|
rather than waiting for it.
|
||||||
|
2. **The pod was never restarted** (step 5 skipped). Symptom: no claim log lines at all from a pod
|
||||||
|
whose `AGE` predates your push. The Secret is correct in git and in the cluster and the running
|
||||||
|
container still holds the placeholder.
|
||||||
|
3. **Flux had not applied the commit yet** when you rolled the pod — step 5 run before step 4
|
||||||
|
finished, so the new pod read the OLD Secret. Check
|
||||||
|
`kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0 -o jsonpath='{.status.lastAppliedRevision}'`
|
||||||
|
against your commit, then roll again.
|
||||||
|
4. **Already claimed** — the exchange is gated on `PlexOnlineToken` being absent from `/config`, so a
|
||||||
|
server claimed earlier ignores every later token silently. This is the benign case.
|
||||||
|
5. **`plex.tv` unreachable.** The upstream `curl` has no `--max-time`/`--connect-timeout`, so a stalled
|
||||||
|
connection can leave the container sitting in cont-init with **nothing listening on 32400** — the
|
||||||
|
one genuine hang risk in this path. Symptom: pod Running but never Ready, log stops after
|
||||||
|
"Attempting…". Check egress (`plex` is `egress: internet`) rather than re-rolling repeatedly.
|
||||||
|
6. **Wrong plex.tv account.** The claim succeeds and binds the server to whoever was signed in at
|
||||||
|
step 1. Unclaiming is a plex.tv-side operation on that account — see "escalate or stop".
|
||||||
|
|
||||||
|
## When to escalate or stop
|
||||||
|
|
||||||
|
- **Stop after two failed token attempts** and fix the timing, not the token. The third rushed attempt
|
||||||
|
is how people start editing live Secrets by hand, which leaves the cluster holding a value that is
|
||||||
|
in no git commit — precisely the drift `qbittorrent-bootstrap` had to be recovered from.
|
||||||
|
- **Never `kubectl edit`/`patch` the `plex-claim` Secret in the cluster, and never
|
||||||
|
`kubectl create secret` it.** Everything durable goes through the SOPS file; a live-only Secret is
|
||||||
|
reverted (or adopted with the wrong value) on the next reconcile.
|
||||||
|
- **Never `sops -d` this file into a terminal, a log, or a shared screen**, and never widen the
|
||||||
|
`PlexOnlineToken=` check into one that prints the value. The token is a plex.tv account credential.
|
||||||
|
- **If the server got claimed to the wrong account, stop and get the operator.** Reversing it means
|
||||||
|
removing the server from that account on plex.tv and clearing `PlexOnlineToken` from
|
||||||
|
`Preferences.xml` on the PVC — a destructive edit of app state on the replicated tier, not a git
|
||||||
|
change.
|
||||||
|
- **Do not delete the `plex-config` PVC to "start clean".** It holds the library database, metadata
|
||||||
|
and watch state (`policy: daily-offcluster`), and it is expensive to rebuild. The claim is a
|
||||||
|
two-line pref, not a reason to lose the DB.
|
||||||
|
- After a successful claim, **leave the expired token in the SOPS file**. It is inert (the exchange is
|
||||||
|
skipped once claimed) and the Secret must keep existing, because the env `secretKeyRef` has no
|
||||||
|
`optional` — deleting it would put the pod into `CreateContainerConfigError`.
|
||||||
|
|
||||||
|
---
|
||||||
|
Source of truth: `runbooks/plex-claim.md`. Index: `/runbooks/`
|
||||||
|
Upstream behaviour cited here is from `plexinc/pms-docker` branch `master`,
|
||||||
|
`root/etc/cont-init.d/40-plex-first-run` (s6-overlay v2), cross-checked against the same file inside
|
||||||
|
the running `plexinc/pms-docker:latest` image.
|
||||||
Reference in new issue
Block a user