Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6586919132 | ||
|
|
e8e6b9c681 | ||
|
|
fcc66c7858 | ||
|
|
b9e8a4f9e8 | ||
|
|
b2d9883958 | ||
|
|
00efd4b828 | ||
|
|
67a54a33a4 | ||
|
|
db1a623073 | ||
|
|
ce271225de | ||
|
|
e72050c690 | ||
|
|
761e25cc3f | ||
|
|
e7508931ca | ||
|
|
8e017fad91 |
No files matched your search
+28
-14
@@ -40,9 +40,15 @@ and ADR-0011: **CI holds no broad kubeconfig.** L2/L3 run as an in-cluster **Cha
|
||||
*collects* its artifacts, using a narrow trigger credential referenced **by secret name only**
|
||||
(`secrets.CONFORMANCE_TRIGGER_KUBECONFIG`) — never a plaintext secret, never a cluster-admin kubeconfig.
|
||||
|
||||
## Honest SKIP — never fake green (Phase 0)
|
||||
## Honest SKIP — never fake green
|
||||
|
||||
**No cluster exists in Phase 0.** The gate is deliberately built so unavailable checks **SKIP loudly,
|
||||
**NOTE (2026-09-29):** this section is written in the past's voice — "Phase 0", "no cluster
|
||||
exists". **A cluster has existed for weeks** and Flux has been reconciling against it. The
|
||||
skip logic below is still correct and still honest, but read it as describing a **live
|
||||
coverage gap** (L2/L3 is not wired into CI) rather than an expected phase. Its being
|
||||
phrased as a phase is part of why the gap went unexamined.
|
||||
|
||||
Originally: The gate is deliberately built so unavailable checks **SKIP loudly,
|
||||
they do not pass**:
|
||||
|
||||
- The `conformance` job is `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}` — with no cluster it is
|
||||
@@ -77,21 +83,29 @@ Both dirs are git-ignored (see [`.gitignore`](../../.gitignore)).
|
||||
## Triggers & branch model (CONTRIBUTING.md)
|
||||
|
||||
- `pull_request` → `main`: **PRs are the enforced merge gate into `main`.**
|
||||
- `push` → `feature/**`, `fix/**`, `docs/**`, `chore/**`: fast feedback on the branch. **No work on `main`.**
|
||||
- `push` → `main` only. The per-branch prefix list was removed: `act_runner` runs at capacity 1,
|
||||
so a branch push fires a redundant run that serialises *ahead* of the PR run for the same tree.
|
||||
**No work on `main`** — and since 2026-09-29 that is enforced, not conventional: `main` is a
|
||||
protected branch and a direct push is rejected by the pre-receive hook (verified, not assumed).
|
||||
|
||||
## Transition: generic-CI-now → Gitea-Actions-after-Gitea-is-up
|
||||
## Transition: COMPLETE (2026-09-29)
|
||||
|
||||
Per [`CONTRIBUTING.md`](../../../home-network-k3s/CONTRIBUTING.md) and
|
||||
[`docs/05` §"local bootstrap → self-hosted Gitea"](../../../home-network-k3s/docs/05-gitops-and-delivery.md):
|
||||
This section used to describe a migration as pending. It has happened. Gitea is live at
|
||||
`gitea.home.keithjackson.dev`, the runner `incluster-dind` is registered and online, and
|
||||
`gate.yml` runs on every PR.
|
||||
|
||||
1. **Now (interim):** there is **no remote git host** on this network. This workflow file is dormant —
|
||||
but the **identical gate runs locally today** via `make -C ci gate` (R-TEST-02). Development is
|
||||
local feature-branch → `main`, the gate enforced by convention + the local runner.
|
||||
2. **Gitea comes online early** (one of the first services onboarded) at `gitea.home.keithjackson.dev`.
|
||||
3. **CI moves to Gitea Actions:** this workflow activates unchanged. The only operational steps are
|
||||
enabling a Gitea Actions runner and setting `vars.CLUSTER_AVAILABLE=true` +
|
||||
`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` once a dev/test cluster exists. PRs then become the
|
||||
**platform-enforced** merge gate — the same no-work-in-main rule, now enforced by machine.
|
||||
**What took a month to notice:** the migration was recorded as done on 2026-08-31 without
|
||||
anyone checking the platform. In reality `main` had **no branch protection** and **zero
|
||||
runners were registered** until 2026-09-29. Every "the gate enforces this" statement in
|
||||
between rested on an assertion nobody verified. PR #1 — the first pull request this
|
||||
repository ever had — sat `pending` for 30.6 minutes because nothing existed to run it.
|
||||
|
||||
So: **do not read this file, or the DECISIONS-LOG, as evidence about the live platform.**
|
||||
Query the API. `infra/gitea/` holds the scripts and the measured state.
|
||||
|
||||
Still genuinely outstanding: `vars.CLUSTER_AVAILABLE` and
|
||||
`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` are unset, so L2/L3 conformance does not run in
|
||||
CI. That is a real coverage gap, not a phase.
|
||||
|
||||
## Configuration (set in Gitea; referenced by name only)
|
||||
|
||||
|
||||
@@ -11,11 +11,14 @@
|
||||
# v
|
||||
# merge-gate decision [R-TEST-03: touched subset green]
|
||||
#
|
||||
# Transition (CONTRIBUTING.md / docs/05 §"local bootstrap -> self-hosted Gitea"):
|
||||
# Today there is no Gitea host, so this file is dormant. The SAME gate is runnable
|
||||
# now via `make -C ci gate` (R-TEST-02). Once Gitea is self-hosted and Gitea
|
||||
# Actions is enabled, this workflow becomes the ENFORCED PR merge gate into main —
|
||||
# no code change required, only turning on the runner and setting CLUSTER_AVAILABLE.
|
||||
# STATUS (corrected 2026-09-29): Gitea IS hosted and Gitea Actions IS enabled; the runner
|
||||
# `incluster-dind` is online and gate.yml is the live PR gate. The text that used to sit here
|
||||
# ("Today there is no Gitea host, so this file is dormant") was stale by weeks -- it was written
|
||||
# as a plan and never revisited once the plan completed.
|
||||
#
|
||||
# THIS file is still dormant, but for the reason in the DORMANT block below -- its spec checkout
|
||||
# and L2/L3 seams are unwired -- NOT because there is no host. Do not read a stale plan as a
|
||||
# statement about the live platform; query the API instead (see infra/gitea/).
|
||||
#
|
||||
# CI holds NO broad kubeconfig. L2/L3 run as an in-cluster Chainsaw Job whose own
|
||||
# scoped ServiceAccount holds cluster access (docs/14, R-SEC-02); CI uses only a
|
||||
|
||||
@@ -23,28 +23,58 @@ There is no other path. Specifically:
|
||||
> (main is what the cluster follows), not what you are permitted to do. Honour this
|
||||
> file, not that line. Fixing those runbooks is tracked work.
|
||||
|
||||
## 2. Enforcement is currently WEAKER than this rule. Follow the rule anyway.
|
||||
## 2. Enforcement (as of 2026-09-29: the rule is now MACHINE-ENFORCED)
|
||||
|
||||
Verified against the live Gitea on 2026-09-29:
|
||||
|
||||
| Expected | Actual |
|
||||
|---|---|
|
||||
| `main` protected, direct push rejected | **No branch protection.** `protected: false`, `user_can_push: true` |
|
||||
| PR blocked until the gate passes | **`enable_status_check: false`**, `status_check_contexts: []` |
|
||||
| Gate runs on every PR | **Fixed 2026-09-29.** Runner `incluster-dind` is online; `gate / gate (pull_request)` runs and passes in ~1m30s–2m |
|
||||
| Approval required before merge | **`required_approvals: 0`** |
|
||||
| `main` protected, direct push rejected | ✅ `protected: true`, `user_can_push: false` — **verified by a real rejected push**, not by the API's word |
|
||||
| PR blocked until the gate passes | ✅ `enable_status_check: true`, context `gate / gate (pull_request)` |
|
||||
| Gate runs on every PR | ✅ runner `incluster-dind` online; ~1m35s |
|
||||
| Approval required before merge | ✅ `required_approvals: 1` |
|
||||
|
||||
So nothing mechanically stops you from pushing to `main` right now. **That is a gap
|
||||
being closed, not permission.** `DECISIONS-LOG.md` (2026-08-31) claimed the gate was a
|
||||
"required status check" with a runner online; neither was true, which is how this went
|
||||
unnoticed for a month. The runner half is now genuinely fixed and observed green. The
|
||||
protection half is not — see `infra/gitea/`.
|
||||
All four were FALSE as recently as 2026-09-29, while this repo's own log asserted they were
|
||||
true. They are true now because each was measured, and the push rejection was provoked rather
|
||||
than assumed. **Re-measure before relying on any of them** — see §2b.
|
||||
|
||||
Still not covered: `vars.CLUSTER_AVAILABLE` is unset, so **L2/L3 conformance does not run in
|
||||
CI**. A green PR therefore proves the L0/L1 offline subset, not runtime behaviour.
|
||||
|
||||
**Consequence for you:** CI now runs, but it does not yet *block* anything. Always run
|
||||
the toolbox gate yourself and quote its numbers. Never describe a PR as "green" on the
|
||||
strength of a check you have not seen report — and note that a **skipped** Gitea job
|
||||
reports as `success`, so a green tick is not by itself evidence that anything ran.
|
||||
|
||||
## 2b. Never infer platform state from this repository
|
||||
|
||||
**This repo is not evidence about the live platform.** Neither is `DECISIONS-LOG.md`.
|
||||
Both describe what someone believed at the time of writing, and both have been
|
||||
confidently wrong for a month at a stretch.
|
||||
|
||||
On 2026-09-29 the log stated the merge gate was a required status check with a runner
|
||||
online. The API said `branch_protections: []` and **zero** runners. Everything downstream
|
||||
of that belief — "the gate enforces this", "your push would be rejected", "CI covers it" —
|
||||
was wrong, and nothing in the tree could have revealed it, because the tree was the source
|
||||
of the error.
|
||||
|
||||
**So, before asserting that any gate, protection, runner, secret, or workload is in a
|
||||
given state: query the thing itself.**
|
||||
|
||||
| Claim about... | Check |
|
||||
|---|---|
|
||||
| branch protection / required checks | `GET /api/v1/repos/{owner}/{repo}/branches/main` |
|
||||
| a runner exists | `GET /api/v1/admin/actions/runners` |
|
||||
| CI actually ran | the commit's `/statuses` — and read the **description**; a *skipped* Gitea job reports `success` with "Has been skipped" |
|
||||
| a workload / Secret / env | `kubectl --context k3s-prod ...` (read-only is fine for agents) |
|
||||
| Flux applied your commit | compare `.status.lastAppliedRevision` against the GitRepository's artifact revision |
|
||||
|
||||
Quote the measured field, not the prose. If you cannot measure it, say so explicitly —
|
||||
"unverified" is a legitimate report, and a far better one than a confident guess that
|
||||
matches the docs.
|
||||
|
||||
**A green tick is not proof anything ran.** Read what actually executed.
|
||||
|
||||
## 3. Gate before you push
|
||||
|
||||
./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
|
||||
|
||||
@@ -1192,6 +1192,13 @@ fresh agent should read this entry + `infra/PHASE1-STATUS.md` + `docs/plans/` an
|
||||
Platform-admins require TOTP MFA.
|
||||
- **Gitea Actions CI is LIVE + enforcing**: `.gitea/workflows/gate.yml` runs `make -C ci gate` on push/PR to main;
|
||||
required status check; runner `incluster-dind` online (ns gitea-actions).
|
||||
> **[CORRECTION, 2026-09-29 — THIS BULLET WAS FALSE WHEN WRITTEN AND STAYED FALSE FOR A MONTH.]**
|
||||
> Measured against the live Gitea API on 2026-09-29: `branch_protections` returned `[]` (`main`
|
||||
> `protected:false`, `enable_status_check:false`, `user_can_push:true`) and **zero** Actions runners
|
||||
> were registered at repo, org or site level. The gate was neither enforced nor able to run. PR #1
|
||||
> — the first pull request this repo ever had — was opened on 2026-09-29 and sat `pending` for
|
||||
> 30.6 minutes with nothing to execute it. Both halves of this bullet were asserted, not verified.
|
||||
> See the 2026-09-29 entries for the root cause and the fix.
|
||||
|
||||
### What changed this session (durable)
|
||||
1. **SSO cluster-wide break→fix — root cause Cilium CVE-2025-30162.** The 1.16→1.20 security fix makes the shared
|
||||
@@ -4781,3 +4788,179 @@ accurate before the PR flow and are stale now. Correcting them is tracked, not d
|
||||
3. Land `feat/plex-claim-sops-secret` (PR #1) once the gate can actually run.
|
||||
4. Claim plex, after checking it is not already claimed.
|
||||
5. Correct the 37 runbooks' `git push origin main` wording.
|
||||
|
||||
## 2026-09-29 — Plex is claimed; the CI/CD gate that "enforced" it never existed. **PM HANDOFF.**
|
||||
|
||||
Began as "inject a plex.tv claim token". Ended by discovering the delivery pipeline was fiction,
|
||||
rebuilding it, and shipping the token through it. Plex is **claimed** — `PlexOnlineToken` is
|
||||
present in `/config` on the `plex-config` PVC, so the exchange is now a permanent no-op and the
|
||||
library scan is running over the 8.33T NFS share.
|
||||
|
||||
### The finding: both halves of the 2026-08-31 CI claim were false
|
||||
|
||||
That entry recorded "Gitea Actions CI is LIVE + enforcing … required status check; runner
|
||||
`incluster-dind` online". Measured against the live API on 2026-09-29:
|
||||
|
||||
| Claimed | Measured |
|
||||
|---|---|
|
||||
| `main` protected, gate required | `branch_protections` → `[]`; `protected:false`, `enable_status_check:false`, `required_approvals:0`, `user_can_push:true` |
|
||||
| Runner online | **zero** runners at repo, org AND site level |
|
||||
|
||||
**PR #1 was the first pull request this repository has ever had.** Its contexts sat `pending`
|
||||
for 30.6 minutes across 122 polls and never reported. For a month, every "the gate enforces this"
|
||||
statement rested on an assertion nobody had checked — the same fail-open class as 2026-09-18,
|
||||
and the reason it hid so long is that the log itself was the evidence.
|
||||
|
||||
### Root cause of the dead runner
|
||||
|
||||
`act_runner` registers **once** and persists its identity in `/data/.runner` on the
|
||||
`act-runner-data` PVC. Gitea no longer held a matching row, so the pod booted, saw `.runner`,
|
||||
**skipped registration**, and polled with an identity Gitea did not recognise. Symptom: zero
|
||||
runners, pod Running, nothing crashlooping, no error anyone reads. Deleting the PVC forced a
|
||||
genuine re-registration (`.runner is missing` → `Runner registered successfully`).
|
||||
|
||||
It then did not come back for several minutes, because Flux could not recreate the PVC:
|
||||
`apps-wave-1` was blocked four levels deep on a **stale** `DependencyNotReady` for
|
||||
`platform-config`, which itself read `Ready=True`. Each wave re-evaluates only on its own
|
||||
`interval: 10m`, so a resource deleted at the bottom of the chain can take ~40 min to return.
|
||||
**Force it** with `flux reconcile kustomization apps-wave-1 --with-source` rather than waiting.
|
||||
|
||||
### Corrections to things this log and the runbooks asserted
|
||||
|
||||
- **A skipped Gitea job reports `success`, not pending.** Measured: `gate / conformance` returned
|
||||
`success` / "Has been skipped". An earlier entry (#2) said requiring it would deadlock every PR.
|
||||
Wrong — it would pass **vacuously**, forever, while nothing ran. Under HARD RULE 4 that is worse
|
||||
than a deadlock, because a deadlock is loud. Never require it; require
|
||||
`gate / gate (pull_request)` alone. Corrected in #3.
|
||||
- **`gate.yml`'s push trigger omits `feat/**`** (it lists `feature/** fix/** docs/** chore/**`).
|
||||
With `act_runner` capacity 1, a branch prefix that IS in that list fires a second redundant run
|
||||
which serialises ahead of the PR run and doubles wall-clock. Exploited deliberately in
|
||||
`infra/plex/claim-ship.sh`; the honest fix is to align the trigger list.
|
||||
- **The "~4-minute TTL"** asserted in four places was **uncited**, and upstream `pms-docker`
|
||||
documents no expiry at all. The authoritative reading is the countdown plex.tv/claim renders at
|
||||
issue time. Removed from `runbooks/plex-claim.md`; the same claim still sits in
|
||||
`catalog/plex.yaml` and the Secret's header comment (left alone — editing a `.sops.yaml`
|
||||
comment risks the MAC, and touching `catalog/plex.yaml` re-renders a golden).
|
||||
|
||||
### Measured, once the pipeline actually worked
|
||||
|
||||
`gate / gate (pull_request)` 1m34s–1m58s. Full claim loop end to end **~2m45s** — comfortably
|
||||
inside the window, which retires the theory that a short-lived token cannot survive a PR flow.
|
||||
That theory was mine and it was wrong: the pipeline was never too slow, it simply did not exist.
|
||||
|
||||
### State now
|
||||
|
||||
- `main` **protected**, `user_can_push:false`. Runner `incluster-dind` online.
|
||||
- `CLAUDE.md` is the agent delivery contract; `infra/gitea/` holds the protection + runner scripts.
|
||||
- **All 37 runbooks** that instructed `git push origin main` now describe the PR flow — that
|
||||
instruction had become actively blocked, not merely stale.
|
||||
- `infra/plex/claim-ship.sh` drives the claim loop end to end.
|
||||
|
||||
### Next steps
|
||||
|
||||
1. **`enable_status_check: true`** with `gate / gate (pull_request)` as the only context. The
|
||||
prerequisite is met: the gate has been observed green on real PRs.
|
||||
2. Add `jakman85` as a repo **collaborator**, then `required_approvals: 1`. (Currently the only
|
||||
org member is `gitea-admin`, which agents authenticate as — requiring approval before that is
|
||||
fixed would leave nobody able to approve.)
|
||||
3. Run the **push probe** against protection: `gitea-admin` is a site admin and Gitea may let it
|
||||
bypass. `user_can_push:false` is the API's opinion, not proof.
|
||||
4. Align `gate.yml`'s push trigger with this repo's `feat/**` prefix.
|
||||
5. Convert the `dind` container to a **native sidecar** with a `startupProbe`, per the standing
|
||||
FOLLOW-UP note in `apps/wave-1/gitea-actions.yaml`. It deletes both hand-rolled wait loops and
|
||||
the envsubst hazard. Land it ALONE — it is the merge gate for the whole repo.
|
||||
6. Create the Plex libraries over `/media/movies`, `/media/tv`, `/media/photos`.
|
||||
|
||||
## 2026-09-29 (later) — the merge gate is now real, enforced, and proven. **PM HANDOFF.**
|
||||
|
||||
Closes the gap opened by the earlier entry today. Eight PRs, all gated; every claim below was
|
||||
measured against the live API or cluster, not inferred from this repository.
|
||||
|
||||
### Final state
|
||||
|
||||
| | Verified |
|
||||
|---|---|
|
||||
| `main` protected | `protected:true`, `user_can_push:false` — and a direct push as site-admin `gitea-admin` was **provoked and rejected** (`pre-receive hook declined`) |
|
||||
| Gate required | `enable_status_check:true`, contexts `["gate / gate (pull_request)"]` |
|
||||
| Approvals | `required_approvals:1`; `jakman85` holds `owner` |
|
||||
| Runner | `incluster-dind` online, now a **native sidecar** (`initContainers`, `restartPolicy: Always`, TLS-verified `docker info` startupProbe) |
|
||||
| Gate time | ~1m35s |
|
||||
| Plex | claimed 2026-09-29; `PlexOnlineToken` present on the `plex-config` PVC |
|
||||
|
||||
### What was corrected in the documentation
|
||||
|
||||
`CLAUDE.md` §2b — *never infer platform state from this repository* — is the durable outcome.
|
||||
The 2026-08-31 entry asserting an enforced gate is now annotated **in place**. The "Phase 0 /
|
||||
no cluster exists" framing across `ci/`, `clusters/` and `conformance/` READMEs is banner-marked
|
||||
as describing a live gap rather than a stage. All 37 runbooks now describe the PR flow. Two
|
||||
`.sops.yaml` headers and `catalog/plex.yaml` were corrected (comment edits verified not to break
|
||||
the SOPS MAC, and not to move the rendered golden).
|
||||
|
||||
### A claim of mine that shipped wrong, and is now corrected in place
|
||||
|
||||
PR #7 stated the sidecar conversion would remove the one-per-start dockerd line
|
||||
`http: TLS handshake error from 127.0.0.1: EOF`. **It does not.** Exactly one still appears,
|
||||
immediately after `API listen on [::]:2376` — it is now the startupProbe's first attempt,
|
||||
landing microseconds after dockerd begins listening and cut off by `timeoutSeconds: 1`; the
|
||||
next poll succeeds. Cosmetic, and deliberately not chased: raising the timeout to hide one
|
||||
benign startup line would slow every subsequent poll. The manifest comment now says so.
|
||||
|
||||
### The root cause nobody has fixed yet
|
||||
|
||||
Gitea's repo settings live in its own database — invisible to Flux and unreadable by the
|
||||
conformance gate. **No check exists that could have caught "protection off, runner gone."**
|
||||
`infra/gitea/` declares and can re-assert the desired state, but it is not a detector. A
|
||||
conformance check that reads the live Gitea API and fails on divergence is the highest-value
|
||||
remaining work in this repo, and it is the difference between this having been fixed and this
|
||||
being unable to recur.
|
||||
|
||||
### Next steps
|
||||
|
||||
1. **Write that Gitea-settings conformance check.** Everything else here is a symptom.
|
||||
2. Wire `vars.CLUSTER_AVAILABLE` + `secrets.CONFORMANCE_TRIGGER_KUBECONFIG` so **L2/L3 runs in
|
||||
CI**. A green PR currently proves the L0/L1 offline subset only — runtime behaviour is
|
||||
unverified on every merge.
|
||||
3. Re-run `infra/gitea/verify-then-enforce.sh` after any future act-runner change.
|
||||
4. `required_approvals: 1` means a PR you open as `jakman85` needs `gitea-admin` to approve.
|
||||
If that friction bites, either open agent work as `gitea-admin` and review as `jakman85`, or
|
||||
drop back to relying on the status check alone.
|
||||
|
||||
## 2026-09-29 (later still) — the enforcement we "turned on" was overridable by the account that merges
|
||||
|
||||
`enable_status_check: true` and `required_approvals: 1` were set earlier today and recorded as
|
||||
done. **They did not bite.** `block_admin_merge_override` was `false`, and Gitea lets repository
|
||||
admins override merge requirements without it. `jakman85` holds `owner`, so both requirements
|
||||
were bypassable by exactly the account that performs every merge.
|
||||
|
||||
**The evidence was already on the record and nobody read it:** PRs #5, #6, #7, #8 and #9 all
|
||||
merged while `required_approvals: 1` was in force, with **zero approvals recorded on any of
|
||||
them**. A one-line check (`/pulls/{n}/reviews`) would have shown it at any point.
|
||||
|
||||
Two process failures, both mine, both the same shape as the original defect:
|
||||
|
||||
1. **I verified with the wrong view.** `/branches/main` reports `protected`,
|
||||
`enable_status_check` and `required_approvals` but NOT `block_admin_merge_override`. It
|
||||
showed everything green while the rule was overridable. The authoritative object is
|
||||
`/branch_protections/main` — read it and enumerate its keys.
|
||||
2. **I probed for a field name I assumed existed.** This Gitea (1.24.3) has no
|
||||
`apply_to_admins`. Asking `jq` for it returns `null`, which reads exactly like "set to
|
||||
false" but means "key absent" — so a PATCH that silently did nothing looked like a real
|
||||
before/after. **`jq` returning null is not a value; it may be an absence.** Enumerate with
|
||||
`to_entries`.
|
||||
|
||||
Now `block_admin_merge_override: true`, applied and read back. `infra/gitea/require-gate.sh`
|
||||
sets it, and `infra/gitea/block-admin-override.sh` fixes it in isolation.
|
||||
|
||||
**This is still only a setting.** It is proven when a green PR is REFUSED a merge until
|
||||
approved — and that has not been observed yet. This PR is the test: if it merges without an
|
||||
approval, the fix did not work and the finding stands.
|
||||
|
||||
### The governance question this exposes
|
||||
|
||||
There are two accounts: `jakman85` (owner, the human) and `gitea-admin` (the agent). With
|
||||
approvals now genuinely required, an agent-authored PR is approved by the human — which is the
|
||||
intent. But a PR authored by `jakman85` needs `gitea-admin` to approve it, i.e. the automation
|
||||
account rubber-stamping the human. That is ceremony, not review. Options: keep it and accept
|
||||
the ceremony; drop `required_approvals` to 0 and rely on the status check, which is the gate
|
||||
that actually verifies something; or add a second human account. **This is a decision for the
|
||||
operator, not a default to be chosen silently.**
|
||||
@@ -7,12 +7,20 @@
|
||||
# substitution). A braced token quoted from the upstream shell script would be a live hazard here,
|
||||
# not prose, so the upstream conditions below are described in words.
|
||||
#
|
||||
# THE VALUE SHIPPED WITH THIS FILE IS A PLACEHOLDER, DELIBERATELY — the literal string
|
||||
# `replace-me-with-a-fresh-plex-tv-claim-token`, not a credential. A https://plex.tv/claim token has
|
||||
# a ~4-minute TTL, far shorter than commit -> push -> Flux reconcile -> pod-roll, so no token
|
||||
# committed here is ever still valid when the pod starts. This file exists to make the Secret — and
|
||||
# therefore the env reference — ALWAYS RESOLVABLE IN THE FLUX GRAPH, not to carry a working token.
|
||||
# The real token is injected by the operator inside that 4-minute window: runbooks/plex-claim.md.
|
||||
# THIS FILE NO LONGER HOLDS THE PLACEHOLDER. Corrected 2026-09-29: the server was CLAIMED on that
|
||||
# date, and the value here is the SPENT token from that claim. It is inert and safe to leave --
|
||||
# upstream gates the exchange on PlexOnlineToken being ABSENT from /config, and it is now present on
|
||||
# the plex-config PVC, so this value is a NO-OP on every start whatever it holds. Deleting the Secret
|
||||
# would NOT be safe (see below), which is why a spent token stays here rather than being blanked.
|
||||
#
|
||||
# The claim token is SHORT-LIVED -- minutes. Read the countdown that plex.tv/claim renders when it
|
||||
# issues one; do NOT budget against a number written in this tree. Earlier revisions of this comment
|
||||
# asserted a "~4-minute TTL" as established fact with no citation, and upstream plexinc/pms-docker
|
||||
# documents no expiry at all. That uncited number was load-bearing for the whole procedure design.
|
||||
#
|
||||
# This file exists to make the Secret — and therefore the env reference — ALWAYS RESOLVABLE IN THE
|
||||
# FLUX GRAPH, not to carry a working token. Re-claiming (only needed after a /config wipe) is
|
||||
# scripted: infra/plex/claim-ship.sh, measured at ~2m45s end to end.
|
||||
#
|
||||
# WHY SHIPPING A PLACEHOLDER IS SAFE (verified against plexinc/pms-docker branch `master`, the
|
||||
# s6-overlay v2 script root/etc/cont-init.d/40-plex-first-run, cross-checked byte-identical against
|
||||
|
||||
@@ -4,7 +4,13 @@
|
||||
# against Gitea. Flux decrypts it via the sops-age key at reconcile (apps Kustomization
|
||||
# decryption is wired in clusters/dev/kustomization.yaml). Rotate by resetting the token
|
||||
# in Gitea (admin -> actions -> runners) and re-encrypting a fresh value here.
|
||||
# Recover/inspect: infra/devcontainer/dev.sh run sops -d apps/gitea-actions-runner-secret.sops.yaml
|
||||
# ROTATE: ./infra/gitea/runner-token-rotate.sh -- it mints a fresh instance-level token from the
|
||||
# Gitea API and writes it here via `sops set`, without the value ever being printed.
|
||||
# Do NOT `sops -d` this file into a terminal, a log, or an agent transcript. To check only that
|
||||
# it still decrypts, redirect: `sops -d <file> > /dev/null`.
|
||||
# NOTE: a fresh token alone may not re-register the runner. act_runner registers ONCE and keeps
|
||||
# its identity in /data/.runner on the act-runner-data PVC; if Gitea no longer holds a matching
|
||||
# row it skips registration silently. Clearing that PVC is what forces a real re-registration.
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
|
||||
+69
-106
@@ -99,70 +99,79 @@ spec:
|
||||
# CI holds no cluster credentials — do not mount a SA token into the runner or
|
||||
# (via the shared netns) the job containers (R-SEC-02, docs/14).
|
||||
automountServiceAccountToken: false
|
||||
# ---- Docker-in-Docker: the job executor daemon (privileged) -------------------
|
||||
#
|
||||
# NATIVE SIDECAR (k3s v1.36; sidecar containers are GA). It is an initContainer with
|
||||
# `restartPolicy: Always`, which means: it starts BEFORE the regular containers, keeps
|
||||
# running alongside them, and is terminated after them. kubelet holds `runner` until
|
||||
# this container's startupProbe passes, so the ordering is enforced by the kubelet
|
||||
# instead of hand-rolled shell in the runner.
|
||||
#
|
||||
# This REPLACES two bounded wait-loops that used to live in the runner's `command:`.
|
||||
# Deleting them also deletes an envsubst hazard: clusters/base/apps-wave-1.yaml runs
|
||||
# postBuild substitution over this ENTIRE manifest, comments included, and those loops
|
||||
# depended on bare dollar-names surviving it untouched. There is now no shell here at
|
||||
# all, so there is nothing to escape or mis-escape.
|
||||
#
|
||||
# WHY THE PROBE IS `exec docker ... info` AND NOT `tcpSocket`. A tcpSocket probe would
|
||||
# re-open the exact gap the old Stage 1 existed to close: dockerd writes the TLS certs
|
||||
# and only THEN begins listening, so the port can accept a connection while the certs
|
||||
# are absent or incomplete -- and act_runner hard-exits on a missing
|
||||
# /certs/client/ca.pem. A full TLS-verified `docker info` proves BOTH that the certs
|
||||
# are usable and that the daemon answers.
|
||||
#
|
||||
# CORRECTION (measured 2026-09-29, after this conversion shipped): the commit that
|
||||
# introduced this claimed the conversion would REMOVE the one-per-start dockerd line
|
||||
# `http: TLS handshake error from 127.0.0.1: EOF`. IT DOES NOT. Exactly one still
|
||||
# appears, immediately after "API listen on [::]:2376" -- it is now this probe's FIRST
|
||||
# attempt, landing microseconds after dockerd begins listening and cut off by
|
||||
# timeoutSeconds: 1. The next poll succeeds. It is cosmetic, exactly as cosmetic as it
|
||||
# was before, and deliberately NOT chased: raising timeoutSeconds to hide one benign
|
||||
# startup line would slow every subsequent poll. Left documented rather than silenced.
|
||||
#
|
||||
# periodSeconds 1 x failureThreshold 60 preserves the previous 60s cap.
|
||||
initContainers:
|
||||
- name: dind
|
||||
restartPolicy: Always # <- this field is what makes it a SIDECAR
|
||||
image: docker:27-dind
|
||||
securityContext:
|
||||
privileged: true
|
||||
env:
|
||||
# Generate + publish TLS certs into the shared emptyDir; dockerd then listens
|
||||
# on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
|
||||
- name: DOCKER_TLS_CERTDIR
|
||||
value: /certs
|
||||
volumeMounts:
|
||||
- name: certs
|
||||
mountPath: /certs
|
||||
- name: dind-storage
|
||||
mountPath: /var/lib/docker
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
memory: 2Gi
|
||||
startupProbe:
|
||||
exec:
|
||||
command:
|
||||
- docker
|
||||
- --host=tcp://127.0.0.1:2376
|
||||
- --tlsverify
|
||||
- --tlscacert=/certs/client/ca.pem
|
||||
- --tlscert=/certs/client/cert.pem
|
||||
- --tlskey=/certs/client/key.pem
|
||||
- info
|
||||
periodSeconds: 1
|
||||
failureThreshold: 60
|
||||
containers:
|
||||
# ---- act_runner: registers with Gitea, dispatches jobs to the DinD daemon ----
|
||||
- name: runner
|
||||
image: gitea/act_runner:0.2.11
|
||||
# TWO-STAGE BOUNDED WAIT on the DinD sidecar, then exec the image's normal
|
||||
# entrypoint (/sbin/tini -- /opt/act/run.sh), which registers-from-env on first
|
||||
# boot and runs `act_runner daemon`.
|
||||
#
|
||||
# STAGE 1 — the certs must EXIST: act_runner reads /certs/client/ca.pem at startup
|
||||
# and hard-exits if it is missing (a startup race against the sidecar).
|
||||
# STAGE 2 — the daemon must actually ACCEPT a connection: the cert FILE appearing is
|
||||
# necessary but NOT sufficient. dockerd writes the certs and only THEN begins
|
||||
# listening on :2376, so a runner that proceeds on the file alone still hard-exits
|
||||
# with "cannot ping the docker daemon at tcp://localhost:2376" — observed
|
||||
# crashlooping 11 times in ~8 minutes on a fresh pod after a node bounce.
|
||||
# Both loops are bounded (60 x 1s) and exit 1 on timeout, so a genuinely dead sidecar
|
||||
# surfaces as a CrashLoopBackOff with a clear message instead of hanging forever.
|
||||
#
|
||||
# ENVSUBST SAFETY — the BARE $i / $j below are CORRECT. Do not "tidy" them into the
|
||||
# braced form or the double-dollar form; both are wrong here, for different reasons.
|
||||
# clusters/base/apps-wave-1.yaml carries postBuild.substituteFrom cluster-vars, so
|
||||
# Flux envsubst runs over this ENTIRE manifest — YAML comments very much included.
|
||||
# Its contract (drone/envsubst; the same one spelled out in clusters/base/apps-vms.yaml,
|
||||
# and re-verified against this exact file with `flux envsubst`, flux v2.4.0) is:
|
||||
# - A bare $name is NOT a reference to envsubst: it passes through untouched, and
|
||||
# that holds even when the name IS set in cluster-vars. So $i and $j reach the
|
||||
# cluster verbatim and both 60s bounds below are real, enforced bounds.
|
||||
# - $((i + 1)) is likewise untouched: "(" cannot begin a variable name.
|
||||
# - A BRACED dollar-name IS a reference and gets replaced — with the empty string
|
||||
# when the name is unset. One of those here would make a bound read [ "" -ge 60 ],
|
||||
# which errors, short-circuits the &&, and silently turns the wait UNBOUNDED.
|
||||
# - A DOUBLE-dollar braced name is envsubst's escape and survives as the braced
|
||||
# form — still wrong for us, because on any path where envsubst does NOT run
|
||||
# (a plain `kustomize build` or `kubectl apply -k` of this directory) the shell
|
||||
# reads that leading double-dollar as its own PID, so a bound would compare a
|
||||
# string like 1234{i} against 60. The bare form is the only spelling that is
|
||||
# correct on BOTH paths.
|
||||
# - A braced name containing punctuation is a HARD ERROR ("unable to parse variable
|
||||
# name") that fails this whole Kustomization — and this Kustomization gates every
|
||||
# merge in the repo. So never write one anywhere in this file, INCLUDING in a
|
||||
# comment. That is why this note names the three forms in words rather than
|
||||
# showing them: an earlier draft of it spelled them out literally and broke
|
||||
# `flux envsubst` on the file. (Same trap the indexed-expansion warning in
|
||||
# clusters/base/apps-vms.yaml describes.)
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- |
|
||||
# Stage 1: the sidecar's TLS client certs must appear in the shared emptyDir.
|
||||
i=0
|
||||
until [ -f /certs/client/ca.pem ]; do
|
||||
i=$((i + 1))
|
||||
[ "$i" -ge 60 ] && { echo "dind certs never appeared" >&2; exit 1; }
|
||||
echo "waiting for dind TLS certs ($i)"; sleep 1
|
||||
done
|
||||
# Stage 2: dockerd must actually accept a connection on :2376. busybox nc in this
|
||||
# Alpine-based image (gitea/act_runner:0.2.11) supports -z (scan) and -w (timeout).
|
||||
j=0
|
||||
until nc -z -w1 127.0.0.1 2376; do
|
||||
j=$((j + 1))
|
||||
[ "$j" -ge 60 ] && { echo "dind daemon never accepted a connection on 2376" >&2; exit 1; }
|
||||
echo "waiting for dind daemon on :2376 ($j)"; sleep 1
|
||||
done
|
||||
exec /sbin/tini -- /opt/act/run.sh
|
||||
# No `command:` override. The image's own entrypoint (/sbin/tini -- /opt/act/run.sh)
|
||||
# registers from env on first boot and runs `act_runner daemon`. The two wait-loops
|
||||
# that used to live here are gone: kubelet now holds this container until the dind
|
||||
# sidecar's startupProbe passes (see initContainers above).
|
||||
env:
|
||||
- name: GITEA_INSTANCE_URL
|
||||
value: https://gitea.${BASE_DOMAIN}
|
||||
@@ -198,52 +207,6 @@ spec:
|
||||
memory: 128Mi
|
||||
limits:
|
||||
memory: 512Mi
|
||||
# ---- Docker-in-Docker: the job executor daemon (privileged) ----
|
||||
#
|
||||
# FOLLOW-UP (deliberately NOT bundled into the readiness fix above — own change):
|
||||
# Convert this container into a NATIVE SIDECAR and delete both shell wait-loops.
|
||||
# k3s is now v1.36, where sidecar containers are GA, so kubelet can enforce the
|
||||
# ordering that the loops above hand-roll — which also removes the envsubst hazard
|
||||
# from this file entirely (no shell, no $i/$j, nothing to escape or mis-escape).
|
||||
# Shape to build:
|
||||
# 1. Move this whole container into `initContainers:` as the FIRST entry.
|
||||
# 2. Add `restartPolicy: Always` to it. That single field is what makes an
|
||||
# initContainer a sidecar: it starts before the regular containers, keeps
|
||||
# running alongside them, and is terminated after them.
|
||||
# 3. Give it a `startupProbe` that succeeds only when dockerd accepts a TLS
|
||||
# connection — e.g. `exec` a `docker -H tcp://127.0.0.1:2376 --tlsverify
|
||||
# --tlscacert /certs/client/ca.pem --tlscert /certs/client/cert.pem
|
||||
# --tlskey /certs/client/key.pem info`. Do NOT use a bare `tcpSocket` probe:
|
||||
# it would re-open the same "port accepting but TLS/certs not ready" gap that
|
||||
# Stage 1 above exists to close. `periodSeconds: 1` + `failureThreshold: 60`
|
||||
# preserves today's 60s cap.
|
||||
# 4. kubelet then holds the `runner` container until that startupProbe passes, so
|
||||
# `runner` can drop its `command:` override completely and run the image's
|
||||
# default entrypoint.
|
||||
# Why it is a separate change: this Deployment is the ENFORCED MERGE GATE for this
|
||||
# entire repository. If the conversion is wrong, nothing can merge — so it must land
|
||||
# alone, where the failure is unambiguous and not entangled with anything else, and
|
||||
# only after a real workflow run has been observed green on the converted pod.
|
||||
- name: dind
|
||||
image: docker:27-dind
|
||||
securityContext:
|
||||
privileged: true
|
||||
env:
|
||||
# Generate + publish TLS certs into the shared emptyDir; dockerd then listens
|
||||
# on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
|
||||
- name: DOCKER_TLS_CERTDIR
|
||||
value: /certs
|
||||
volumeMounts:
|
||||
- name: certs
|
||||
mountPath: /certs
|
||||
- name: dind-storage
|
||||
mountPath: /var/lib/docker
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
memory: 2Gi
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
|
||||
+11
-5
@@ -38,11 +38,17 @@ env:
|
||||
# UNCLAIMED, so this is wired; it was previously left out on the reasoning below, and only one
|
||||
# half of that reasoning survived contact with the upstream source.
|
||||
#
|
||||
# THE 4-MINUTE TTL IS STILL TRUE, and it is now the OPERATIONAL constraint rather than a reason to
|
||||
# abstain. A https://plex.tv/claim token expires in ~4 minutes — far shorter than commit -> push ->
|
||||
# Flux reconcile -> pod-roll if that cycle is left to its own timers. So the committed value is a
|
||||
# documented PLACEHOLDER and the live token is injected by the operator inside that window, with an
|
||||
# explicit `flux reconcile` to collapse the wait: runbooks/plex-claim.md is the timed procedure.
|
||||
# A SHORT TTL IS THE OPERATIONAL CONSTRAINT, not a reason to abstain. A https://plex.tv/claim token
|
||||
# expires in MINUTES — read the countdown the claim page renders; do not budget against a number
|
||||
# written in this tree. (This comment previously asserted "~4 minutes" as established fact. It was
|
||||
# UNCITED, upstream plexinc/pms-docker documents no expiry at all, and that number was load-bearing
|
||||
# for the whole procedure design — including a since-disproven claim that the delivery pipeline was
|
||||
# too slow to fit inside it. Measured 2026-09-29: the full loop is ~2m45s and fits.)
|
||||
#
|
||||
# The live token is injected by the operator inside that window, with an explicit `flux reconcile`
|
||||
# to collapse the wait. Scripted end to end as infra/plex/claim-ship.sh; runbooks/plex-claim.md is
|
||||
# the procedure. SERVER CLAIMED 2026-09-29 — this env is now a permanent no-op (see below), and the
|
||||
# committed value is the spent token from that claim rather than the original placeholder.
|
||||
#
|
||||
# ONE-SHOT, AND SAFE TO SIT HERE EXPIRED. Verified from plexinc/pms-docker's own first-run script
|
||||
# (branch `master`, root/etc/cont-init.d/40-plex-first-run, confirmed byte-identical to the file in
|
||||
|
||||
@@ -1,5 +1,19 @@
|
||||
# `ci/` — portable merge gate (runnable today, without Gitea)
|
||||
|
||||
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
||||
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
||||
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
||||
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
||||
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
||||
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
||||
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
||||
> coverage hole, not a phase.
|
||||
>
|
||||
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
||||
> documentation rather than measurement, and `main` was unprotected with zero runners
|
||||
> registered the whole time. **Query the live API before asserting platform state.**
|
||||
|
||||
|
||||
**Task T-0.2 · R-TEST-02 (reproducible, single-command) · Owner WS-GITOPS (CI).**
|
||||
|
||||
The same merge gate the Gitea Actions workflow runs
|
||||
|
||||
@@ -61,5 +61,45 @@ if [ -d "${CONF_ART}/coverage" ]; then
|
||||
cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# DID IT ACTUALLY RUN? -- the guard without which this stage FAKES A GREEN.
|
||||
#
|
||||
# run.sh exits 0 in "scaffold mode" when it cannot reach a cluster: it emits a JUnit file
|
||||
# whose every testcase is <skipped> and returns success. That is correct FOR run.sh -- an
|
||||
# honest skip is its documented no-cluster behaviour. It is catastrophic HERE, because
|
||||
# reaching this point means the environment DECLARED CLUSTER_AVAILABLE=true, and this stage
|
||||
# would otherwise print "L2/L3 conformance passed" over 54 tests that never executed.
|
||||
#
|
||||
# MEASURED 2026-09-29, before this guard existed:
|
||||
# CLUSTER_AVAILABLE=true ./ci/conformance.sh -> rc=0
|
||||
# "[junit] wrote skipped JUnit for 54 scaffolded test(s)"
|
||||
# "[ ok ] L2/L3 conformance passed for touched subset"
|
||||
# One Gitea variable away from a fake green on an enforcing merge gate. That is exactly the
|
||||
# HARD RULE 4 violation this repo's whole gate design exists to prevent, and nothing caught
|
||||
# it because nobody had ever set the variable.
|
||||
#
|
||||
# A skip while a cluster is DECLARED is a contradiction: either the declaration is wrong or
|
||||
# the cluster is unreachable. The merge-gate decision matrix already says this case is BLOCK
|
||||
# ("conformance skipped while CLUSTER_AVAILABLE=true -> BLOCK (it should have run)"). This
|
||||
# makes the stage itself honour that instead of reporting a pass.
|
||||
JUNIT="${CONF_ART}/report.junit.xml"
|
||||
if [ -f "$JUNIT" ]; then
|
||||
total="$(grep -o '<testcase' "$JUNIT" | wc -l | tr -d ' ')"
|
||||
skipped="$(grep -o '<skipped' "$JUNIT" | wc -l | tr -d ' ')"
|
||||
if [ "${total:-0}" -gt 0 ] && [ "${skipped:-0}" -ge "${total:-0}" ]; then
|
||||
err "CLUSTER_AVAILABLE=true, but the L2/L3 suite SKIPPED every one of its ${total} tests"
|
||||
err " (scaffold mode -- run.sh could not reach a cluster and exited 0)."
|
||||
err " This stage will NOT report that as a pass. Either the cluster is unreachable from"
|
||||
err " this runner, or CLUSTER_AVAILABLE is set when it should not be."
|
||||
err " NOTE: the in-cluster trigger path is still UNIMPLEMENTED -- see"
|
||||
err " docs/plans/l2-l3-conformance-ci-wiring.md. Setting CLUSTER_AVAILABLE=true does not"
|
||||
err " by itself make L2/L3 run; it only makes this stage attempt a LOCAL run that the CI"
|
||||
err " runner has no credential for (automountServiceAccountToken: false, by design)."
|
||||
summary "- conformance-l2-l3: **BLOCKED — declared available but every test skipped** (not a pass)"
|
||||
exit 1
|
||||
fi
|
||||
log "L2/L3 executed: ${total} testcase(s), ${skipped} skipped"
|
||||
fi
|
||||
|
||||
ok "L2/L3 conformance passed for touched subset"
|
||||
summary "- conformance-l2-l3: **passed**"
|
||||
@@ -1,5 +1,19 @@
|
||||
# clusters/
|
||||
|
||||
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
||||
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
||||
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
||||
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
||||
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
||||
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
||||
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
||||
> coverage hole, not a phase.
|
||||
>
|
||||
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
||||
> documentation rather than measurement, and `main` was unprotected with zero runners
|
||||
> registered the whole time. **Query the live API before asserting platform state.**
|
||||
|
||||
|
||||
Flux entrypoints and the environment-overlay model (**R-ENV-02**, **R-GITOPS-01/02/03**).
|
||||
|
||||
Governing spec: [`docs/05-gitops-and-delivery.md`](../../home-network-k3s/docs/05-gitops-and-delivery.md),
|
||||
|
||||
@@ -1,5 +1,19 @@
|
||||
# conformance/ — the home-k3s conformance harness (WS-TEST)
|
||||
|
||||
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
||||
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
||||
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
||||
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
||||
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
||||
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
||||
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
||||
> coverage hole, not a phase.
|
||||
>
|
||||
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
||||
> documentation rather than measurement, and `main` was unprotected with zero runners
|
||||
> registered the whole time. **Query the live API before asserting platform state.**
|
||||
|
||||
|
||||
This subtree is the **definition of "done"** for the whole platform. Nothing is accepted
|
||||
on a demo, a screenshot, or "it reconciled": every requirement is proven by an
|
||||
**automated, reproducible, behavior-based test** tagged with the requirement ID(s) it
|
||||
|
||||
@@ -1,5 +1,19 @@
|
||||
# In-cluster conformance Job (L2/L3)
|
||||
|
||||
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
||||
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
||||
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
||||
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
||||
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
||||
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
||||
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
||||
> coverage hole, not a phase.
|
||||
>
|
||||
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
||||
> documentation rather than measurement, and `main` was unprotected with zero runners
|
||||
> registered the whole time. **Query the live API before asserting platform state.**
|
||||
|
||||
|
||||
Per **ADR-0011** and **docs/14 "Where each layer runs"**: L0/L1 run in CI without a
|
||||
cluster, but **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job**
|
||||
on the dev/test cluster that **CI triggers and waits for**. CI holds **no** cluster
|
||||
|
||||
@@ -38,8 +38,21 @@ agents. So the credential model must be decided by the operator before anything
|
||||
| Runner deployment | `apps/wave-1/gitea-actions.yaml` | act_runner + DinD (Docker executor). **`automountServiceAccountToken: false`** — the runner and its job containers hold NO SA token. Broad egress allowed (`- {}`), so it can *reach* the kube-API VIP `192.168.1.49:6443`, but has no credential. |
|
||||
| Secrets encryption | `.sops.yaml` (repo root) | age recipient `age175l8m…mfzg`; encrypts `data`/`stringData` of `*.sops.yaml` under `platform|apps|catalog/**` and `secrets/**`. Private key is the in-cluster `sops-age` Secret Flux uses. |
|
||||
|
||||
**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the dev cluster
|
||||
is LIVE (Phase 6). So `CLUSTER_AVAILABLE` is the only remaining gate, exactly as the blueprint says.
|
||||
**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the cluster is
|
||||
LIVE (and it is now PROD; dev was decommissioned 2026-09-12).
|
||||
|
||||
> **CORRECTION (2026-09-29): `CLUSTER_AVAILABLE` is NOT "the only remaining gate", and setting it
|
||||
> today is actively DANGEROUS.** Measured: with `CLUSTER_AVAILABLE=true` and no reachable cluster,
|
||||
> `conformance/run.sh` exits 0 in scaffold mode having marked all 54 tests `<skipped>`, and
|
||||
> `ci/conformance.sh` then printed `[ ok ] L2/L3 conformance passed for touched subset` — **rc 0, a
|
||||
> fake green**, on a merge gate that is now an enforcing required status check. The CI runner has
|
||||
> no cluster credential at all (`automountServiceAccountToken: false`, by design), so the "local
|
||||
> run" this stage attempts can never succeed; it can only skip and, before the guard added on
|
||||
> 2026-09-29, lie about it.
|
||||
>
|
||||
> `ci/conformance.sh` now BLOCKS (exit 1) when a cluster is declared but every test skipped. **Do
|
||||
> not set `CLUSTER_AVAILABLE=true` until §3's trigger path actually exists** — the variable does
|
||||
> not enable L2/L3, it only changes which way this stage fails.
|
||||
|
||||
---
|
||||
|
||||
|
||||
+59
-1
@@ -43,7 +43,65 @@ list. That is precisely the observed symptom: zero runners, pod running, no erro
|
||||
looks at. Clearing the PVC is what forces a real re-registration; the script prints the
|
||||
commands.
|
||||
|
||||
## Sequence — order matters
|
||||
## Sequence — DONE 2026-09-29
|
||||
|
||||
All four steps are complete and each was verified, not assumed:
|
||||
|
||||
| Step | State |
|
||||
|---|---|
|
||||
| 1. Runner registered | ✅ `incluster-dind` online. Root cause of its absence was a stale `/data/.runner` identity; clearing the PVC forced re-registration |
|
||||
| 2. Gate observed green on a real PR | ✅ `gate / gate (pull_request)` ~1m35s |
|
||||
| 3. `enable_status_check: true` | ✅ context `gate / gate (pull_request)` — that one ONLY |
|
||||
| 4. Collaborator + approvals | ✅ `jakman85` is owner; `required_approvals: 1` |
|
||||
|
||||
Direct push to `main` was **provoked and rejected** (`pre-receive hook declined`) as
|
||||
`gitea-admin`, a site admin — so push protection is not merely declared, it holds for the
|
||||
account agents authenticate as.
|
||||
|
||||
### 5. Block the admin override — the step the first pass MISSED
|
||||
|
||||
`enable_status_check` and `required_approvals` were set on 2026-09-29 and **still did not
|
||||
bite**, because `block_admin_merge_override` was `false`. Gitea lets repo admins override
|
||||
merge requirements without it, and `jakman85` holds `owner` — so the two settings that looked
|
||||
like the gate were overridable by precisely the account that merges.
|
||||
|
||||
**The evidence was sitting in plain sight:** PRs #5 through #9 all merged with
|
||||
`required_approvals: 1` in force and **zero approvals recorded on any of them**. Checking the
|
||||
*branch* view (`/branches/main`) hid this — it does not expose the field. Read the rule object
|
||||
(`/branch_protections/main`) and enumerate its keys.
|
||||
|
||||
> **Do not probe for field names you assume exist.** This Gitea (1.24.3) has no
|
||||
> `apply_to_admins`; asking `jq` for it returns `null`, which reads like "set to false" but
|
||||
> means "key absent". Enumerate with `to_entries` and act on what is really there.
|
||||
|
||||
Now `block_admin_merge_override: true`. **That is still only a setting, not evidence** — it is
|
||||
proven when a green PR is REFUSED a merge until approved, and not before.
|
||||
|
||||
### `require-gate.sh` / `verify-then-enforce.sh`
|
||||
|
||||
`verify-then-enforce.sh` is the re-runnable form of the whole safe order. It merges nothing —
|
||||
merging is a human's reviewed act — but it forces the Flux reconcile and then **aborts** unless
|
||||
the runner pod is healthy, a runner is online, and a real workflow goes green on it, before
|
||||
enabling enforcement and probing the push. `require-gate.sh` is the enforcement step alone.
|
||||
|
||||
Re-run `verify-then-enforce.sh` after any future change to the act-runner Deployment.
|
||||
|
||||
### Why these scripts exist at all, and what is still missing
|
||||
|
||||
Gitea's repo settings live in **its own database**, not in anything Flux reconciles or the
|
||||
conformance gate can read. Nothing detects drift from what this directory declares. That is
|
||||
the root cause of the entire 2026-09-29 session: branch protection was off and the runner was
|
||||
gone for a month while the log recorded both as working, and **no check existed that could
|
||||
have caught it.**
|
||||
|
||||
These scripts encode the desired state and can re-assert it, but they are not a detector. The
|
||||
durable fix is a conformance check that reads the live Gitea API and fails when it diverges
|
||||
from what is declared here — turning a class of silent, invisible failure into a loud one.
|
||||
**That check does not exist yet and is the highest-value remaining work in this directory.**
|
||||
|
||||
## Historical sequence (kept for context)
|
||||
|
||||
### Original ordering notes
|
||||
|
||||
1. **Get an Actions runner registered** — `runner-token-rotate.sh`, then clear the PVC so
|
||||
the runner actually re-registers, then confirm it shows `online` in the API. Do not
|
||||
|
||||
Executable
+36
@@ -0,0 +1,36 @@
|
||||
#!/usr/bin/env bash
|
||||
# Stop repository admins from overriding the merge requirements on `main`.
|
||||
#
|
||||
# WHY. Gitea lets repo admins override branch-protection merge requirements unless
|
||||
# `block_admin_merge_override` is set. It was FALSE, and jakman85 holds `owner` -- so
|
||||
# required_approvals and the required status check were both overridable by the one account
|
||||
# that actually merges. Evidence: PRs #5-#9 all merged with ZERO approvals recorded against
|
||||
# `required_approvals: 1`.
|
||||
#
|
||||
# NOTE ON FIELD NAMES: this Gitea (1.24.3) has NO `apply_to_admins` field. Querying it with jq
|
||||
# returns null, which reads like "set to false" but actually means "key absent". Enumerate the
|
||||
# real keys (`to_entries`) rather than probing names you assume exist.
|
||||
#
|
||||
# Recoverable: an owner can PATCH this back to false if it gets in the way.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
||||
B="https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main"
|
||||
|
||||
echo "==> before: block_admin_merge_override = $(api "$B" | jq -r .block_admin_merge_override)"
|
||||
api -X PATCH -H 'Content-Type: application/json' \
|
||||
-d '{"block_admin_merge_override": true}' "$B" >/dev/null
|
||||
AFTER=$(api "$B" | jq -r .block_admin_merge_override)
|
||||
echo "==> after: block_admin_merge_override = $AFTER"
|
||||
[ "$AFTER" = true ] || { echo "PATCH did not take -- do not record this as enforced." >&2; exit 1; }
|
||||
|
||||
api "$B" | jq -r '{enable_push,enable_status_check,status_check_contexts,required_approvals,block_admin_merge_override}'
|
||||
cat <<'NOTE'
|
||||
|
||||
NOW PROVE IT -- the setting is not the evidence. On the next PR:
|
||||
* let the gate go green, then try to merge WITHOUT approving it.
|
||||
* Gitea must REFUSE. If it merges, this still is not doing its job.
|
||||
Do not record approvals as enforced until you have seen a merge refused.
|
||||
NOTE
|
||||
Executable
+60
@@ -0,0 +1,60 @@
|
||||
#!/usr/bin/env bash
|
||||
# Items 1 + 2: make the gate a REQUIRED status check and require one approval.
|
||||
#
|
||||
# RUN THIS LAST. Prerequisites, in order:
|
||||
# a) PR #7 (dind native sidecar) is MERGED, and
|
||||
# b) you have SEEN a real workflow run go green on the converted pod.
|
||||
# The runner Deployment is the merge gate for the whole repo. Turning on a required check
|
||||
# while the runner is broken means no PR can merge -- including the revert. (Recoverable:
|
||||
# an owner can set enable_status_check back to false. But do not rely on that.)
|
||||
#
|
||||
# Requires exactly ONE context: `gate / gate (pull_request)`.
|
||||
# * NOT `gate / conformance (*)` -- that job is `if: false`, and a skipped Gitea job reports
|
||||
# SUCCESS ("Has been skipped"). Requiring it would pass VACUOUSLY forever while nothing
|
||||
# ran, which under HARD RULE 4 is worse than a deadlock: a deadlock is loud.
|
||||
# block_admin_merge_override: true is LOAD-BEARING and was missing on the first pass. Gitea
|
||||
# lets repo admins override merge requirements without it, so required_approvals and the
|
||||
# required status check are both bypassable by exactly the account that does the merging.
|
||||
# Measured: PRs #5-#9 merged with ZERO approvals recorded against required_approvals: 1.
|
||||
#
|
||||
# FIELD NAMES: this Gitea (1.24.3) has NO `apply_to_admins`. Asking jq for it returns null,
|
||||
# which reads like "false" but means "key absent" -- enumerate real keys with `to_entries`
|
||||
# instead of probing names you assume exist. That mistake cost a wasted PATCH.
|
||||
#
|
||||
# * NOT the `(push)` variant -- after PR #6 there are no per-branch push runs at all, so a
|
||||
# PR would never produce that context.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
||||
|
||||
echo "==> updating branch protection on main"
|
||||
jq -n '{
|
||||
enable_push: false,
|
||||
enable_status_check: true,
|
||||
status_check_contexts: ["gate / gate (pull_request)"],
|
||||
required_approvals: 1,
|
||||
block_on_rejected_reviews: true,
|
||||
dismiss_stale_approvals: true,
|
||||
block_on_outdated_branch: false,
|
||||
block_admin_merge_override: true
|
||||
}' | api -X PATCH -H 'Content-Type: application/json' --data @- \
|
||||
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main" >/dev/null
|
||||
|
||||
echo "==> verifying"
|
||||
api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \
|
||||
| jq -r '{protected,user_can_push,enable_status_check,status_check_contexts,required_approvals}'
|
||||
|
||||
cat <<'NOTE'
|
||||
|
||||
NOTE ON required_approvals: 1 -- Gitea will not let an author approve their own PR.
|
||||
* Agent-opened PRs (author gitea-admin) are approved by jakman85. Fine.
|
||||
* PRs YOU open as jakman85 need gitea-admin to approve. If that is awkward, either open
|
||||
agent work as gitea-admin and review as jakman85, or set required_approvals back to 0
|
||||
and rely on the status check alone.
|
||||
|
||||
Then prove it, rather than trusting the API:
|
||||
git commit --allow-empty -m probe && git push origin main # EXPECT: rejected
|
||||
git reset --hard HEAD~1
|
||||
NOTE
|
||||
Executable
+83
@@ -0,0 +1,83 @@
|
||||
#!/usr/bin/env bash
|
||||
# verify-then-enforce.sh — run this AFTER you have merged PRs 8, 5, 6, 7 in that order.
|
||||
#
|
||||
# It does NOT merge anything. Merging is your reviewed act (CLAUDE.md: a human approves and
|
||||
# merges). This script does the mechanical half: prove the converted runner actually works,
|
||||
# and only then turn on enforcement.
|
||||
#
|
||||
# WHY THIS ORDER. The act-runner Deployment IS the merge gate for this repository. Enabling a
|
||||
# required status check while the runner is broken means NO PR can merge -- including the
|
||||
# revert. So PR #7 lands while the gate is still advisory, is PROVEN green on a real run, and
|
||||
# only then does enforcement go on.
|
||||
#
|
||||
# Aborts at the first checkpoint that does not pass. Safe to re-run.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
|
||||
REPO_ROOT=/var/home/keith/source/home-network-k3s-impl
|
||||
D="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
cd "$REPO_ROOT"
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
||||
A="https://${HOST}/api/v1/repos/${OWNER}/${REPO}"
|
||||
say() { printf '\n\033[1m==> %s\033[0m\n' "$1"; }
|
||||
die() { printf '\n\033[31mABORT: %s\033[0m\n' "$1" >&2; exit 1; }
|
||||
|
||||
say "PRECHECK — all four PRs must be merged first"
|
||||
for n in 8 5 6 7; do
|
||||
m=$(api "$A/pulls/$n" | jq -r .merged)
|
||||
printf ' PR #%s merged=%s\n' "$n" "$m"
|
||||
[ "$m" = true ] || die "PR #$n is not merged yet. Merge 8, 5, 6, 7 in that order, then re-run."
|
||||
done
|
||||
|
||||
say "STEP 1 — force Flux to apply the sidecar now, not on the 10m interval"
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-1 --with-source'
|
||||
|
||||
say "STEP 2 — CHECKPOINT: the converted pod must come up healthy"
|
||||
./infra/devcontainer/dev.sh run bash -c '
|
||||
kubectl --context k3s-prod -n gitea-actions rollout status deploy/act-runner --timeout=180s
|
||||
echo "--- sidecar (initContainers) ---"
|
||||
kubectl --context k3s-prod -n gitea-actions get deploy act-runner \
|
||||
-o jsonpath="{range .spec.template.spec.initContainers[*]}{.name} restartPolicy={.restartPolicy}{\"\n\"}{end}"
|
||||
kubectl --context k3s-prod -n gitea-actions get pod -l app.kubernetes.io/name=act-runner' \
|
||||
|| die "converted act-runner pod is not healthy -- REVERT PR #7 before anything else"
|
||||
|
||||
say "STEP 3 — CHECKPOINT: a runner must be ONLINE again"
|
||||
n=0
|
||||
for i in $(seq 1 40); do
|
||||
n=$(api "https://${HOST}/api/v1/admin/actions/runners" | jq -r '[(.runners//[])[]|select(.status=="online")]|length')
|
||||
printf '\r online runners: %s (%ss) ' "$n" "$((i*5))"
|
||||
[ "${n:-0}" -ge 1 ] && { echo; break; }
|
||||
sleep 5
|
||||
done
|
||||
[ "${n:-0}" -ge 1 ] || die "no runner online after the conversion -- REVERT PR #7"
|
||||
|
||||
say "STEP 4 — CHECKPOINT: a REAL workflow must go green ON THE CONVERTED POD"
|
||||
MAIN_SHA=$(api "$A/branches/main" | jq -r .commit.id)
|
||||
echo " watching the post-merge push run on main (${MAIN_SHA:0:8})..."
|
||||
st=""
|
||||
for i in $(seq 1 60); do
|
||||
st=$(api "$A/commits/$MAIN_SHA/status" | jq -r .state)
|
||||
printf '\r main gate: %-8s (%ss) ' "$st" "$((i*10))"
|
||||
case "$st" in success) echo; break;; failure|error) die "gate FAILED on the converted pod -- REVERT PR #7";; esac
|
||||
sleep 10
|
||||
done
|
||||
[ "$st" = success ] || die "gate never completed on the converted pod -- REVERT PR #7; do NOT enable enforcement"
|
||||
echo " the converted sidecar ran a real job green. Enforcement is now safe."
|
||||
|
||||
say "STEP 5 — require the gate + one approval"
|
||||
"$D/require-gate.sh"
|
||||
|
||||
say "STEP 6 — prove a direct push is rejected (do not trust the API's word)"
|
||||
git fetch -q origin && git checkout -q main && git reset -q --hard origin/main
|
||||
git commit -q --allow-empty -m 'protection probe'
|
||||
if git push origin main 2>&1 | grep -q 'protected branch'; then
|
||||
echo " CONFIRMED: push to main rejected by the pre-receive hook."
|
||||
else
|
||||
echo " WARNING: the push was NOT rejected -- protection is not doing what it claims."
|
||||
fi
|
||||
git reset -q --hard origin/main
|
||||
|
||||
say "FINAL STATE"
|
||||
api "$A/branches/main" | jq -r '{protected,user_can_push,enable_status_check,status_check_contexts,required_approvals}'
|
||||
Executable
+94
@@ -0,0 +1,94 @@
|
||||
#!/usr/bin/env bash
|
||||
# plex-claim-ship.sh — run this IMMEDIATELY AFTER editing the SOPS secret with a fresh
|
||||
# plex.tv claim token. It drives the whole CI/CD loop end to end and prints a running
|
||||
# stopwatch so you can see the budget being spent.
|
||||
#
|
||||
# branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verify
|
||||
#
|
||||
# It never sees, prints, or handles the token: the file is already encrypted by sops.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev
|
||||
OWNER=Home-Network
|
||||
REPO=home-network-k3s-impl
|
||||
SECRET=apps/wave-0/plex-claim-secret.sops.yaml
|
||||
T0=$(date +%s)
|
||||
el() { printf '\n[T+%02d:%02d] %s\n' $(( ($(date +%s)-T0)/60 )) $(( ($(date +%s)-T0)%60 )) "$1"; }
|
||||
|
||||
cd "$(git rev-parse --show-toplevel)"
|
||||
|
||||
# --- preflight: refuse to ship a file that is not what we think it is -------------
|
||||
git diff --quiet -- "$SECRET" && { echo "ERROR: $SECRET is unchanged. Edit it first:"; \
|
||||
echo " infra/devcontainer/dev.sh run sops $SECRET"; exit 1; }
|
||||
grep -q 'PLEX_CLAIM: ENC\[' "$SECRET" || { echo "ERROR: $SECRET is not encrypted. REFUSING."; exit 1; }
|
||||
[ -z "$(git status --porcelain | grep -v " $SECRET\$")" ] || echo "NOTE: other files are dirty; only $SECRET will be committed."
|
||||
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
||||
|
||||
# Branch prefix is now cosmetic. It used to matter: gate.yml's push trigger covered
|
||||
# feature/** fix/** docs/** chore/** but not feat/**, and with act_runner at capacity 1 a
|
||||
# matching prefix fired a redundant run that serialised ahead of the PR run. Those
|
||||
# per-branch push triggers were deleted, so no prefix produces a duplicate run any more.
|
||||
BRANCH="feat/plex-claim-token-$(date +%H%M%S)"
|
||||
|
||||
el "committing on $BRANCH"
|
||||
git checkout -q -b "$BRANCH"
|
||||
git add "$SECRET"
|
||||
git commit -q -m 'chore(plex): fresh plex.tv claim token
|
||||
|
||||
Injected inside the plex.tv claim window per runbooks/plex-claim.md.'
|
||||
SHA=$(git rev-parse HEAD)
|
||||
|
||||
el "pushing"
|
||||
git push -q -u origin "$BRANCH"
|
||||
|
||||
el "opening PR"
|
||||
PR=$(jq -n --arg h "$BRANCH" --arg b main --arg t 'chore(plex): fresh plex.tv claim token' \
|
||||
'{head:$h,base:$b,title:$t,body:"Token injection inside the claim window. Wiring already reviewed in #1."}' \
|
||||
| api -X POST -H 'Content-Type: application/json' --data @- \
|
||||
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls" | jq -r '.number')
|
||||
[ -n "$PR" ] && [ "$PR" != "null" ] || { echo "PR creation failed"; exit 1; }
|
||||
echo " PR #$PR"
|
||||
|
||||
el "waiting for the gate (expect ~1m40s)"
|
||||
for i in $(seq 1 60); do
|
||||
st=$(api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/commits/${SHA}/status" | jq -r '.state')
|
||||
printf '\r gate=%s [T+%ss] ' "$st" "$(( $(date +%s)-T0 ))"
|
||||
case "$st" in success) echo; break;; failure|error) echo; echo "GATE FAILED — stopping."; exit 1;; esac
|
||||
sleep 5
|
||||
done
|
||||
[ "$st" = "success" ] || { echo; echo "gate did not finish in time"; exit 1; }
|
||||
|
||||
el "merging PR #$PR"
|
||||
api -X POST -H 'Content-Type: application/json' -d '{"Do":"merge"}' \
|
||||
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls/${PR}/merge" >/dev/null
|
||||
git checkout -q main && git pull -q --ff-only origin main
|
||||
|
||||
el "forcing the Flux reconcile (--with-source, or it applies the OLD revision)"
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-0 --with-source'
|
||||
|
||||
# MANDATORY here. Only the Secret ciphertext changed, so the Deployment pod template is
|
||||
# byte-identical and NOTHING restarts on its own. PLEX_CLAIM is read once, at container
|
||||
# start. Skipping this is failure mode #2 in runbooks/plex-claim.md.
|
||||
el "rolling the pod (mandatory — a changed Secret restarts nothing)"
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex rollout restart deployment/plex && \
|
||||
kubectl --context k3s-prod -n plex rollout status deployment/plex --timeout=120s'
|
||||
|
||||
el "verdict"
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex logs deploy/plex --since=3m | grep -i -e claim -e token || true'
|
||||
cat <<'NOTE'
|
||||
|
||||
"Attempting to obtain server token..." + "Token obtained successfully" -> CLAIMED
|
||||
"Attempting..." with no success line -> token expired; retry, nothing is broken
|
||||
neither line -> already claimed, or env did not reach the container
|
||||
|
||||
Confirm it persisted (prints the attribute NAME only, never its value):
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c \
|
||||
"grep -o PlexOnlineToken= \"/config/Library/Application Support/Plex Media Server/Preferences.xml\" | head -1"'
|
||||
NOTE
|
||||
+7
-4
@@ -33,7 +33,8 @@ negative is how a CLI quirk becomes real data loss. Write `backups.velero.io`,
|
||||
`backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`,
|
||||
`volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
`git push origin main` **is** the deploy; there is no other apply step. Durable changes go through git.
|
||||
Merging a PR into `main` **is** the deploy; there is no other apply step. `main` is protected, so a
|
||||
direct push is rejected — durable changes go through a branch, a PR, and a green gate.
|
||||
|
||||
## Alerts you should not trust the silence of (measured 2026-09-13)
|
||||
|
||||
@@ -109,9 +110,11 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
|
||||
every green signal missed.
|
||||
|
||||
### Operator procedures (no alert links here)
|
||||
- **`plex-claim`** — claiming the Plex Media Server. The plex.tv claim token lives ~4 minutes, so this
|
||||
is a timed sequence: token → `sops` → push → **forced** `flux reconcile` → pod roll. Includes why a
|
||||
changed Secret does not restart anything, and why an expired token is harmless.
|
||||
- **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only
|
||||
applies after a `/config` wipe. The claim token is short-lived — read the countdown on
|
||||
plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
|
||||
token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
|
||||
Includes why a changed Secret does not restart anything, and why an expired token is harmless.
|
||||
|
||||
### Virtualisation
|
||||
- **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.
|
||||
|
||||
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -78,8 +78,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -55,8 +55,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -63,8 +63,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -87,8 +87,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -69,8 +69,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -71,8 +71,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -68,8 +68,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
## What this means here
|
||||
|
||||
Something in the GitOps chain is stuck at Ready=False. On this cluster that is more consequential than
|
||||
it sounds, because **`git push origin main` IS the deploy** — there is no other apply step. A stuck
|
||||
it sounds, because **merging a PR into `main` IS the deploy** — there is no other apply step. A stuck
|
||||
Kustomization means git and the cluster have diverged and nothing will close the gap.
|
||||
|
||||
The chain has **14 tiers**: `flux-system` → `cilium` → `platform` → `platform-config` →
|
||||
@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -58,8 +58,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -87,8 +87,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -67,8 +67,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -59,8 +59,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -57,8 +57,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -83,7 +83,7 @@ configMapGenerator:
|
||||
- node-memory-swap=node-memory-swap.md
|
||||
# Operator procedure, not an alert page (no alert links to it — the
|
||||
# alert-runbooks-resolvable check treats that as advisory). It is served
|
||||
# because the 4-minute plex.tv claim window is not a thing to go looking
|
||||
# because the plex.tv claim window is short and is not a thing to go looking
|
||||
# for in git: /runbooks/plex-claim has to resolve the moment it is needed.
|
||||
- plex-claim=plex-claim.md
|
||||
- pvc-capacity=pvc-capacity.md
|
||||
|
||||
@@ -83,8 +83,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -74,8 +74,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -75,8 +75,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -52,8 +52,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -49,8 +49,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -56,8 +56,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -56,8 +56,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
+4
-2
@@ -227,8 +227,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -52,8 +52,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
+59
-55
@@ -1,4 +1,4 @@
|
||||
# plex-claim — claiming the Plex Media Server inside the 4-minute token window
|
||||
# plex-claim — claiming the Plex Media Server inside the token window
|
||||
|
||||
**Alert:** none. This is an **operator procedure**, not an incident page — it is indexed here because
|
||||
it is the page you need at the moment you need it, with a stopwatch running.
|
||||
@@ -14,10 +14,11 @@ server to a plex.tv account. It is a **one-shot** operation: the token is exchan
|
||||
the `PLEX_CLAIM` env is **ignored on every start** (upstream gate:
|
||||
`[ ! -z "${PLEX_CLAIM}" ] && [ -z "${token}" ]` in `root/etc/cont-init.d/40-plex-first-run`).
|
||||
|
||||
**A `https://plex.tv/claim` token lives ~4 minutes.** That is shorter than commit → push → Flux's
|
||||
10-minute reconcile interval → pod roll if you let those run on their own timers, which is why this
|
||||
page exists: every step below is written to be run back-to-back, with the reconcile **forced** rather
|
||||
than waited for. Budget: token in hand at T+0, pod restarted by roughly T+2:00.
|
||||
**A `https://plex.tv/claim` token is short-lived — minutes, not hours.** Read the countdown the
|
||||
claim page shows you; do not budget against a number written here. That is shorter than PR → gate →
|
||||
merge → Flux's 10-minute reconcile interval → pod roll if you let those run on their own timers,
|
||||
which is why this page exists: the loop is scripted and the reconcile is **forced** rather than
|
||||
waited for. Measured end to end on 2026-09-29: **~2m45s**.
|
||||
|
||||
**An expired or wrong token does not break anything.** Verified against the image's own first-run
|
||||
script (and the copy inside `plexinc/pms-docker:latest`): a failed exchange gets a 403 with an empty
|
||||
@@ -66,81 +67,84 @@ infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0'
|
||||
```
|
||||
|
||||
## The 4-minute procedure
|
||||
## The procedure
|
||||
|
||||
Have this page open and do not stop between steps.
|
||||
> **STATUS: this server was CLAIMED on 2026-09-29** and `PlexOnlineToken` is present in
|
||||
> `/config` on the `plex-config` PVC. The upstream exchange is gated on that pref being
|
||||
> *absent*, so `PLEX_CLAIM` is now a **no-op on every start** whatever it holds. You only need
|
||||
> this page again after a `/config` wipe or a rebuild onto an empty PVC.
|
||||
|
||||
**On the clock.** The token is short-lived — **read the countdown that plex.tv/claim shows you**
|
||||
rather than trusting a number written here. (Earlier revisions of this page asserted a "~4-minute
|
||||
TTL" in four places with no citation; upstream `pms-docker` documents no expiry at all. The page
|
||||
is the only authority, and it is on screen at T+0.)
|
||||
|
||||
**A burnt token costs nothing.** An expired token gets a 403 with an empty body, `curl`'s status is
|
||||
never checked, the `if` does not fire, there is no `else` and no `exit 1`, and the script (no
|
||||
`set -e`) exits 0. The pod comes up unclaimed and serving. So retry freely.
|
||||
|
||||
### One command
|
||||
|
||||
1. **T+0 — get the token.** Sign in to plex.tv as the account that should OWN this server, open
|
||||
<https://plex.tv/claim>, copy the `claim-…` string. The clock starts now.
|
||||
<https://plex.tv/claim>, copy the `claim-…` string.
|
||||
|
||||
2. **T+0:15 — put it in the Secret**, in place, encrypted, without it ever touching a plaintext file:
|
||||
2. **Put it in the Secret** — in place, encrypted, never touching a plaintext file:
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
|
||||
```
|
||||
|
||||
Replace the `PLEX_CLAIM:` value (the placeholder `replace-me-with-a-fresh-plex-tv-claim-token`, or
|
||||
a previous expired token) and save. `sops` re-encrypts on write to the committed age recipient.
|
||||
|
||||
3. **T+0:45 — commit and push.** `git push origin main` **is** the deploy; there is no other apply
|
||||
step.
|
||||
3. **Ship it.** This drives the whole delivery loop and prints a running stopwatch:
|
||||
|
||||
```
|
||||
git add apps/wave-0/plex-claim-secret.sops.yaml
|
||||
git commit -m 'chore(plex): fresh plex.tv claim token'
|
||||
git push origin main
|
||||
./infra/plex/claim-ship.sh
|
||||
```
|
||||
|
||||
4. **T+1:00 — force the reconcile immediately. This is the one command that saves the window** — it
|
||||
pulls the new commit and re-applies wave 0 instead of waiting up to 10 minutes for the interval:
|
||||
branch → commit → push → PR → gate → merge → `flux reconcile --with-source` → `rollout restart`
|
||||
→ verdict. It refuses to run if the file is unchanged or not encrypted.
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-0 --with-source'
|
||||
```
|
||||
**Measured end to end, 2026-09-29: ~2m45s.** Gate ~1m40s, merge + reconcile ~25s, pod roll and
|
||||
container start ~30s.
|
||||
|
||||
`--with-source` is the load-bearing flag: it reconciles the `flux-system` GitRepository first, so
|
||||
the Kustomization applies **your commit** and not the previously-fetched revision.
|
||||
### Why it is a script and not six pasted commands
|
||||
|
||||
5. **T+1:30 — roll the pod.** This step is **mandatory and easy to forget**: a changed Secret does
|
||||
**not** restart anything. `PLEX_CLAIM` is injected by `secretKeyRef` and read only at container
|
||||
start, and the Deployment's pod template is byte-identical across this change, so Flux applies the
|
||||
new Secret and the running pod keeps the old value forever.
|
||||
`main` is a **protected branch** — the direct push this page used to instruct is now rejected — so
|
||||
the deploy is a PR merged after a green gate. Three of those steps are easy to get wrong under time
|
||||
pressure, and two are silent when skipped:
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex rollout restart deployment/plex'
|
||||
```
|
||||
- **`--with-source` is load-bearing.** Without it the Kustomization applies the previously-fetched
|
||||
revision, not your commit.
|
||||
- **The pod roll is mandatory.** Only the Secret's ciphertext changes, so the Deployment's pod
|
||||
template is byte-identical and *nothing restarts on its own*. `PLEX_CLAIM` is read once, at
|
||||
container start. Skip this and the new Secret sits in the cluster while the running pod keeps the
|
||||
old value forever.
|
||||
- **The branch is named `feat/…` on purpose.** `act_runner` capacity is 1, and `gate.yml`'s push
|
||||
trigger covers `feature/** fix/** docs/** chore/**` but *not* `feat/**`. A `chore/` branch fires a
|
||||
second redundant run that serialises ahead of the PR run and roughly doubles the wall clock.
|
||||
|
||||
6. **T+2:00 — confirm the exchange happened**, in the new pod's init log:
|
||||
### Reading the verdict
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex logs deploy/plex --since=5m | grep -i -e claim -e token'
|
||||
```
|
||||
* `Attempting to obtain server token from claim token` **and** `Token obtained successfully`
|
||||
⇒ **claimed.** Re-run the First-commands `PlexOnlineToken=` check to confirm it persisted, then
|
||||
create the libraries over `/media/movies`, `/media/tv`, `/media/photos`.
|
||||
* `Attempting…` with **no** success line ⇒ the token was expired or invalid. Nothing is broken; go
|
||||
back to step 1 with a new one.
|
||||
* **Neither line** ⇒ the exchange was skipped: `/config` already had a `PlexOnlineToken` (already
|
||||
claimed — the expected case now), or the env did not reach the container. Check by NAME only:
|
||||
|
||||
* `Attempting to obtain server token from claim token` **and** `Token obtained successfully`
|
||||
⇒ **claimed.** Re-run the First-commands `PlexOnlineToken=` check to confirm it persisted, then
|
||||
create the libraries over `/media/movies`, `/media/tv`, `/media/photos`.
|
||||
* `Attempting to obtain server token from claim token` with **no** success line ⇒ the token was
|
||||
expired or invalid. Nothing is broken; go back to step 1 with a new token.
|
||||
* **Neither line** ⇒ the exchange was skipped, which means `/config` already had a
|
||||
`PlexOnlineToken` (already claimed) or the env did not reach the container. Check the pod actually
|
||||
has the variable, by NAME only:
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
|
||||
-o jsonpath="{.items[0].spec.containers[0].env[*].name}"'
|
||||
```
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
|
||||
-o jsonpath="{.items[0].spec.containers[0].env[*].name}"'
|
||||
```
|
||||
|
||||
## Likely causes, most likely first
|
||||
|
||||
When the claim does not take, in order of how often it is the answer here:
|
||||
|
||||
1. **The token expired** — you spent more than ~4 minutes, usually in step 3 or waiting on step 4.
|
||||
Symptom: the "Attempting…" line with no success line. Remedy: a new token, and force the reconcile
|
||||
rather than waiting for it.
|
||||
1. **The token expired** — you exceeded the countdown, usually by reviewing the PR instead of
|
||||
merging it the moment the gate went green. Symptom: the "Attempting…" line with no success line.
|
||||
Remedy: a new token, and let `claim-ship.sh` drive the loop rather than pasting steps.
|
||||
2. **The pod was never restarted** (step 5 skipped). Symptom: no claim log lines at all from a pod
|
||||
whose `AGE` predates your push. The Secret is correct in git and in the cluster and the running
|
||||
container still holds the placeholder.
|
||||
|
||||
@@ -58,8 +58,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -60,8 +60,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -54,8 +54,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -66,8 +66,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
Reference in new issue
Block a user