Compare commits

..
Author SHA1 Message Date
jakman85andClaude Opus 5 b2d9883958 docs(ci): close out the hardening -- land the scripts, correct a claim that shipped wrong
gate / gate (pull_request) Successful in 2m2s
gate / conformance (pull_request) Has been skipped
The branch-protection + PR-gate + sidecar sequence is complete and verified. This records
the finished state and fixes what the sequence itself proved wrong.

CORRECTS A FALSE CLAIM I SHIPPED. PR #7 said the sidecar conversion would remove the
one-per-start dockerd line "http: TLS handshake error from 127.0.0.1: EOF". It does NOT.
Exactly one still appears, immediately after "API listen on [::]:2376" -- it is now the
startupProbe's FIRST attempt, landing microseconds after dockerd begins listening and cut
off by timeoutSeconds: 1; the next poll succeeds. Cosmetic, and deliberately not chased:
raising the timeout to hide one benign startup line would slow every subsequent poll. The
manifest comment now states this instead of the wrong prediction.

Lands infra/gitea/require-gate.sh and verify-then-enforce.sh, which drove the rollout from
a session temp dir. verify-then-enforce.sh merges nothing -- merging is a human's reviewed
act -- but forces the reconcile and ABORTS unless the pod is healthy, a runner is online,
and a real workflow goes green on it, before enabling enforcement and probing the push.
Re-run it after any future act-runner change.

Trims the now-obsolete feat/ rationale from claim-ship.sh: per-branch push triggers were
deleted, so no prefix produces a duplicate run any more.

CLAUDE.md's enforcement table flips from "weaker than this rule" to machine-enforced, with
all four rows marked verified and the push rejection noted as PROVOKED rather than assumed.
It also now states what a green PR does NOT prove: CLUSTER_AVAILABLE is unset, so L2/L3
never runs in CI and only the L0/L1 offline subset is covered.

infra/gitea/README.md marks the sequence done and names the root cause that remains
unfixed: Gitea repo settings live in its own database, invisible to Flux and unreadable by
the conformance gate, so NOTHING detects drift from what this directory declares. These
scripts assert desired state; they are not a detector. The check that reads the live API
and fails on divergence is the highest-value work left, and it is the difference between
this being fixed and this being unable to recur.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 02:28:55 -05:00
jakman85 00efd4b828 Merge pull request 'fix(ci): convert dind to a native sidecar; delete both wait loops' (#7) from fix/dind-native-sidecar into main
gate / gate (push) Successful in 1m35s
gate / conformance (push) Has been skipped
Reviewed-on: #7
2026-09-29 07:22:50 +00:00
jakman85 67a54a33a4 Merge pull request 'fix(ci): delete the per-branch push triggers instead of adding feat/**' (#6) from fix/gate-push-trigger-matches-feat-prefix into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
Reviewed-on: #6
2026-09-29 07:22:20 +00:00
jakman85 db1a623073 Merge pull request 'docs(runbooks): the PR flow, not git push origin main' (#5) from docs/runbooks-pr-flow into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
Reviewed-on: #5
2026-09-29 07:21:28 +00:00
jakman85 ce271225de Merge pull request 'docs: correct CI/platform documentation that was stale or wrong' (#8) from docs/ci-reality-vs-phase-0 into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
Reviewed-on: #8
2026-09-29 07:20:48 +00:00
jakman85andClaude Opus 5 e72050c690 docs: correct CI/platform documentation that was stale, wrong, or load-bearing
gate / gate (push) Successful in 1m37s
gate / gate (pull_request) Successful in 1m36s
gate / conformance (push) Has been skipped
gate / conformance (pull_request) Has been skipped
The 2026-09-29 session found the merge gate did not exist -- no branch protection, no
runner -- while the tree asserted both. This corrects the documents that carried, and in
several cases CAUSED, that belief.

CLAUDE.md gains section 2b: never infer platform state from this repository. It carries
the specific API and kubectl checks for protection, runners, CI status, workloads and
Flux revisions, and the rule that a green tick is not proof anything ran (a skipped Gitea
job reports success). This is the rule whose absence produced every other finding.

.gitea/workflows/README.md and merge-gate.yaml both said "there is no remote git host,
this file is dormant". Gitea had been live for weeks. Both were written as plans and never
revisited once the plan completed. The Transition section is now marked COMPLETE and says
what is genuinely still outstanding (CLUSTER_AVAILABLE unset, so L2/L3 never runs in CI).

DECISIONS-LOG 2026-08-31 is annotated IN PLACE. Anyone reading chronologically previously
hit "required status check; runner incluster-dind online" with nothing marking it false
until a month later. The correction now sits on the bullet itself.

ci/README.md, clusters/README.md and both conformance READMEs carry a status banner: they
speak as though no cluster exists, which makes a live L2/L3 coverage gap read as an
expected phase. That framing is why the gap went unexamined.

apps/wave-1/gitea-actions-runner-secret.sops.yaml: fixed a wrong path, replaced advice to
`sops -d` into a terminal (which CLAUDE.md forbids) with the rotation script, and recorded
that a fresh token alone does not re-register a runner holding a stale /data/.runner.

apps/wave-0/plex-claim-secret.sops.yaml: the header claimed the file ships the placeholder.
It has not since the claim -- it holds the spent token, which is inert and deliberately
left in place.

catalog/plex.yaml and the plex secret both asserted a "~4-minute TTL" as fact. It was
uncited, upstream documents no expiry, and it was the basis for a since-disproven claim
that the pipeline was too slow to fit inside it. Measured: ~2m45s.

Both .sops.yaml files were verified to still decrypt after the comment edits
(`sops -d > /dev/null`), and render==golden confirms the catalog comment does not
propagate into the rendered manifest.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 02:14:26 -05:00
jakman85andClaude Opus 5 e7508931ca fix(ci): convert dind to a native sidecar; delete both hand-rolled wait loops
gate / gate (push) Successful in 1m34s
gate / gate (pull_request) Successful in 1m35s
gate / conformance (push) Has been skipped
gate / conformance (pull_request) Has been skipped
Executes the FOLLOW-UP that has been carried in this manifest since the readiness fix,
now that k3s is v1.36 and sidecar containers are GA.

dind moves into initContainers as the first entry with restartPolicy: Always -- the one
field that makes an initContainer a sidecar: it starts before the regular containers,
runs alongside them, and is terminated after them. kubelet therefore enforces the
ordering that the runner used to hand-roll, so the runner drops its command: override
and runs the image's own entrypoint.

The startupProbe execs a full TLS-verified `docker ... info`, NOT a tcpSocket probe. A
tcpSocket probe would re-open the exact gap the old Stage 1 existed to close: dockerd
writes the TLS certs and only THEN starts listening, so the port can accept a connection
while the certs are absent, and act_runner hard-exits on a missing
/certs/client/ca.pem. `docker info` proves both that the certs are usable and that the
daemon answers. periodSeconds 1 x failureThreshold 60 preserves the previous 60s cap.

It also removes the one-per-start dockerd log line
  http: TLS handshake error from 127.0.0.1: EOF
which was the old `nc -z` probe opening a TCP connection and closing it without a TLS
handshake -- an error we were generating ourselves and then reading in the logs.

SECONDARY WIN: deleting the shell deletes an envsubst hazard. apps-wave-1 runs postBuild
substitution over this ENTIRE manifest, comments included, and those loops depended on
bare dollar-names passing through untouched -- a braced or double-dollar spelling would
have silently unbounded the waits or failed the whole Kustomization. The only dollar left
in the file is the intended ${BASE_DOMAIN}.

LANDS ALONE, DELIBERATELY. This Deployment is the merge gate for the entire repository:
if the conversion is wrong, nothing can merge. Merge this BEFORE enabling the required
status check, and watch a real workflow run green on the converted pod before doing so --
otherwise a bad conversion locks the repo with no way to revert through the gate.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 01:54:47 -05:00
jakman85andClaude Opus 5 bf98a99af6 fix(ci): delete the per-branch push triggers instead of adding feat/** to them
gate / gate (pull_request) Successful in 1m34s
gate / conformance (pull_request) Has been skipped
The tracked next-step said "align gate.yml's push trigger with this repo's feat/**
prefix". Doing that literally would have made things worse, so this does the opposite.

act_runner runs at capacity 1. A branch push and its PR fire the SAME gate over the SAME
tree, and the redundant run SERIALISES AHEAD of the PR run -- doubling the wall clock on
the one result that gates the merge. Adding another prefix widens that, it does not fix
it. infra/plex/claim-ship.sh currently has to pick a feat/ branch name specifically to
DODGE this, during a loop that is racing a short-lived token.

The prefix list also silently failed to match this repo's own convention: branches here
are feat/**, which was absent, so the "fast feedback" never fired for the most common
prefix. Nobody noticed for a month because the only symptom of a wrong entry is a MISSING
run. Any hand-maintained prefix list has that failure mode. Deleting the list removes the
whole class rather than correcting one entry.

It was redundant anyway: CLAUDE.md already requires ./ci/gate.sh in the pinned toolbox
before pushing (74s locally), and the PR gate re-runs it authoritatively.

push: [main] is KEPT -- post-merge main must still be verified, and it is the only run for
which no PR context exists.

FOLLOW-UP once this and #5 are both merged: the feat/ rationale in
infra/plex/claim-ship.sh becomes obsolete and its comment should be trimmed. The script
keeps working either way.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 01:50:59 -05:00
17 changed files with 490 additions and 155 deletions

No files matched your search

+28 -14
View File
@@ -40,9 +40,15 @@ and ADR-0011: **CI holds no broad kubeconfig.** L2/L3 run as an in-cluster **Cha
*collects* its artifacts, using a narrow trigger credential referenced **by secret name only**
(`secrets.CONFORMANCE_TRIGGER_KUBECONFIG`) — never a plaintext secret, never a cluster-admin kubeconfig.
## Honest SKIP — never fake green (Phase 0)
## Honest SKIP — never fake green
**No cluster exists in Phase 0.** The gate is deliberately built so unavailable checks **SKIP loudly,
**NOTE (2026-09-29):** this section is written in the past's voice — "Phase 0", "no cluster
exists". **A cluster has existed for weeks** and Flux has been reconciling against it. The
skip logic below is still correct and still honest, but read it as describing a **live
coverage gap** (L2/L3 is not wired into CI) rather than an expected phase. Its being
phrased as a phase is part of why the gap went unexamined.
Originally: The gate is deliberately built so unavailable checks **SKIP loudly,
they do not pass**:
- The `conformance` job is `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}` — with no cluster it is
@@ -77,21 +83,29 @@ Both dirs are git-ignored (see [`.gitignore`](../../.gitignore)).
## Triggers & branch model (CONTRIBUTING.md)
- `pull_request` → `main`: **PRs are the enforced merge gate into `main`.**
- `push` → `feature/**`, `fix/**`, `docs/**`, `chore/**`: fast feedback on the branch. **No work on `main`.**
- `push` → `main` only. The per-branch prefix list was removed: `act_runner` runs at capacity 1,
so a branch push fires a redundant run that serialises *ahead* of the PR run for the same tree.
**No work on `main`** — and since 2026-09-29 that is enforced, not conventional: `main` is a
protected branch and a direct push is rejected by the pre-receive hook (verified, not assumed).
## Transition: generic-CI-now → Gitea-Actions-after-Gitea-is-up
## Transition: COMPLETE (2026-09-29)
Per [`CONTRIBUTING.md`](../../../home-network-k3s/CONTRIBUTING.md) and
[`docs/05` §"local bootstrap → self-hosted Gitea"](../../../home-network-k3s/docs/05-gitops-and-delivery.md):
This section used to describe a migration as pending. It has happened. Gitea is live at
`gitea.home.keithjackson.dev`, the runner `incluster-dind` is registered and online, and
`gate.yml` runs on every PR.
1. **Now (interim):** there is **no remote git host** on this network. This workflow file is dormant —
but the **identical gate runs locally today** via `make -C ci gate` (R-TEST-02). Development is
local feature-branch → `main`, the gate enforced by convention + the local runner.
2. **Gitea comes online early** (one of the first services onboarded) at `gitea.home.keithjackson.dev`.
3. **CI moves to Gitea Actions:** this workflow activates unchanged. The only operational steps are
enabling a Gitea Actions runner and setting `vars.CLUSTER_AVAILABLE=true` +
`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` once a dev/test cluster exists. PRs then become the
**platform-enforced** merge gate — the same no-work-in-main rule, now enforced by machine.
**What took a month to notice:** the migration was recorded as done on 2026-08-31 without
anyone checking the platform. In reality `main` had **no branch protection** and **zero
runners were registered** until 2026-09-29. Every "the gate enforces this" statement in
between rested on an assertion nobody verified. PR #1 — the first pull request this
repository ever had — sat `pending` for 30.6 minutes because nothing existed to run it.
So: **do not read this file, or the DECISIONS-LOG, as evidence about the live platform.**
Query the API. `infra/gitea/` holds the scripts and the measured state.
Still genuinely outstanding: `vars.CLUSTER_AVAILABLE` and
`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` are unset, so L2/L3 conformance does not run in
CI. That is a real coverage gap, not a phase.
## Configuration (set in Gitea; referenced by name only)
+19 -4
View File
@@ -44,16 +44,31 @@
---
name: gate
# TRIGGERS — deliberately only `pull_request` into main, plus `push` to main itself.
#
# There is NO per-branch push trigger. There used to be a prefix list (feature/** fix/**
# docs/** chore/**) for "fast feedback on the branch", and it was a net negative here:
#
# * `act_runner` runs at capacity 1. A branch push and its PR fire the SAME gate over the
# SAME tree, and the redundant run SERIALISES AHEAD of the PR run — doubling the wall
# clock on the one result that actually gates the merge. That cost real time during the
# 2026-09-29 plex claim, where the loop is racing a short-lived token.
# * The list silently did not match this repo's own convention. Branches here are `feat/**`,
# which was absent, so the "fast feedback" never fired for the most common prefix anyway —
# a mismatch nobody noticed for a month because its only symptom was a MISSING run.
# Any hand-maintained prefix list has that failure mode; deleting it removes the class.
# * It is redundant. CLAUDE.md already requires `./ci/gate.sh` in the pinned toolbox before
# you push (74s locally), and the PR gate re-runs it authoritatively. A third run of the
# same checks buys nothing.
#
# `push: [main]` is KEPT: post-merge main must still be verified, and it is the only run for
# which no PR context exists.
on:
pull_request:
branches: [main]
push:
branches:
- main
- 'feature/**'
- 'fix/**'
- 'docs/**'
- 'chore/**'
concurrency:
group: gate-${{ github.ref }}
+8 -5
View File
@@ -11,11 +11,14 @@
# v
# merge-gate decision [R-TEST-03: touched subset green]
#
# Transition (CONTRIBUTING.md / docs/05 §"local bootstrap -> self-hosted Gitea"):
# Today there is no Gitea host, so this file is dormant. The SAME gate is runnable
# now via `make -C ci gate` (R-TEST-02). Once Gitea is self-hosted and Gitea
# Actions is enabled, this workflow becomes the ENFORCED PR merge gate into main —
# no code change required, only turning on the runner and setting CLUSTER_AVAILABLE.
# STATUS (corrected 2026-09-29): Gitea IS hosted and Gitea Actions IS enabled; the runner
# `incluster-dind` is online and gate.yml is the live PR gate. The text that used to sit here
# ("Today there is no Gitea host, so this file is dormant") was stale by weeks -- it was written
# as a plan and never revisited once the plan completed.
#
# THIS file is still dormant, but for the reason in the DORMANT block below -- its spec checkout
# and L2/L3 seams are unwired -- NOT because there is no host. Do not read a stale plan as a
# statement about the live platform; query the API instead (see infra/gitea/).
#
# CI holds NO broad kubeconfig. L2/L3 run as an in-cluster Chainsaw Job whose own
# scoped ServiceAccount holds cluster access (docs/14, R-SEC-02); CI uses only a
+40 -10
View File
@@ -23,28 +23,58 @@ There is no other path. Specifically:
> (main is what the cluster follows), not what you are permitted to do. Honour this
> file, not that line. Fixing those runbooks is tracked work.
## 2. Enforcement is currently WEAKER than this rule. Follow the rule anyway.
## 2. Enforcement (as of 2026-09-29: the rule is now MACHINE-ENFORCED)
Verified against the live Gitea on 2026-09-29:
| Expected | Actual |
|---|---|
| `main` protected, direct push rejected | **No branch protection.** `protected: false`, `user_can_push: true` |
| PR blocked until the gate passes | **`enable_status_check: false`**, `status_check_contexts: []` |
| Gate runs on every PR | **Fixed 2026-09-29.** Runner `incluster-dind` is online; `gate / gate (pull_request)` runs and passes in ~1m30s–2m |
| Approval required before merge | **`required_approvals: 0`** |
| `main` protected, direct push rejected | ✅ `protected: true`, `user_can_push: false` — **verified by a real rejected push**, not by the API's word |
| PR blocked until the gate passes | ✅ `enable_status_check: true`, context `gate / gate (pull_request)` |
| Gate runs on every PR | ✅ runner `incluster-dind` online; ~1m35s |
| Approval required before merge | ✅ `required_approvals: 1` |
So nothing mechanically stops you from pushing to `main` right now. **That is a gap
being closed, not permission.** `DECISIONS-LOG.md` (2026-08-31) claimed the gate was a
"required status check" with a runner online; neither was true, which is how this went
unnoticed for a month. The runner half is now genuinely fixed and observed green. The
protection half is not — see `infra/gitea/`.
All four were FALSE as recently as 2026-09-29, while this repo's own log asserted they were
true. They are true now because each was measured, and the push rejection was provoked rather
than assumed. **Re-measure before relying on any of them** — see §2b.
Still not covered: `vars.CLUSTER_AVAILABLE` is unset, so **L2/L3 conformance does not run in
CI**. A green PR therefore proves the L0/L1 offline subset, not runtime behaviour.
**Consequence for you:** CI now runs, but it does not yet *block* anything. Always run
the toolbox gate yourself and quote its numbers. Never describe a PR as "green" on the
strength of a check you have not seen report — and note that a **skipped** Gitea job
reports as `success`, so a green tick is not by itself evidence that anything ran.
## 2b. Never infer platform state from this repository
**This repo is not evidence about the live platform.** Neither is `DECISIONS-LOG.md`.
Both describe what someone believed at the time of writing, and both have been
confidently wrong for a month at a stretch.
On 2026-09-29 the log stated the merge gate was a required status check with a runner
online. The API said `branch_protections: []` and **zero** runners. Everything downstream
of that belief — "the gate enforces this", "your push would be rejected", "CI covers it" —
was wrong, and nothing in the tree could have revealed it, because the tree was the source
of the error.
**So, before asserting that any gate, protection, runner, secret, or workload is in a
given state: query the thing itself.**
| Claim about... | Check |
|---|---|
| branch protection / required checks | `GET /api/v1/repos/{owner}/{repo}/branches/main` |
| a runner exists | `GET /api/v1/admin/actions/runners` |
| CI actually ran | the commit's `/statuses` — and read the **description**; a *skipped* Gitea job reports `success` with "Has been skipped" |
| a workload / Secret / env | `kubectl --context k3s-prod ...` (read-only is fine for agents) |
| Flux applied your commit | compare `.status.lastAppliedRevision` against the GitRepository's artifact revision |
Quote the measured field, not the prose. If you cannot measure it, say so explicitly —
"unverified" is a legitimate report, and a far better one than a confident guess that
matches the docs.
**A green tick is not proof anything ran.** Read what actually executed.
## 3. Gate before you push
./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
+61
View File
@@ -1192,6 +1192,13 @@ fresh agent should read this entry + `infra/PHASE1-STATUS.md` + `docs/plans/` an
Platform-admins require TOTP MFA.
- **Gitea Actions CI is LIVE + enforcing**: `.gitea/workflows/gate.yml` runs `make -C ci gate` on push/PR to main;
required status check; runner `incluster-dind` online (ns gitea-actions).
> **[CORRECTION, 2026-09-29 — THIS BULLET WAS FALSE WHEN WRITTEN AND STAYED FALSE FOR A MONTH.]**
> Measured against the live Gitea API on 2026-09-29: `branch_protections` returned `[]` (`main`
> `protected:false`, `enable_status_check:false`, `user_can_push:true`) and **zero** Actions runners
> were registered at repo, org or site level. The gate was neither enforced nor able to run. PR #1
> — the first pull request this repo ever had — was opened on 2026-09-29 and sat `pending` for
> 30.6 minutes with nothing to execute it. Both halves of this bullet were asserted, not verified.
> See the 2026-09-29 entries for the root cause and the fix.
### What changed this session (durable)
1. **SSO cluster-wide break→fix — root cause Cilium CVE-2025-30162.** The 1.16→1.20 security fix makes the shared
@@ -4863,3 +4870,57 @@ That theory was mine and it was wrong: the pipeline was never too slow, it simpl
FOLLOW-UP note in `apps/wave-1/gitea-actions.yaml`. It deletes both hand-rolled wait loops and
the envsubst hazard. Land it ALONE — it is the merge gate for the whole repo.
6. Create the Plex libraries over `/media/movies`, `/media/tv`, `/media/photos`.
## 2026-09-29 (later) — the merge gate is now real, enforced, and proven. **PM HANDOFF.**
Closes the gap opened by the earlier entry today. Eight PRs, all gated; every claim below was
measured against the live API or cluster, not inferred from this repository.
### Final state
| | Verified |
|---|---|
| `main` protected | `protected:true`, `user_can_push:false` — and a direct push as site-admin `gitea-admin` was **provoked and rejected** (`pre-receive hook declined`) |
| Gate required | `enable_status_check:true`, contexts `["gate / gate (pull_request)"]` |
| Approvals | `required_approvals:1`; `jakman85` holds `owner` |
| Runner | `incluster-dind` online, now a **native sidecar** (`initContainers`, `restartPolicy: Always`, TLS-verified `docker info` startupProbe) |
| Gate time | ~1m35s |
| Plex | claimed 2026-09-29; `PlexOnlineToken` present on the `plex-config` PVC |
### What was corrected in the documentation
`CLAUDE.md` §2b — *never infer platform state from this repository* — is the durable outcome.
The 2026-08-31 entry asserting an enforced gate is now annotated **in place**. The "Phase 0 /
no cluster exists" framing across `ci/`, `clusters/` and `conformance/` READMEs is banner-marked
as describing a live gap rather than a stage. All 37 runbooks now describe the PR flow. Two
`.sops.yaml` headers and `catalog/plex.yaml` were corrected (comment edits verified not to break
the SOPS MAC, and not to move the rendered golden).
### A claim of mine that shipped wrong, and is now corrected in place
PR #7 stated the sidecar conversion would remove the one-per-start dockerd line
`http: TLS handshake error from 127.0.0.1: EOF`. **It does not.** Exactly one still appears,
immediately after `API listen on [::]:2376` — it is now the startupProbe's first attempt,
landing microseconds after dockerd begins listening and cut off by `timeoutSeconds: 1`; the
next poll succeeds. Cosmetic, and deliberately not chased: raising the timeout to hide one
benign startup line would slow every subsequent poll. The manifest comment now says so.
### The root cause nobody has fixed yet
Gitea's repo settings live in its own database — invisible to Flux and unreadable by the
conformance gate. **No check exists that could have caught "protection off, runner gone."**
`infra/gitea/` declares and can re-assert the desired state, but it is not a detector. A
conformance check that reads the live Gitea API and fails on divergence is the highest-value
remaining work in this repo, and it is the difference between this having been fixed and this
being unable to recur.
### Next steps
1. **Write that Gitea-settings conformance check.** Everything else here is a symptom.
2. Wire `vars.CLUSTER_AVAILABLE` + `secrets.CONFORMANCE_TRIGGER_KUBECONFIG` so **L2/L3 runs in
CI**. A green PR currently proves the L0/L1 offline subset only — runtime behaviour is
unverified on every merge.
3. Re-run `infra/gitea/verify-then-enforce.sh` after any future act-runner change.
4. `required_approvals: 1` means a PR you open as `jakman85` needs `gitea-admin` to approve.
If that friction bites, either open agent work as `gitea-admin` and review as `jakman85`, or
drop back to relying on the status check alone.
+14 -6
View File
@@ -7,12 +7,20 @@
# substitution). A braced token quoted from the upstream shell script would be a live hazard here,
# not prose, so the upstream conditions below are described in words.
#
# THE VALUE SHIPPED WITH THIS FILE IS A PLACEHOLDER, DELIBERATELY — the literal string
# `replace-me-with-a-fresh-plex-tv-claim-token`, not a credential. A https://plex.tv/claim token has
# a ~4-minute TTL, far shorter than commit -> push -> Flux reconcile -> pod-roll, so no token
# committed here is ever still valid when the pod starts. This file exists to make the Secret — and
# therefore the env reference — ALWAYS RESOLVABLE IN THE FLUX GRAPH, not to carry a working token.
# The real token is injected by the operator inside that 4-minute window: runbooks/plex-claim.md.
# THIS FILE NO LONGER HOLDS THE PLACEHOLDER. Corrected 2026-09-29: the server was CLAIMED on that
# date, and the value here is the SPENT token from that claim. It is inert and safe to leave --
# upstream gates the exchange on PlexOnlineToken being ABSENT from /config, and it is now present on
# the plex-config PVC, so this value is a NO-OP on every start whatever it holds. Deleting the Secret
# would NOT be safe (see below), which is why a spent token stays here rather than being blanked.
#
# The claim token is SHORT-LIVED -- minutes. Read the countdown that plex.tv/claim renders when it
# issues one; do NOT budget against a number written in this tree. Earlier revisions of this comment
# asserted a "~4-minute TTL" as established fact with no citation, and upstream plexinc/pms-docker
# documents no expiry at all. That uncited number was load-bearing for the whole procedure design.
#
# This file exists to make the Secret — and therefore the env reference — ALWAYS RESOLVABLE IN THE
# FLUX GRAPH, not to carry a working token. Re-claiming (only needed after a /config wipe) is
# scripted: infra/plex/claim-ship.sh, measured at ~2m45s end to end.
#
# WHY SHIPPING A PLACEHOLDER IS SAFE (verified against plexinc/pms-docker branch `master`, the
# s6-overlay v2 script root/etc/cont-init.d/40-plex-first-run, cross-checked byte-identical against
@@ -4,7 +4,13 @@
# against Gitea. Flux decrypts it via the sops-age key at reconcile (apps Kustomization
# decryption is wired in clusters/dev/kustomization.yaml). Rotate by resetting the token
# in Gitea (admin -> actions -> runners) and re-encrypting a fresh value here.
# Recover/inspect: infra/devcontainer/dev.sh run sops -d apps/gitea-actions-runner-secret.sops.yaml
# ROTATE: ./infra/gitea/runner-token-rotate.sh -- it mints a fresh instance-level token from the
# Gitea API and writes it here via `sops set`, without the value ever being printed.
# Do NOT `sops -d` this file into a terminal, a log, or an agent transcript. To check only that
# it still decrypts, redirect: `sops -d <file> > /dev/null`.
# NOTE: a fresh token alone may not re-register the runner. act_runner registers ONCE and keeps
# its identity in /data/.runner on the act-runner-data PVC; if Gitea no longer holds a matching
# row it skips registration silently. Clearing that PVC is what forces a real re-registration.
apiVersion: v1
kind: Secret
metadata:
+69 -106
View File
@@ -99,70 +99,79 @@ spec:
# CI holds no cluster credentials — do not mount a SA token into the runner or
# (via the shared netns) the job containers (R-SEC-02, docs/14).
automountServiceAccountToken: false
# ---- Docker-in-Docker: the job executor daemon (privileged) -------------------
#
# NATIVE SIDECAR (k3s v1.36; sidecar containers are GA). It is an initContainer with
# `restartPolicy: Always`, which means: it starts BEFORE the regular containers, keeps
# running alongside them, and is terminated after them. kubelet holds `runner` until
# this container's startupProbe passes, so the ordering is enforced by the kubelet
# instead of hand-rolled shell in the runner.
#
# This REPLACES two bounded wait-loops that used to live in the runner's `command:`.
# Deleting them also deletes an envsubst hazard: clusters/base/apps-wave-1.yaml runs
# postBuild substitution over this ENTIRE manifest, comments included, and those loops
# depended on bare dollar-names surviving it untouched. There is now no shell here at
# all, so there is nothing to escape or mis-escape.
#
# WHY THE PROBE IS `exec docker ... info` AND NOT `tcpSocket`. A tcpSocket probe would
# re-open the exact gap the old Stage 1 existed to close: dockerd writes the TLS certs
# and only THEN begins listening, so the port can accept a connection while the certs
# are absent or incomplete -- and act_runner hard-exits on a missing
# /certs/client/ca.pem. A full TLS-verified `docker info` proves BOTH that the certs
# are usable and that the daemon answers.
#
# CORRECTION (measured 2026-09-29, after this conversion shipped): the commit that
# introduced this claimed the conversion would REMOVE the one-per-start dockerd line
# `http: TLS handshake error from 127.0.0.1: EOF`. IT DOES NOT. Exactly one still
# appears, immediately after "API listen on [::]:2376" -- it is now this probe's FIRST
# attempt, landing microseconds after dockerd begins listening and cut off by
# timeoutSeconds: 1. The next poll succeeds. It is cosmetic, exactly as cosmetic as it
# was before, and deliberately NOT chased: raising timeoutSeconds to hide one benign
# startup line would slow every subsequent poll. Left documented rather than silenced.
#
# periodSeconds 1 x failureThreshold 60 preserves the previous 60s cap.
initContainers:
- name: dind
restartPolicy: Always # <- this field is what makes it a SIDECAR
image: docker:27-dind
securityContext:
privileged: true
env:
# Generate + publish TLS certs into the shared emptyDir; dockerd then listens
# on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
- name: DOCKER_TLS_CERTDIR
value: /certs
volumeMounts:
- name: certs
mountPath: /certs
- name: dind-storage
mountPath: /var/lib/docker
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
memory: 2Gi
startupProbe:
exec:
command:
- docker
- --host=tcp://127.0.0.1:2376
- --tlsverify
- --tlscacert=/certs/client/ca.pem
- --tlscert=/certs/client/cert.pem
- --tlskey=/certs/client/key.pem
- info
periodSeconds: 1
failureThreshold: 60
containers:
# ---- act_runner: registers with Gitea, dispatches jobs to the DinD daemon ----
- name: runner
image: gitea/act_runner:0.2.11
# TWO-STAGE BOUNDED WAIT on the DinD sidecar, then exec the image's normal
# entrypoint (/sbin/tini -- /opt/act/run.sh), which registers-from-env on first
# boot and runs `act_runner daemon`.
#
# STAGE 1 — the certs must EXIST: act_runner reads /certs/client/ca.pem at startup
# and hard-exits if it is missing (a startup race against the sidecar).
# STAGE 2 — the daemon must actually ACCEPT a connection: the cert FILE appearing is
# necessary but NOT sufficient. dockerd writes the certs and only THEN begins
# listening on :2376, so a runner that proceeds on the file alone still hard-exits
# with "cannot ping the docker daemon at tcp://localhost:2376" — observed
# crashlooping 11 times in ~8 minutes on a fresh pod after a node bounce.
# Both loops are bounded (60 x 1s) and exit 1 on timeout, so a genuinely dead sidecar
# surfaces as a CrashLoopBackOff with a clear message instead of hanging forever.
#
# ENVSUBST SAFETY — the BARE $i / $j below are CORRECT. Do not "tidy" them into the
# braced form or the double-dollar form; both are wrong here, for different reasons.
# clusters/base/apps-wave-1.yaml carries postBuild.substituteFrom cluster-vars, so
# Flux envsubst runs over this ENTIRE manifest — YAML comments very much included.
# Its contract (drone/envsubst; the same one spelled out in clusters/base/apps-vms.yaml,
# and re-verified against this exact file with `flux envsubst`, flux v2.4.0) is:
# - A bare $name is NOT a reference to envsubst: it passes through untouched, and
# that holds even when the name IS set in cluster-vars. So $i and $j reach the
# cluster verbatim and both 60s bounds below are real, enforced bounds.
# - $((i + 1)) is likewise untouched: "(" cannot begin a variable name.
# - A BRACED dollar-name IS a reference and gets replaced — with the empty string
# when the name is unset. One of those here would make a bound read [ "" -ge 60 ],
# which errors, short-circuits the &&, and silently turns the wait UNBOUNDED.
# - A DOUBLE-dollar braced name is envsubst's escape and survives as the braced
# form — still wrong for us, because on any path where envsubst does NOT run
# (a plain `kustomize build` or `kubectl apply -k` of this directory) the shell
# reads that leading double-dollar as its own PID, so a bound would compare a
# string like 1234{i} against 60. The bare form is the only spelling that is
# correct on BOTH paths.
# - A braced name containing punctuation is a HARD ERROR ("unable to parse variable
# name") that fails this whole Kustomization — and this Kustomization gates every
# merge in the repo. So never write one anywhere in this file, INCLUDING in a
# comment. That is why this note names the three forms in words rather than
# showing them: an earlier draft of it spelled them out literally and broke
# `flux envsubst` on the file. (Same trap the indexed-expansion warning in
# clusters/base/apps-vms.yaml describes.)
command:
- /bin/sh
- -c
- |
# Stage 1: the sidecar's TLS client certs must appear in the shared emptyDir.
i=0
until [ -f /certs/client/ca.pem ]; do
i=$((i + 1))
[ "$i" -ge 60 ] && { echo "dind certs never appeared" >&2; exit 1; }
echo "waiting for dind TLS certs ($i)"; sleep 1
done
# Stage 2: dockerd must actually accept a connection on :2376. busybox nc in this
# Alpine-based image (gitea/act_runner:0.2.11) supports -z (scan) and -w (timeout).
j=0
until nc -z -w1 127.0.0.1 2376; do
j=$((j + 1))
[ "$j" -ge 60 ] && { echo "dind daemon never accepted a connection on 2376" >&2; exit 1; }
echo "waiting for dind daemon on :2376 ($j)"; sleep 1
done
exec /sbin/tini -- /opt/act/run.sh
# No `command:` override. The image's own entrypoint (/sbin/tini -- /opt/act/run.sh)
# registers from env on first boot and runs `act_runner daemon`. The two wait-loops
# that used to live here are gone: kubelet now holds this container until the dind
# sidecar's startupProbe passes (see initContainers above).
env:
- name: GITEA_INSTANCE_URL
value: https://gitea.${BASE_DOMAIN}
@@ -198,52 +207,6 @@ spec:
memory: 128Mi
limits:
memory: 512Mi
# ---- Docker-in-Docker: the job executor daemon (privileged) ----
#
# FOLLOW-UP (deliberately NOT bundled into the readiness fix above — own change):
# Convert this container into a NATIVE SIDECAR and delete both shell wait-loops.
# k3s is now v1.36, where sidecar containers are GA, so kubelet can enforce the
# ordering that the loops above hand-roll — which also removes the envsubst hazard
# from this file entirely (no shell, no $i/$j, nothing to escape or mis-escape).
# Shape to build:
# 1. Move this whole container into `initContainers:` as the FIRST entry.
# 2. Add `restartPolicy: Always` to it. That single field is what makes an
# initContainer a sidecar: it starts before the regular containers, keeps
# running alongside them, and is terminated after them.
# 3. Give it a `startupProbe` that succeeds only when dockerd accepts a TLS
# connection — e.g. `exec` a `docker -H tcp://127.0.0.1:2376 --tlsverify
# --tlscacert /certs/client/ca.pem --tlscert /certs/client/cert.pem
# --tlskey /certs/client/key.pem info`. Do NOT use a bare `tcpSocket` probe:
# it would re-open the same "port accepting but TLS/certs not ready" gap that
# Stage 1 above exists to close. `periodSeconds: 1` + `failureThreshold: 60`
# preserves today's 60s cap.
# 4. kubelet then holds the `runner` container until that startupProbe passes, so
# `runner` can drop its `command:` override completely and run the image's
# default entrypoint.
# Why it is a separate change: this Deployment is the ENFORCED MERGE GATE for this
# entire repository. If the conversion is wrong, nothing can merge — so it must land
# alone, where the failure is unambiguous and not entangled with anything else, and
# only after a real workflow run has been observed green on the converted pod.
- name: dind
image: docker:27-dind
securityContext:
privileged: true
env:
# Generate + publish TLS certs into the shared emptyDir; dockerd then listens
# on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
- name: DOCKER_TLS_CERTDIR
value: /certs
volumeMounts:
- name: certs
mountPath: /certs
- name: dind-storage
mountPath: /var/lib/docker
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
memory: 2Gi
volumes:
- name: data
persistentVolumeClaim:
+11 -5
View File
@@ -38,11 +38,17 @@ env:
# UNCLAIMED, so this is wired; it was previously left out on the reasoning below, and only one
# half of that reasoning survived contact with the upstream source.
#
# THE 4-MINUTE TTL IS STILL TRUE, and it is now the OPERATIONAL constraint rather than a reason to
# abstain. A https://plex.tv/claim token expires in ~4 minutes — far shorter than commit -> push ->
# Flux reconcile -> pod-roll if that cycle is left to its own timers. So the committed value is a
# documented PLACEHOLDER and the live token is injected by the operator inside that window, with an
# explicit `flux reconcile` to collapse the wait: runbooks/plex-claim.md is the timed procedure.
# A SHORT TTL IS THE OPERATIONAL CONSTRAINT, not a reason to abstain. A https://plex.tv/claim token
# expires in MINUTES — read the countdown the claim page renders; do not budget against a number
# written in this tree. (This comment previously asserted "~4 minutes" as established fact. It was
# UNCITED, upstream plexinc/pms-docker documents no expiry at all, and that number was load-bearing
# for the whole procedure design — including a since-disproven claim that the delivery pipeline was
# too slow to fit inside it. Measured 2026-09-29: the full loop is ~2m45s and fits.)
#
# The live token is injected by the operator inside that window, with an explicit `flux reconcile`
# to collapse the wait. Scripted end to end as infra/plex/claim-ship.sh; runbooks/plex-claim.md is
# the procedure. SERVER CLAIMED 2026-09-29 — this env is now a permanent no-op (see below), and the
# committed value is the spent token from that claim rather than the original placeholder.
#
# ONE-SHOT, AND SAFE TO SIT HERE EXPIRED. Verified from plexinc/pms-docker's own first-run script
# (branch `master`, root/etc/cont-init.d/40-plex-first-run, confirmed byte-identical to the file in
+14
View File
@@ -1,5 +1,19 @@
# `ci/` — portable merge gate (runnable today, without Gitea)
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
> This document speaks as though no cluster exists and the platform is scaffolding. **A
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
> coverage hole, not a phase.
>
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
> documentation rather than measurement, and `main` was unprotected with zero runners
> registered the whole time. **Query the live API before asserting platform state.**
**Task T-0.2 · R-TEST-02 (reproducible, single-command) · Owner WS-GITOPS (CI).**
The same merge gate the Gitea Actions workflow runs
+14
View File
@@ -1,5 +1,19 @@
# clusters/
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
> This document speaks as though no cluster exists and the platform is scaffolding. **A
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
> coverage hole, not a phase.
>
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
> documentation rather than measurement, and `main` was unprotected with zero runners
> registered the whole time. **Query the live API before asserting platform state.**
Flux entrypoints and the environment-overlay model (**R-ENV-02**, **R-GITOPS-01/02/03**).
Governing spec: [`docs/05-gitops-and-delivery.md`](../../home-network-k3s/docs/05-gitops-and-delivery.md),
+14
View File
@@ -1,5 +1,19 @@
# conformance/ — the home-k3s conformance harness (WS-TEST)
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
> This document speaks as though no cluster exists and the platform is scaffolding. **A
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
> coverage hole, not a phase.
>
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
> documentation rather than measurement, and `main` was unprotected with zero runners
> registered the whole time. **Query the live API before asserting platform state.**
This subtree is the **definition of "done"** for the whole platform. Nothing is accepted
on a demo, a screenshot, or "it reconciled": every requirement is proven by an
**automated, reproducible, behavior-based test** tagged with the requirement ID(s) it
+14
View File
@@ -1,5 +1,19 @@
# In-cluster conformance Job (L2/L3)
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
> This document speaks as though no cluster exists and the platform is scaffolding. **A
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
> coverage hole, not a phase.
>
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
> documentation rather than measurement, and `main` was unprotected with zero runners
> registered the whole time. **Query the live API before asserting platform state.**
Per **ADR-0011** and **docs/14 "Where each layer runs"**: L0/L1 run in CI without a
cluster, but **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job**
on the dev/test cluster that **CI triggers and waits for**. CI holds **no** cluster
+40 -1
View File
@@ -43,7 +43,46 @@ list. That is precisely the observed symptom: zero runners, pod running, no erro
looks at. Clearing the PVC is what forces a real re-registration; the script prints the
commands.
## Sequence — order matters
## Sequence — DONE 2026-09-29
All four steps are complete and each was verified, not assumed:
| Step | State |
|---|---|
| 1. Runner registered | ✅ `incluster-dind` online. Root cause of its absence was a stale `/data/.runner` identity; clearing the PVC forced re-registration |
| 2. Gate observed green on a real PR | ✅ `gate / gate (pull_request)` ~1m35s |
| 3. `enable_status_check: true` | ✅ context `gate / gate (pull_request)` — that one ONLY |
| 4. Collaborator + approvals | ✅ `jakman85` is owner; `required_approvals: 1` |
Direct push to `main` was **provoked and rejected** (`pre-receive hook declined`) as
`gitea-admin`, a site admin — so protection is not merely declared, it holds for the account
agents authenticate as.
### `require-gate.sh` / `verify-then-enforce.sh`
`verify-then-enforce.sh` is the re-runnable form of the whole safe order. It merges nothing —
merging is a human's reviewed act — but it forces the Flux reconcile and then **aborts** unless
the runner pod is healthy, a runner is online, and a real workflow goes green on it, before
enabling enforcement and probing the push. `require-gate.sh` is the enforcement step alone.
Re-run `verify-then-enforce.sh` after any future change to the act-runner Deployment.
### Why these scripts exist at all, and what is still missing
Gitea's repo settings live in **its own database**, not in anything Flux reconciles or the
conformance gate can read. Nothing detects drift from what this directory declares. That is
the root cause of the entire 2026-09-29 session: branch protection was off and the runner was
gone for a month while the log recorded both as working, and **no check existed that could
have caught it.**
These scripts encode the desired state and can re-assert it, but they are not a detector. The
durable fix is a conformance check that reads the live Gitea API and fails when it diverges
from what is declared here — turning a class of silent, invisible failure into a loud one.
**That check does not exist yet and is the highest-value remaining work in this directory.**
## Historical sequence (kept for context)
### Original ordering notes
1. **Get an Actions runner registered** — `runner-token-rotate.sh`, then clear the PVC so
the runner actually re-registers, then confirm it shows `online` in the API. Do not
+50
View File
@@ -0,0 +1,50 @@
#!/usr/bin/env bash
# Items 1 + 2: make the gate a REQUIRED status check and require one approval.
#
# RUN THIS LAST. Prerequisites, in order:
# a) PR #7 (dind native sidecar) is MERGED, and
# b) you have SEEN a real workflow run go green on the converted pod.
# The runner Deployment is the merge gate for the whole repo. Turning on a required check
# while the runner is broken means no PR can merge -- including the revert. (Recoverable:
# an owner can set enable_status_check back to false. But do not rely on that.)
#
# Requires exactly ONE context: `gate / gate (pull_request)`.
# * NOT `gate / conformance (*)` -- that job is `if: false`, and a skipped Gitea job reports
# SUCCESS ("Has been skipped"). Requiring it would pass VACUOUSLY forever while nothing
# ran, which under HARD RULE 4 is worse than a deadlock: a deadlock is loud.
# * NOT the `(push)` variant -- after PR #6 there are no per-branch push runs at all, so a
# PR would never produce that context.
set -euo pipefail
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
echo "==> updating branch protection on main"
jq -n '{
enable_push: false,
enable_status_check: true,
status_check_contexts: ["gate / gate (pull_request)"],
required_approvals: 1,
block_on_rejected_reviews: true,
dismiss_stale_approvals: true,
block_on_outdated_branch: false
}' | api -X PATCH -H 'Content-Type: application/json' --data @- \
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main" >/dev/null
echo "==> verifying"
api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \
| jq -r '{protected,user_can_push,enable_status_check,status_check_contexts,required_approvals}'
cat <<'NOTE'
NOTE ON required_approvals: 1 -- Gitea will not let an author approve their own PR.
* Agent-opened PRs (author gitea-admin) are approved by jakman85. Fine.
* PRs YOU open as jakman85 need gitea-admin to approve. If that is awkward, either open
agent work as gitea-admin and review as jakman85, or set required_approvals back to 0
and rely on the status check alone.
Then prove it, rather than trusting the API:
git commit --allow-empty -m probe && git push origin main # EXPECT: rejected
git reset --hard HEAD~1
NOTE
+83
View File
@@ -0,0 +1,83 @@
#!/usr/bin/env bash
# verify-then-enforce.sh — run this AFTER you have merged PRs 8, 5, 6, 7 in that order.
#
# It does NOT merge anything. Merging is your reviewed act (CLAUDE.md: a human approves and
# merges). This script does the mechanical half: prove the converted runner actually works,
# and only then turn on enforcement.
#
# WHY THIS ORDER. The act-runner Deployment IS the merge gate for this repository. Enabling a
# required status check while the runner is broken means NO PR can merge -- including the
# revert. So PR #7 lands while the gate is still advisory, is PROVEN green on a real run, and
# only then does enforcement go on.
#
# Aborts at the first checkpoint that does not pass. Safe to re-run.
set -euo pipefail
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
REPO_ROOT=/var/home/keith/source/home-network-k3s-impl
D="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
cd "$REPO_ROOT"
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
A="https://${HOST}/api/v1/repos/${OWNER}/${REPO}"
say() { printf '\n\033[1m==> %s\033[0m\n' "$1"; }
die() { printf '\n\033[31mABORT: %s\033[0m\n' "$1" >&2; exit 1; }
say "PRECHECK — all four PRs must be merged first"
for n in 8 5 6 7; do
m=$(api "$A/pulls/$n" | jq -r .merged)
printf ' PR #%s merged=%s\n' "$n" "$m"
[ "$m" = true ] || die "PR #$n is not merged yet. Merge 8, 5, 6, 7 in that order, then re-run."
done
say "STEP 1 — force Flux to apply the sidecar now, not on the 10m interval"
./infra/devcontainer/dev.sh run bash -c \
'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-1 --with-source'
say "STEP 2 — CHECKPOINT: the converted pod must come up healthy"
./infra/devcontainer/dev.sh run bash -c '
kubectl --context k3s-prod -n gitea-actions rollout status deploy/act-runner --timeout=180s
echo "--- sidecar (initContainers) ---"
kubectl --context k3s-prod -n gitea-actions get deploy act-runner \
-o jsonpath="{range .spec.template.spec.initContainers[*]}{.name} restartPolicy={.restartPolicy}{\"\n\"}{end}"
kubectl --context k3s-prod -n gitea-actions get pod -l app.kubernetes.io/name=act-runner' \
|| die "converted act-runner pod is not healthy -- REVERT PR #7 before anything else"
say "STEP 3 — CHECKPOINT: a runner must be ONLINE again"
n=0
for i in $(seq 1 40); do
n=$(api "https://${HOST}/api/v1/admin/actions/runners" | jq -r '[(.runners//[])[]|select(.status=="online")]|length')
printf '\r online runners: %s (%ss) ' "$n" "$((i*5))"
[ "${n:-0}" -ge 1 ] && { echo; break; }
sleep 5
done
[ "${n:-0}" -ge 1 ] || die "no runner online after the conversion -- REVERT PR #7"
say "STEP 4 — CHECKPOINT: a REAL workflow must go green ON THE CONVERTED POD"
MAIN_SHA=$(api "$A/branches/main" | jq -r .commit.id)
echo " watching the post-merge push run on main (${MAIN_SHA:0:8})..."
st=""
for i in $(seq 1 60); do
st=$(api "$A/commits/$MAIN_SHA/status" | jq -r .state)
printf '\r main gate: %-8s (%ss) ' "$st" "$((i*10))"
case "$st" in success) echo; break;; failure|error) die "gate FAILED on the converted pod -- REVERT PR #7";; esac
sleep 10
done
[ "$st" = success ] || die "gate never completed on the converted pod -- REVERT PR #7; do NOT enable enforcement"
echo " the converted sidecar ran a real job green. Enforcement is now safe."
say "STEP 5 — require the gate + one approval"
"$D/require-gate.sh"
say "STEP 6 — prove a direct push is rejected (do not trust the API's word)"
git fetch -q origin && git checkout -q main && git reset -q --hard origin/main
git commit -q --allow-empty -m 'protection probe'
if git push origin main 2>&1 | grep -q 'protected branch'; then
echo " CONFIRMED: push to main rejected by the pre-receive hook."
else
echo " WARNING: the push was NOT rejected -- protection is not doing what it claims."
fi
git reset -q --hard origin/main
say "FINAL STATE"
api "$A/branches/main" | jq -r '{protected,user_can_push,enable_status_check,status_check_contexts,required_approvals}'
+4 -3
View File
@@ -26,9 +26,10 @@ creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
# feat/ prefix ON PURPOSE. act_runner capacity is 1, and gate.yml's PUSH trigger covers
# feature/** fix/** docs/** chore/** but NOT feat/** — so a chore/ branch would fire a
# second, redundant run that SERIALISES ahead of the PR run and doubles the wall clock.
# Branch prefix is now cosmetic. It used to matter: gate.yml's push trigger covered
# feature/** fix/** docs/** chore/** but not feat/**, and with act_runner at capacity 1 a
# matching prefix fired a redundant run that serialised ahead of the PR run. Those
# per-branch push triggers were deleted, so no prefix produces a duplicate run any more.
BRANCH="feat/plex-claim-token-$(date +%H%M%S)"
el "committing on $BRANCH"