Compare commits

...
Author SHA1 Message Date
jakman85andClaude Opus 5 e7508931ca fix(ci): convert dind to a native sidecar; delete both hand-rolled wait loops
gate / gate (push) Successful in 1m34s
gate / gate (pull_request) Successful in 1m35s
gate / conformance (push) Has been skipped
gate / conformance (pull_request) Has been skipped
Executes the FOLLOW-UP that has been carried in this manifest since the readiness fix,
now that k3s is v1.36 and sidecar containers are GA.

dind moves into initContainers as the first entry with restartPolicy: Always -- the one
field that makes an initContainer a sidecar: it starts before the regular containers,
runs alongside them, and is terminated after them. kubelet therefore enforces the
ordering that the runner used to hand-roll, so the runner drops its command: override
and runs the image's own entrypoint.

The startupProbe execs a full TLS-verified `docker ... info`, NOT a tcpSocket probe. A
tcpSocket probe would re-open the exact gap the old Stage 1 existed to close: dockerd
writes the TLS certs and only THEN starts listening, so the port can accept a connection
while the certs are absent, and act_runner hard-exits on a missing
/certs/client/ca.pem. `docker info` proves both that the certs are usable and that the
daemon answers. periodSeconds 1 x failureThreshold 60 preserves the previous 60s cap.

It also removes the one-per-start dockerd log line
  http: TLS handshake error from 127.0.0.1: EOF
which was the old `nc -z` probe opening a TCP connection and closing it without a TLS
handshake -- an error we were generating ourselves and then reading in the logs.

SECONDARY WIN: deleting the shell deletes an envsubst hazard. apps-wave-1 runs postBuild
substitution over this ENTIRE manifest, comments included, and those loops depended on
bare dollar-names passing through untouched -- a braced or double-dollar spelling would
have silently unbounded the waits or failed the whole Kustomization. The only dollar left
in the file is the intended ${BASE_DOMAIN}.

LANDS ALONE, DELIBERATELY. This Deployment is the merge gate for the entire repository:
if the conversion is wrong, nothing can merge. Merge this BEFORE enabling the required
status check, and watch a real workflow run green on the converted pod before doing so --
otherwise a bad conversion locks the repo with no way to revert through the gate.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 01:54:47 -05:00
gitea-admin 27d0ef3324 Merge pull request 'chore(plex): fresh plex.tv claim token' (#4) from feat/plex-claim-token-013124 into main
gate / gate (push) Successful in 1m33s
gate / conformance (push) Has been skipped
2026-09-29 06:33:04 +00:00
jakman85 bbd4f682a6 chore(plex): fresh plex.tv claim token
gate / gate (pull_request) Successful in 1m35s
gate / conformance (pull_request) Has been skipped
Injected inside the plex.tv claim window per runbooks/plex-claim.md.
2026-09-29 01:31:24 -05:00
jakman85 0391139240 Merge pull request 'feat(plex): wire PLEX_CLAIM from a SOPS Secret (placeholder value)' (#1) from feat/plex-claim-sops-secret into main
gate / gate (push) Successful in 1m34s
gate / conformance (push) Has been skipped
Reviewed-on: #1
2026-09-29 06:18:14 +00:00
jakman85 e9f441b5ce Merge pull request 'docs(gitea): a skipped job reports SUCCESS, not pending' (#3) from fix/conformance-context-is-fail-open into main
gate / gate (push) Successful in 1m34s
gate / conformance (push) Has been skipped
Reviewed-on: #3
2026-09-29 06:12:30 +00:00
jakman85andClaude Opus 5 234cc6a71a docs(gitea): a skipped job reports SUCCESS, not pending -- correct the deadlock claim
gate / gate (push) Successful in 1m33s
gate / gate (pull_request) Successful in 1m32s
gate / conformance (push) Has been skipped
gate / conformance (pull_request) Has been skipped
The runner came up on 2026-09-29 and the first CI jobs this repo has ever run went
green, which falsified a claim merged in #2.

MEASURED:
  gate / gate (pull_request)          success   1m58s (PR 1), 1m34s (PR 2)
  gate / gate (push)                  success   1m32s (PR 2 only)
  gate / conformance (pull_request)   success   "Has been skipped"

#2 said listing the conformance context as a required check would deadlock every PR,
because the job is `if: false` and never runs. That is wrong. Gitea reports a skipped
job as SUCCESS. Requiring it would not deadlock anything -- it would pass VACUOUSLY,
forever, while nothing ran.

The conclusion is unchanged (never require it) but the reason inverts, and the corrected
reason matters more: under HARD RULE 4 a permanently-green check that verifies nothing is
worse than a deadlock. A deadlock is loud. This is the exact fail-open shape the
2026-09-18 audit and this one both keep finding.

Also records that the (push) contexts only appear for branches matching gate.yml's push
trigger -- feature/**, fix/**, docs/**, chore/** -- and NOT feat/**, which is why PR 1's
feat/ branch produced only pull_request contexts. Require the pull_request context alone,
or PRs from non-matching prefixes never satisfy the rule.

CLAUDE.md's enforcement table is updated: the runner half is genuinely fixed and observed
green, the branch-protection half is not. Adds the warning that a green tick is not by
itself evidence that anything ran.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 01:07:13 -05:00
jakman85 69a0d1a1c7 Merge pull request 'docs: delivery contract for agents + record that the merge gate does not exist' (#2) from docs/ci-delivery-flow-for-agents into main
gate / gate (push) Successful in 1m34s
gate / conformance (push) Has been skipped
Reviewed-on: #2
2026-09-29 05:50:31 +00:00
jakman85 d3a0d587a4 Added gitea action runner secret
gate / gate (push) Successful in 1m32s
gate / gate (pull_request) Successful in 1m34s
gate / conformance (push) Has been skipped
gate / conformance (pull_request) Has been skipped
2026-09-29 00:50:02 -05:00
jakman85andClaude Opus 5 fe862ab527 infra(gitea): recover the Actions runner before anything requires the gate
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
gate / gate (pull_request) Successful in 1m32s
gate / conformance (pull_request) Has been skipped
No runner is registered at repo, org or site level, so gate.yml has never executed.
PR #1 sat pending for 30.6 minutes across 122 polls and never reported. Requiring the
gate as a status check in that state would lock the repo permanently.

Adds runner-token-rotate.sh: mints a fresh instance-level registration token from the
Gitea API and writes it into the SOPS secret via sops set, without ever printing the
value.

Records the likely root cause, which a new token alone does NOT fix. act_runner
registers once and persists its identity in /data/.runner on the act-runner-data PVC.
If Gitea no longer holds a matching runner row, the pod starts, sees .runner, SKIPS
registration entirely, and polls with an identity Gitea does not recognise. Symptom:
zero runners, pod running, nothing obviously broken. Clearing the PVC is what forces a
genuine re-registration.

The README now sequences runner-first, enforcement-second, and requires CONFIRMING the
runner reports online rather than assuming it. That assumption is the whole defect:
DECISIONS-LOG recorded the runner as online on 2026-08-31 and nobody checked since.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 00:43:47 -05:00
jakman85andClaude Opus 5 a2665c0511 infra(gitea): script the branch protection that was never applied
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
gate / gate (pull_request) Successful in 1m33s
gate / conformance (pull_request) Has been skipped
Gitea repo settings live in its own database, not in a manifest Flux reconciles, so
they are drift-prone and invisible to the conformance gate -- which is how main came
to accept direct pushes while the log recorded it as protected.

Adds infra/gitea/branch-protection.sh (blocks direct pushes to main) and a README
carrying the measured before-state and the ORDER the remaining steps must happen in.

The script deliberately leaves enable_status_check:false and required_approvals:0.
Both would deadlock today: there is no runner to satisfy a required context, and the
only human account is not yet a repo collaborator, so no one could supply an approval.
The README sequences both.

It prints a push probe to run afterwards. The API response is not proof -- Gitea may
let site admins bypass protection, and that has to be observed, not assumed.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 00:20:12 -05:00
jakman85andClaude Opus 5 5a9e455149 docs: delivery contract for agents + record that the merge gate does not exist
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
gate / gate (pull_request) Successful in 1m33s
gate / conformance (pull_request) Has been skipped
Going to land the plex claim wiring through the documented PR flow surfaced that
the flow has never run. PR #1 is the first pull request this repo has ever had.

Verified against the live Gitea, not inferred from the docs:
  * branch_protections is [] -- main is protected:false, enable_status_check:false,
    status_check_contexts:[], required_approvals:0, user_can_push:true
  * ZERO Actions runners registered at repo, org and site level

DECISIONS-LOG 2026-08-31 asserts a "required status check" and a runner
"incluster-dind online". Both are false of the live system. The gate is neither
enforced nor able to run, and that went unnoticed because the log claimed
otherwise -- the same fail-open class as the two holes found on 2026-09-18.

Adds CLAUDE.md, the first agent-facing delivery doc here: the
branch -> PR -> gate -> approval -> merge rule, stated alongside the honest fact
that enforcement is currently weaker than the rule, so agents follow the rule
rather than what they can get away with. Carries the toolbox gate invocation, the
two git-worktree workarounds, the rc 0/1/3 contract, and secret handling.

Records the ordering trap: enabling a required status check before a runner exists
deadlocks every PR permanently, including the one that would fix it.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 00:11:58 -05:00
8 changed files with 449 additions and 112 deletions

No files matched your search

+117
View File
@@ -0,0 +1,117 @@
# CLAUDE.md — how work lands in this repo
Read this before doing anything durable. It is the delivery contract for AI coding
sessions and agents. `DECISIONS-LOG.md`'s last entry is the current PM handoff and
takes precedence on *what* to work on; this file governs *how* it lands.
## 1. Never push to `main`. Ever.
Every durable change reaches `main` exactly one way:
branch -> push the branch -> open a PR -> CI gate green -> human approval -> merge
There is no other path. Specifically:
- **Do not `git push origin main`.**
- **Do not `git push origin HEAD:main`.** Same thing wearing a hat.
- **Do not merge your own PR.** A human approves and clicks merge.
- **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
> **Many runbooks under `runbooks/` still say `git push origin main` IS the deploy.**
> That wording is **stale** — it predates the PR flow. It describes what Flux does
> (main is what the cluster follows), not what you are permitted to do. Honour this
> file, not that line. Fixing those runbooks is tracked work.
## 2. Enforcement is currently WEAKER than this rule. Follow the rule anyway.
Verified against the live Gitea on 2026-09-29:
| Expected | Actual |
|---|---|
| `main` protected, direct push rejected | **No branch protection.** `protected: false`, `user_can_push: true` |
| PR blocked until the gate passes | **`enable_status_check: false`**, `status_check_contexts: []` |
| Gate runs on every PR | **Fixed 2026-09-29.** Runner `incluster-dind` is online; `gate / gate (pull_request)` runs and passes in ~1m30s–2m |
| Approval required before merge | **`required_approvals: 0`** |
So nothing mechanically stops you from pushing to `main` right now. **That is a gap
being closed, not permission.** `DECISIONS-LOG.md` (2026-08-31) claimed the gate was a
"required status check" with a runner online; neither was true, which is how this went
unnoticed for a month. The runner half is now genuinely fixed and observed green. The
protection half is not — see `infra/gitea/`.
**Consequence for you:** CI now runs, but it does not yet *block* anything. Always run
the toolbox gate yourself and quote its numbers. Never describe a PR as "green" on the
strength of a check you have not seen report — and note that a **skipped** Gitea job
reports as `success`, so a green tick is not by itself evidence that anything ran.
## 3. Gate before you push
./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
Exit codes are a contract — do not flatten them:
| rc | Meaning |
|---|---|
| 0 | PASS |
| 1 | BLOCKED — something failed |
| 3 | UNDETERMINED — a stage did not run. **Never report this as green.** |
`spec-immutability` reaches rc 3 whenever the sibling spec repo is not checked out
beside this one; that single skip is expected and tolerated. Any *other* skip is a
real gap in your run.
**Never gate on the host.** `promtool` and `kustomize` are not on the host `$PATH`,
so checks SKIP and you get a false rc 3. Only the pinned toolbox result counts.
### Running the gate from a git worktree
Two unrelated failures, both with known workarounds — put both in any agent brief:
1. **`dev.sh` refuses to start:** `infra/ansible/.cache/prod.kubeconfig` is untracked
and exists only in the main checkout. Do **not** copy it (it is a credential).
Use `DEV_KUBECONFIG=/dev/null` — the gate contacts no cluster, so L2/L3 scaffold-skip
normally.
2. **Every `git` call inside the container fails:** a worktree's `.git` is a pointer to a
gitdir outside the mounted root. `dev.sh` takes no extra-mount flag, so inject one via
the `ENGINE` hook — a shim that intercepts `run` and re-execs
`podman run -v /path/to/main/.git:/path/to/main/.git:ro "$@"`. A traceback in
`tracked_files()` is this bug, not a broken tree.
## 4. Branch naming
Use `feat/`, `fix/`, `docs/`, `chore/`. Note that `.gitea/workflows/gate.yml`'s **push**
trigger lists `feature/**` — not `feat/**` — so branch pushes get no fast-feedback run.
The `pull_request` trigger fires regardless of branch name, so the PR gate is unaffected.
## 5. Cluster access
`kubectl` does not work on the operator workstation; the toolchain lives only in the
pinned container. Always name the context explicitly:
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
`dev.sh` merges both clusters and picks a current-context **per invocation**; that
default has already changed once. Never rely on it. Do not run two `dev.sh`
invocations concurrently.
**Mutating `kubectl` and `flux reconcile` are operator actions, not agent actions.**
## 6. Secrets
- **Never `sops -d` a file into a terminal, a log, or a transcript.**
- **Never print a Secret's `.data`/`.stringData`** when checking whether it exists —
list key names or object names only. A password has been leaked into a transcript
this way before.
- Encrypted files are `*.sops.yaml` or live under `secrets/`. The root `.sops.yaml` is
the single creation-rules file; do not add a second one in a subtree.
- A short-lived credential (e.g. a plex.tv claim token, ~minutes) cannot survive the PR
flow. Do not design around committing one — read the expiry from the issuing page and
treat a burnt token as free to retry.
## 7. Before you finish
Append a handoff entry to `DECISIONS-LOG.md` describing what you changed, what you
verified, and what you did **not** verify. State gaps plainly — an honest
"UNVERIFIED" is worth more than an optimistic "green", and this repo has already been
burned by the difference.
+65
View File
@@ -4716,3 +4716,68 @@ asleep. `git worktree remove` preserves branches and their commits, so only that
6. **Five `public` services have no working public path** — waiting on WAN forwarding, deliberately.
7. Backlog cherry-picks: `feat/egress-gateway-staleness-alert`, two `rescued/*` fixes, two rendered
manifests nothing applies (the keycloak one would stand up a conflicting instance).
## 2026-09-29 — The CI/CD merge gate does not exist: no runner, no branch protection. **PM HANDOFF.**
Went to land the plex claim wiring through the documented PR flow and discovered the flow has
never run. **PR #1 is the first pull request this repository has ever had.** It is still queued.
### What is actually true of the live Gitea (verified via API, 2026-09-29)
| Claimed (DECISIONS-LOG 2026-08-31) | Measured |
|---|---|
| "required status check" on `main` | `branch_protections` returns `[]`. `protected: false`, `enable_status_check: false`, `status_check_contexts: []`, `required_approvals: 0`, `user_can_push: true` |
| "runner `incluster-dind` online (ns gitea-actions)" | **Zero runners registered** at repo, org and site level |
Both halves of that entry are false. Nothing enforces the gate, and the gate cannot run.
PR #1's two contexts (`gate / gate`, `gate / conformance`) sat `pending` for 14+ minutes with
nothing to execute them, and will sit there indefinitely.
This is a **fail-open governance hole**, the same class as the two found on 2026-09-18 — and it
was invisible precisely because the log asserted the opposite. Every "the gate is enforcing"
statement since 2026-08-31 rested on an unverified claim.
### Ordering trap — read before fixing
**Do not enable required status checks until a runner is live.** Requiring
`gate / gate (pull_request)` with no runner deadlocks every PR permanently, including the PR
that would fix it. Sequence: runner first, verify the gate actually runs and passes on a real
PR, *then* turn on enforcement with the context name CI actually reported.
Related: `gate.yml`'s `conformance` job is `if: false` by design (honest skip, never fake green),
but it still publishes a `gate / conformance (pull_request)` context. If that context is ever
listed as required, it deadlocks every PR for the same reason. Wire it or exclude it.
### Delivery contract written down
Added `CLAUDE.md` at the repo root — the first agent-facing delivery doc here. It states the
branch -> PR -> gate -> approval -> merge rule, records that enforcement is currently weaker than
the rule (so agents follow the rule rather than what they can get away with), and carries the
toolbox-gate invocation, the two git-worktree workarounds, the rc 0/1/3 contract, and the secret
handling rules.
It also flags that **37 of 39 runbooks still instruct `git push origin main`**. Those were
accurate before the PR flow and are stale now. Correcting them is tracked, not done.
### Also found, not fixed
- `gate.yml`'s push trigger lists `feature/**`; this repo's branches use `feat/**`, so branch
pushes get no fast-feedback run. PR trigger is unaffected.
- The gitea-actions runner is GitOps-managed (`apps/wave-1/gitea-actions.yaml` +
`gitea-actions-runner-secret.sops.yaml`), so it is deployed but not registering — most likely a
stale registration token in that SOPS secret. Needs in-cluster diagnosis (operator).
- **Plex is still unclaimed and it was never checked** whether it is already claimed. Do that
before spending a token — `runbooks/plex-claim.md` §First commands.
- `runbooks/plex-claim.md` asserts a "~4-minute TTL" for a plex.tv claim token in four places,
**uncited**, and the whole stopwatch procedure is built on it. Upstream `pms-docker`'s README
says nothing about expiry. The number is plausible community folklore, not a measured fact.
The authoritative reading is the countdown rendered on plex.tv/claim at issue time.
### Next steps
1. **Operator:** diagnose the gitea-actions runner; get one registered.
2. **Operator:** enable branch protection on `main` — push blocked, approvals required — and only
then add the gate as a required status check.
3. Land `feat/plex-claim-sops-secret` (PR #1) once the gate can actually run.
4. Claim plex, after checking it is not already claimed.
5. Correct the 37 runbooks' `git push origin main` wording.
+3 -3
View File
@@ -46,7 +46,7 @@ metadata:
app.kubernetes.io/name: plex
type: Opaque
stringData:
PLEX_CLAIM: ENC[AES256_GCM,data:8O9nvFw+6OR+53mXvHfe321aSmYUQcZuAqLKZpcSlUwNyeU+Nbp/tRPBlg==,iv:EbvvOsXlXjAFOewnWbK28ErmUdWcJJN0Q5Gm53El3ZM=,tag:wXaInGFQ+wqUt765vvjCDA==,type:str]
PLEX_CLAIM: ENC[AES256_GCM,data:nLmzeUBWRhRjUakrGiGzQpQuqZNcIYvRH6M=,iv:z0on/dArsNejNLrp3V1nSiNLFzKcREhPuMO1634CwQw=,tag:Ih23CT1JYna/5lfYQUb1Dw==,type:str]
sops:
kms: []
gcp_kms: []
@@ -62,8 +62,8 @@ sops:
QUtYdHEvN042ajNoNktOLzlYQmQrc1kK4NHozMbXVzNGYbUBRbK6a9Uw/sVPAfe6
RW5KWzZ4I/hloqjN3zzVISoSMieMkBGDnF5qZS5N6FqIc6/HrsDD8A==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-09-29T03:25:20Z"
mac: ENC[AES256_GCM,data:u5uVmOdXN78UaKUMkPZzqeW84222+tcG57rFKIKcvRKwJkzMTOuSFOeQpKh4eK4KZXVKK7+Q3mFf5G8a+jjuKjYbjSCpR38AF3TJv3xrj8fhnqR+8QHRp8tsgA0jJZBS55N7KKDyOztLFo49sI7mNnlrUwhWTQUkbLgN1LxF5MA=,iv:P4W9G2mI/3VTkvSrDZvnkJ0Qxb1nFmXcF35CrSNhako=,tag:yVsg7xsUALUoCJH0cHfQjw==,type:str]
lastmodified: "2026-09-29T06:31:16Z"
mac: ENC[AES256_GCM,data:62BmBAq45CD2g6LDyx+Ho4bTfrhJcaro/RB/iM+OaRc4a/D83MhjbcBL11Uz/O726inqhkvCKDJ/fFemoc50JDbtC79cYMaxrGJNjFM0sdckWrowPvh6ie7HHHUBRBaefl+gNfnc+pbvcncvRzwA4RyzcZse1wbxKfHN8FjEopE=,iv:h0cMShs3tIgnY6eruUYymYrcLFzQZLk/uXQJ/Jsd5ek=,tag:nZ9Pau7p4xWT1QCN1vv2vA==,type:str]
pgp: []
encrypted_regex: ^(data|stringData)$
version: 3.9.4
@@ -14,7 +14,7 @@ metadata:
app.kubernetes.io/name: act-runner
type: Opaque
stringData:
token: ENC[AES256_GCM,data:W48mEtBqGeukX13lMhrhrer+zB21cLVp0acy3m1Xy/oo89DqO8FX/Q==,iv:wfTUU7RiFLdEc1fmEFS9DfE2SmGCFLPK17m0IKspft0=,tag:UhoXA47p7386KHcYNbPzCg==,type:str]
token: ENC[AES256_GCM,data:f574mAUkLvMGq0TYycwCd4tduMuBgQLiTg/140kuRDgFUYYm1ZoSjQ==,iv:Tn7gFWBLm2hHk/BbfVX7qIZegwZp9ah1cMou5i2O91E=,tag:TkdSiMI4sdCNn2JuN++Pig==,type:str]
sops:
kms: []
gcp_kms: []
@@ -30,8 +30,8 @@ sops:
MDZLUE52QzJXbFlrNWxOcVhJbDNnY00KxCpFkbhhCuk4Fa8CC3C4+R9AcsUnaXQR
UjAfwKFoXAneXUih7TTXMbR/6+VgQBww5Bmwpn7jVLnzP0N0pIZVTA==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-08-26T04:48:14Z"
mac: ENC[AES256_GCM,data:b99/zkL332RIT3FYBa1sR3o9+pwlmrUgKvVHawbjWYSJZ7ERUfO+LXuhKJ0HKrp6i1JfkurZlRO27iBcfrBisAXS9KLaqmsCJf4kKfFc4skANrmKCzprrXUmZexr1uiE8eDUzjNCLNxSCZfnFR6d/AkMXJqi6Hsj9/O+wH2lWsc=,iv:3DIPszSdxK81ERO3o/nTs33qQYHaAKLNcGP4qlZViFc=,tag:/B6VKkzMUwh0byNUicS5wQ==,type:str]
lastmodified: "2026-09-29T05:48:35Z"
mac: ENC[AES256_GCM,data:nobqr794nrdRYFrd8Ct8bw9M8NtftJXawRGk9zrL+pTWahwlWrwXVtOAxMGEPVojz3WemsxTJESzmzy0gcT1QcrjpYTsF2MVHcr6+umk8q9gAh2enyxy/0QkYiSt6XjvRVqfqleLanEkeIq1FPf2zhSLSJaiw4/oZTfkCHD1/Zg=,iv:JxaP0htmGcvE4ZU1YAmGF4A6JkejV29NNygyMl1OKF8=,tag:G70NvcPyVuagXhugWoUfTw==,type:str]
pgp: []
encrypted_regex: ^(data|stringData)$
version: 3.9.4
+61 -106
View File
@@ -99,70 +99,71 @@ spec:
# CI holds no cluster credentials — do not mount a SA token into the runner or
# (via the shared netns) the job containers (R-SEC-02, docs/14).
automountServiceAccountToken: false
# ---- Docker-in-Docker: the job executor daemon (privileged) -------------------
#
# NATIVE SIDECAR (k3s v1.36; sidecar containers are GA). It is an initContainer with
# `restartPolicy: Always`, which means: it starts BEFORE the regular containers, keeps
# running alongside them, and is terminated after them. kubelet holds `runner` until
# this container's startupProbe passes, so the ordering is enforced by the kubelet
# instead of hand-rolled shell in the runner.
#
# This REPLACES two bounded wait-loops that used to live in the runner's `command:`.
# Deleting them also deletes an envsubst hazard: clusters/base/apps-wave-1.yaml runs
# postBuild substitution over this ENTIRE manifest, comments included, and those loops
# depended on bare dollar-names surviving it untouched. There is now no shell here at
# all, so there is nothing to escape or mis-escape.
#
# WHY THE PROBE IS `exec docker ... info` AND NOT `tcpSocket`. A tcpSocket probe would
# re-open the exact gap the old Stage 1 existed to close: dockerd writes the TLS certs
# and only THEN begins listening, so the port can accept a connection while the certs
# are absent or incomplete -- and act_runner hard-exits on a missing
# /certs/client/ca.pem. A full TLS-verified `docker info` proves BOTH that the certs
# are usable and that the daemon answers. It also removes the one-per-start
# "TLS handshake error ... EOF" that the old `nc -z` probe logged in dockerd.
#
# periodSeconds 1 x failureThreshold 60 preserves the previous 60s cap.
initContainers:
- name: dind
restartPolicy: Always # <- this field is what makes it a SIDECAR
image: docker:27-dind
securityContext:
privileged: true
env:
# Generate + publish TLS certs into the shared emptyDir; dockerd then listens
# on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
- name: DOCKER_TLS_CERTDIR
value: /certs
volumeMounts:
- name: certs
mountPath: /certs
- name: dind-storage
mountPath: /var/lib/docker
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
memory: 2Gi
startupProbe:
exec:
command:
- docker
- --host=tcp://127.0.0.1:2376
- --tlsverify
- --tlscacert=/certs/client/ca.pem
- --tlscert=/certs/client/cert.pem
- --tlskey=/certs/client/key.pem
- info
periodSeconds: 1
failureThreshold: 60
containers:
# ---- act_runner: registers with Gitea, dispatches jobs to the DinD daemon ----
- name: runner
image: gitea/act_runner:0.2.11
# TWO-STAGE BOUNDED WAIT on the DinD sidecar, then exec the image's normal
# entrypoint (/sbin/tini -- /opt/act/run.sh), which registers-from-env on first
# boot and runs `act_runner daemon`.
#
# STAGE 1 — the certs must EXIST: act_runner reads /certs/client/ca.pem at startup
# and hard-exits if it is missing (a startup race against the sidecar).
# STAGE 2 — the daemon must actually ACCEPT a connection: the cert FILE appearing is
# necessary but NOT sufficient. dockerd writes the certs and only THEN begins
# listening on :2376, so a runner that proceeds on the file alone still hard-exits
# with "cannot ping the docker daemon at tcp://localhost:2376" — observed
# crashlooping 11 times in ~8 minutes on a fresh pod after a node bounce.
# Both loops are bounded (60 x 1s) and exit 1 on timeout, so a genuinely dead sidecar
# surfaces as a CrashLoopBackOff with a clear message instead of hanging forever.
#
# ENVSUBST SAFETY — the BARE $i / $j below are CORRECT. Do not "tidy" them into the
# braced form or the double-dollar form; both are wrong here, for different reasons.
# clusters/base/apps-wave-1.yaml carries postBuild.substituteFrom cluster-vars, so
# Flux envsubst runs over this ENTIRE manifest — YAML comments very much included.
# Its contract (drone/envsubst; the same one spelled out in clusters/base/apps-vms.yaml,
# and re-verified against this exact file with `flux envsubst`, flux v2.4.0) is:
# - A bare $name is NOT a reference to envsubst: it passes through untouched, and
# that holds even when the name IS set in cluster-vars. So $i and $j reach the
# cluster verbatim and both 60s bounds below are real, enforced bounds.
# - $((i + 1)) is likewise untouched: "(" cannot begin a variable name.
# - A BRACED dollar-name IS a reference and gets replaced — with the empty string
# when the name is unset. One of those here would make a bound read [ "" -ge 60 ],
# which errors, short-circuits the &&, and silently turns the wait UNBOUNDED.
# - A DOUBLE-dollar braced name is envsubst's escape and survives as the braced
# form — still wrong for us, because on any path where envsubst does NOT run
# (a plain `kustomize build` or `kubectl apply -k` of this directory) the shell
# reads that leading double-dollar as its own PID, so a bound would compare a
# string like 1234{i} against 60. The bare form is the only spelling that is
# correct on BOTH paths.
# - A braced name containing punctuation is a HARD ERROR ("unable to parse variable
# name") that fails this whole Kustomization — and this Kustomization gates every
# merge in the repo. So never write one anywhere in this file, INCLUDING in a
# comment. That is why this note names the three forms in words rather than
# showing them: an earlier draft of it spelled them out literally and broke
# `flux envsubst` on the file. (Same trap the indexed-expansion warning in
# clusters/base/apps-vms.yaml describes.)
command:
- /bin/sh
- -c
- |
# Stage 1: the sidecar's TLS client certs must appear in the shared emptyDir.
i=0
until [ -f /certs/client/ca.pem ]; do
i=$((i + 1))
[ "$i" -ge 60 ] && { echo "dind certs never appeared" >&2; exit 1; }
echo "waiting for dind TLS certs ($i)"; sleep 1
done
# Stage 2: dockerd must actually accept a connection on :2376. busybox nc in this
# Alpine-based image (gitea/act_runner:0.2.11) supports -z (scan) and -w (timeout).
j=0
until nc -z -w1 127.0.0.1 2376; do
j=$((j + 1))
[ "$j" -ge 60 ] && { echo "dind daemon never accepted a connection on 2376" >&2; exit 1; }
echo "waiting for dind daemon on :2376 ($j)"; sleep 1
done
exec /sbin/tini -- /opt/act/run.sh
# No `command:` override. The image's own entrypoint (/sbin/tini -- /opt/act/run.sh)
# registers from env on first boot and runs `act_runner daemon`. The two wait-loops
# that used to live here are gone: kubelet now holds this container until the dind
# sidecar's startupProbe passes (see initContainers above).
env:
- name: GITEA_INSTANCE_URL
value: https://gitea.${BASE_DOMAIN}
@@ -198,52 +199,6 @@ spec:
memory: 128Mi
limits:
memory: 512Mi
# ---- Docker-in-Docker: the job executor daemon (privileged) ----
#
# FOLLOW-UP (deliberately NOT bundled into the readiness fix above — own change):
# Convert this container into a NATIVE SIDECAR and delete both shell wait-loops.
# k3s is now v1.36, where sidecar containers are GA, so kubelet can enforce the
# ordering that the loops above hand-roll — which also removes the envsubst hazard
# from this file entirely (no shell, no $i/$j, nothing to escape or mis-escape).
# Shape to build:
# 1. Move this whole container into `initContainers:` as the FIRST entry.
# 2. Add `restartPolicy: Always` to it. That single field is what makes an
# initContainer a sidecar: it starts before the regular containers, keeps
# running alongside them, and is terminated after them.
# 3. Give it a `startupProbe` that succeeds only when dockerd accepts a TLS
# connection — e.g. `exec` a `docker -H tcp://127.0.0.1:2376 --tlsverify
# --tlscacert /certs/client/ca.pem --tlscert /certs/client/cert.pem
# --tlskey /certs/client/key.pem info`. Do NOT use a bare `tcpSocket` probe:
# it would re-open the same "port accepting but TLS/certs not ready" gap that
# Stage 1 above exists to close. `periodSeconds: 1` + `failureThreshold: 60`
# preserves today's 60s cap.
# 4. kubelet then holds the `runner` container until that startupProbe passes, so
# `runner` can drop its `command:` override completely and run the image's
# default entrypoint.
# Why it is a separate change: this Deployment is the ENFORCED MERGE GATE for this
# entire repository. If the conversion is wrong, nothing can merge — so it must land
# alone, where the failure is unambiguous and not entangled with anything else, and
# only after a real workflow run has been observed green on the converted pod.
- name: dind
image: docker:27-dind
securityContext:
privileged: true
env:
# Generate + publish TLS certs into the shared emptyDir; dockerd then listens
# on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
- name: DOCKER_TLS_CERTDIR
value: /certs
volumeMounts:
- name: certs
mountPath: /certs
- name: dind-storage
mountPath: /var/lib/docker
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
memory: 2Gi
volumes:
- name: data
persistentVolumeClaim:
+82
View File
@@ -0,0 +1,82 @@
# infra/gitea — platform settings that are NOT GitOps-managed
Gitea's own repo settings (branch protection, required status checks, approvals) live
in Gitea's database, not in a manifest Flux reconciles. They are therefore **drift-prone
and invisible to the gate** — exactly how the state below went unnoticed.
## The gap this closes
Verified against the live API on 2026-09-29, against a `DECISIONS-LOG.md` entry
(2026-08-31) asserting the opposite:
| Claimed | Measured |
|---|---|
| `main` is a protected branch, gate is a required status check | `branch_protections` → `[]`; `protected:false`, `enable_status_check:false`, `required_approvals:0`, `user_can_push:true` |
| Runner `incluster-dind` online | **zero** runners registered at repo, org and site level |
So `main` accepted direct pushes, and the gate could not run at all. `CLAUDE.md` is the
rule; this directory is how the rule gets enforced by the machine.
## `branch-protection.sh`
Blocks direct pushes to `main`. Run it, then run the push probe it prints — **verify,
do not trust the API response**, since Gitea may let site admins bypass protection.
./infra/gitea/branch-protection.sh
It authenticates with the credential `git` already has stored for this host, and never
prints the secret.
## `runner-token-rotate.sh`
Mints a fresh instance-level registration token from the Gitea API and writes it into
`apps/wave-1/gitea-actions-runner-secret.sops.yaml` via `sops set`. **The token value is
never printed.**
./infra/gitea/runner-token-rotate.sh
A fresh token is usually **not sufficient on its own**. `act_runner` registers exactly once
and persists its identity in `/data/.runner` on the `act-runner-data` PVC. If Gitea no
longer holds a matching runner row, the pod starts, sees `.runner`, **skips registration**,
and polls with an identity Gitea does not recognise — so it never appears in the runner
list. That is precisely the observed symptom: zero runners, pod running, no error anyone
looks at. Clearing the PVC is what forces a real re-registration; the script prints the
commands.
## Sequence — order matters
1. **Get an Actions runner registered** — `runner-token-rotate.sh`, then clear the PVC so
the runner actually re-registers, then confirm it shows `online` in the API. Do not
skip the confirmation: the whole reason this went unnoticed is that a runner was
*recorded* as online without anyone checking.
2. **Watch the gate actually report on a real PR**, and note the context name it produced.
3. **Only then set `enable_status_check: true`** with that exact context.
4. Add the human reviewer as a repo collaborator, then raise `required_approvals` to 1.
> **Do not do step 3 before step 1.** A required status check with no runner leaves every
> PR permanently pending — including the PR that would fix it. That half is measured: PR #1
> sat `pending` for 30.6 minutes across 122 polls with no runner registered, and never
> reported.
### Measured, once the runner came up (2026-09-29)
The first CI jobs this repo ever ran, on PR #1 and PR #2:
| Context | Result | Time |
|---|---|---|
| `gate / gate (pull_request)` | success | 1m58s (PR #1), 1m34s (PR #2) |
| `gate / gate (push)` | success | 1m32s (PR #2 only) |
| `gate / conformance (pull_request)` | **success — "Has been skipped"** | — |
**Require `gate / gate (pull_request)`. Do NOT require either `conformance` context —
but not for the reason an earlier draft of this file gave.** A skipped Gitea Actions job
does not hang a status check: Gitea reports it as **`success`** with the description
"Has been skipped". So requiring it would not deadlock anything — it would pass
*vacuously*, every time, forever, while nothing ran. Under HARD RULE 4 that is worse than
a deadlock: a deadlock is loud, a permanently-green check that verifies nothing is
exactly the fail-open shape this repo keeps finding.
The `(push)` contexts appear only for branches matching `gate.yml`'s push trigger
(`feature/**`, `fix/**`, `docs/**`, `chore/**`). `feat/**` is **not** in that list, which
is why PR #1's `feat/` branch produced only `pull_request` contexts. Require only the
`pull_request` context, or PRs from non-matching prefixes will never satisfy the rule.
+57
View File
@@ -0,0 +1,57 @@
#!/usr/bin/env bash
# Enable branch protection on main for home-network-k3s-impl.
#
# Sets enable_push:false -> no direct pushes to main; changes land only by merging a PR.
#
# enable_status_check is deliberately FALSE. There is currently NO Gitea Actions runner
# registered (verified 2026-09-29: zero at repo, org and site level), so requiring the
# gate context would deadlock EVERY PR permanently -- including the PR that fixes it.
# Turn it on only after a runner is live and you have seen the gate actually report on
# a real PR, using the context name CI itself produced.
#
# required_approvals is 0 for the same class of reason: jakman85 is a site admin but NOT
# a repo collaborator, so requiring an approval now could leave nobody able to approve.
# Add the collaborator first, then raise this to 1.
set -euo pipefail
HOST=gitea.home.keithjackson.dev
OWNER=Home-Network
REPO=home-network-k3s-impl
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"
P="$(sed -n 's/^password=//p' <<<"$creds")"
[ -n "$U" ] && [ -n "$P" ] || { echo "no stored credential for $HOST" >&2; exit 1; }
echo "authenticating as '${U}' (secret not shown)"
echo "==> creating branch protection on 'main'"
jq -n '{
branch_name: "main",
enable_push: false,
enable_status_check: false,
status_check_contexts: [],
required_approvals: 0,
block_on_rejected_reviews: true,
block_on_outdated_branch: false,
dismiss_stale_approvals: true
}' | curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
-X POST -H 'Content-Type: application/json' --data @- \
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections" \
| jq -r 'if .branch_name then "created" else "API said: \(.message // tostring)" end'
echo "==> verifying (this is the part that matters)"
curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \
| jq -r '{protected, user_can_push, required_approvals, enable_status_check, status_check_contexts}'
cat <<'NOTE'
Now prove it actually blocks you -- do not take the API's word for it. From a clean
checkout on main, with nothing to lose:
git commit --allow-empty -m 'protection probe'
git push origin main # EXPECT: rejected by the remote
git reset --hard HEAD~1 # tidy up either way
If that push SUCCEEDS, Gitea is letting site admins bypass protection and this is not
yet doing what you want.
NOTE
+61
View File
@@ -0,0 +1,61 @@
#!/usr/bin/env bash
# Mint a fresh Gitea Actions registration token and write it into the SOPS secret.
#
# The token value is NEVER printed. It goes from the Gitea API into `sops set` and
# nowhere else. (It is briefly visible in this host's process list as a sops argument;
# acceptable on a single-operator workstation, and far better than echoing it.)
#
# This ALONE may not bring the runner back -- see "IMPORTANT" at the end of the output.
set -euo pipefail
HOST=gitea.home.keithjackson.dev
SECRET=apps/wave-1/gitea-actions-runner-secret.sops.yaml
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
cd "$REPO_ROOT"
[ -f "$SECRET" ] || { echo "not in the repo root, or $SECRET is missing" >&2; exit 1; }
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"
P="$(sed -n 's/^password=//p' <<<"$creds")"
[ -n "$U" ] && [ -n "$P" ] || { echo "no stored git credential for $HOST" >&2; exit 1; }
echo "==> minting an instance-level registration token as '${U}'"
TOKEN="$(curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
"https://${HOST}/api/v1/admin/runners/registration-token" | jq -r '.token // empty')"
[ -n "$TOKEN" ] || { echo "Gitea did not return a token" >&2; exit 1; }
echo "==> got a ${#TOKEN}-character token (value not shown)"
echo "==> writing it into ${SECRET} via sops"
DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run \
sops set "$SECRET" '["stringData"]["token"]' "\"${TOKEN}\""
unset TOKEN
echo "==> done. Verify the file is still encrypted (no plaintext token):"
grep -q 'ENC\[' "$SECRET" && echo " OK - stringData is still SOPS-encrypted" \
|| { echo " REFUSING TO CONTINUE: $SECRET is not encrypted" >&2; exit 1; }
cat <<'NOTE'
Next: commit this on a branch, open a PR, and merge it so Flux applies the new Secret.
IMPORTANT -- a new token is usually NOT sufficient on its own.
act_runner registers exactly ONCE and then persists its identity in /data/.runner on the
act-runner-data PVC. If Gitea no longer has a matching runner row (DB restore, the row was
deleted, the instance was rebuilt), the pod starts, sees .runner, SKIPS registration, and
polls with an identity Gitea does not recognise -- so it never appears in the runner list,
which is exactly the observed symptom (zero runners, pod running).
So after the Secret lands, force a genuine re-registration by clearing that state:
kubectl --context k3s-prod -n gitea-actions scale deploy/act-runner --replicas=0
kubectl --context k3s-prod -n gitea-actions delete pvc act-runner-data
kubectl --context k3s-prod -n gitea-actions scale deploy/act-runner --replicas=1
(The PVC holds only runner registration state -- no build artifacts, nothing durable.
Deleting it is the supported way to re-register. Flux recreates it from the manifest.)
Then confirm it actually registered -- do not assume:
curl -s -u '<user>' https://gitea.home.keithjackson.dev/api/v1/admin/actions/runners | jq
A runner must appear with status "online" and the labels ubuntu-latest / homelab.
NOTE