Compare commits

...
Author SHA1 Message Date
jakman85andClaude Opus 5 72a46ad82d chore(log): the handoff log was serialising every parallel branch
gate / gate (pull_request) Successful in 1m35s
gate / conformance (pull_request) Has been skipped
DECISIONS-LOG.md was append-only in ONE file and every agent appended to its
tail, so any two concurrent branches collided on the same last hunk. Measured
against the live Gitea today: four of the five open PRs were mergeable=false,
and `git merge-tree` named DECISIONS-LOG.md as the ONLY conflicting path in
all four.

    #22 fix/crafty-oauth2-proxy-signin       CONFLICT: DECISIONS-LOG.md
    #20 feat/conformance-job-template        CONFLICT: DECISIONS-LOG.md
    #19 feat/conformance-trigger-kubeconfig  CONFLICT: DECISIONS-LOG.md
    #18 feat/conformance-runner-image        CONFLICT: DECISIONS-LOG.md
    #17 feat/ci-is-the-approver              clean — it wrote no handoff

Nothing was wrong with any of those changes. The log's storage format
serialised work that was otherwise parallel, and the cost grew with every
concurrent branch (N branches -> N-1 rebases, each invalidating the gate run
before it). The agents work in isolated worktrees precisely so they do not
block each other; this one file undid that.

So: stop sharing a file. Handoffs are now one-per-file under decisions/,
named YYYY-MM-DD-<slug>.md. Disjoint paths have nothing to merge. There is
deliberately no index file (a shared mutable file is the thing that just
failed) and no counter (two agents would race for the same number) — the
directory listing is the index.

DECISIONS-LOG.md is frozen as the pre-2026-09-29 archive, its SHA-256 pinned
in decisions/ARCHIVE.sha256, and a new gate stage rejects any change to it.
That is a guard and not a paragraph because the old habit is written into the
runbooks and into every agent that has read the log, and getting it wrong does
not fail: it produces a clean-looking commit that re-creates the conflict on
OTHER branches, later, misattributed.

  ci/decisions-log.sh        new stage: archive pinned; every entry named with
                             a real date, carrying date/branch/title frontmatter
                             that agrees with its filename, and all three of
                             CLAUDE.md §7's sections with a real body under each.
                             Exit 0 or 1 only — no skip path by construction, so
                             no fail-open surface on a guard that closes one.
  ci/gate.sh                 runs it; fail and (impossible) skip both BLOCK.
                             No REQUIRE_DECISIONS_LOG knob: a guard with an off
                             switch is not a guard.
  ci/tests/decisions-log.test.sh   22 assertions, incl. the one that matters —
                             an append to the archive is rejected, and the
                             message names the file to write instead.
  ci/tests/gate-decision.test.sh   +3 assertions for the new wiring, AND a
                             pre-existing bug: the stub list never included
                             gitea-settings, so run_stage saw exit 127 and
                             recorded a FAILED stage in every case. 15 of 47
                             assertions failed on main for that reason alone —
                             the suite was measuring its own missing stub.
                             Now 50/50.
  CLAUDE.md                  §7 rewritten; §2b widened to the new path.
  ci/Makefile, ci/README.md, .gitea/workflows/README.md — stage named where
                             the other stages are named.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 22:35:55 -05:00
jakman85 cadf64a514 Merge pull request 'docs(ci): the wiring comment named a secret Gitea cannot accept' (#21) from fix/gitea-settings-comment-names-impossible-secret into main
gate / gate (push) Successful in 1m36s
gate / conformance (push) Has been skipped
Reviewed-on: #21
Reviewed-by: Keith Jackson <[email protected]>
2026-09-30 03:22:06 +00:00
jakman85 cfbc596b8b Merge pull request 'docs(crafty): Bedrock on 19132 RESOLVED — a player has joined; the wall was an empty allowlist' (#23) from fix/crafty-bedrock-19132 into main
gate / gate (push) Successful in 1m34s
gate / conformance (push) Has been skipped
Reviewed-on: #23
Reviewed-by: Keith Jackson <[email protected]>
2026-09-30 03:19:04 +00:00
jakman85andClaude Opus 5 c59a4b76bb docs(crafty): a player has joined -- the allowlist chain is closed, recommendation now PROVEN
gate / gate (pull_request) Successful in 1m35s
gate / conformance (pull_request) Has been skipped
Operator added one gamertag and the Bedrock client connected and joined. The
three-pass investigation closes with a working server, no manifest change, no
transport change, no network change.

Moved out of UNVERIFIED:

* A successful join -- OPERATOR-REPORTED (I did not watch it; the distinction is
  kept deliberately). Corroborated directly on disk, which is stronger than the
  report alone: allowlist.json went [] -> one entry, 75 bytes, and that entry's
  xuid is POPULATED (mtime 22:03:17). audit.log shows the operator wrote only the
  name (Edited file .../allowlist.json, 22:02:13). Vendor doc: xuid "will be
  populated when someone with a matching name connects". So the server wrote the
  XUID on connect, ~64s after the edit -- server-authored evidence of an
  XBL-authenticated join.
* The reasoned mapping of "You are not invited" to the empty allowlist is
  CONFIRMED: one gamertag added, one successful join. allow-list=true with
  allowlist.json == [] was the wall, and nothing else was.
* transport=raknet survives a restart: the fix involved one (pid 47006 -> 47875);
  afterwards server.properties mtime is still 21:33:31 with transport=raknet, UDP
  4ABC bound, still NO TCP 4ABC, and a fresh pong from 192.168.2.22:19132 at
  22:06.
* The operator's actual route was file-edit + restart (route 2), not the console
  `allowlist add` I listed first. Both proven; console still preferred (no restart).

README: join-gate section marked RESOLVED; decision heading and option 1 promoted
from provisional to PROVEN; Bedrock human-test 4 rewritten as a 5-step verified
connect procedure, each step with its distinct failure signature, stating plainly
that the TRANSPORT TYPE ERROR banner and the "too few fields" warning BOTH appear
on a fully working server -- that pair misled two consecutive sessions.

NOT weakened, and sharpened where possible:

* auto_start=0 / crash_detection=0 STILL 0, re-measured at 22:06 after the join
  and after the operator's 22:02:28 "modified the server" action, which did not
  change them. The failure mode is now worse, not better: "worked, then vanished".
  Merging fix/crafty-oauth2-proxy-signin is such a restart.
* Silent-reset now precisely characterised: a RESTART preserves both files
  (observed) but leaves the server down (auto_start=0); a server RECREATE or
  reinstall silently resets server.properties to transport=nethernet AND
  allowlist.json to [].
* L2/L3 conformance still uncertified; multi-player SNAT still untested (exactly
  one player has joined -- that is precisely the evidence that does not settle it);
  BDS console output still never seen (log_path is '').

No manifest touched, in any of the three passes. Read-only throughout.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 22:08:54 -05:00
jakman85andClaude Opus 5 cd2617e5b3 docs(crafty): "You are not invited" is an empty allowlist -- and it proves RakNet works
gate / gate (pull_request) Successful in 1m36s
gate / conformance (pull_request) Has been skipped
New operator observation: a Bedrock client got "Unable to connect / You are not
invited to play on this server". That was put to me as evidence that NetherNet
signalling is working, that the refusal may have come from Microsoft rather than
our server, and that BDS players therefore never traverse crafty-l4-udp.
Measured read-only at 21:56, all three are wrong.

  allow-list=true            (server.properties line 37)
  allowlist.json == []       (3 bytes, mtime 20:51, never edited since create)
  online-mode=true
  transport=raknet           (line 71; file mtime STILL 21:33:31)
  UDP 19132 bound            (/proc/net/udp 00000000:4ABC, pid 47006)
  TCP 19132 ABSENT           (no 4ABC in /proc/net/tcp or tcp6)
  bedrock_server pid 47006, same process since 21:34, never restarted

1. transport is still raknet -- nobody reverted it, the binary did not rewrite it
   (mtime unchanged, audit.log records no later edit, same pid). NetherNet is NOT
   in use and cannot be what served this client.
2. NetherNet could not have served it in principle: it signals over TCP on
   server-port, and there is no TCP 19132 listener in the pod at all.
3. So the refusal came from OUR server, over RakNet, over 19132/udp, through the
   VIP. The 19132-19134/UDP exposure is the opposite of inert -- it carried both
   the earlier pong and this refusal.
4. Cause is certain and trivial: allow-list=true with an empty allowlist.json
   refuses every player. The shipped how_to says so, and it keys on Xbox Gamertag.

The requested log correlation cannot be done from disk: K3s-Bedrock's log_path in
crafty.sqlite is the empty string and Dedicated_Server.txt is 0 bytes -- Crafty
persists no BDS stdout at all, the console is panel-UI-only. commander.log
21:39-21:56 holds only homelab crash-detection and raknet_ping lines. That silence
is evidence about Crafty's logging, not about the refusal's origin.

Adds a README section on the join authorization model: the measured table, the
proof the refusal is ours, the "console is not on disk" caveat, gamertag/XUID
identity, and three operator routes (console `allowlist add`, edit-then-`allowlist
reload`, or allow-list=false -- discouraged, since the game port is deliberately
un-gated so the allowlist is the only join control). Notes allowlist.json is PVC
state that resets to [] on a server recreate. Options table and human test 4 now
check the allowlist before blaming the network.

Ranking unchanged at the top: keep transport=raknet, add gamertags. Still zero
manifest change. NetherNet would ADD work (TCP 19132 in crafty-l4-tcp and in
crafty-allow-ingress, plus server-udp-ports) and admit not one extra player.

Read-only throughout: allowlist.json, server.properties and the DB were read; no
console command issued, no kubectl mutation, no flux reconcile.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 21:59:15 -05:00
jakman85andClaude Opus 5 8697702d51 docs(decisions-log): record the gate result -- rc 3, and why CI's green is not rc 0
The handoff entry asserted conclusions but did not carry its own gate numbers.
Adds them, per CLAUDE.md section 7:

  lint / l0 / render-golden / l1 / spec-immutability : green
  gitea-settings  : SKIPPED (SETTINGS_API_TOKEN unset)
  conformance L2/L3: SKIPPED (CLUSTER_AVAILABLE != true)
  => exit 3, UNDETERMINED. Not a PASS.

Also records two things future sessions will otherwise re-learn the hard way:

1. spec-immutability skipped SPURIOUSLY from the worktree -- dev.sh looks for the
   spec repo at $REPO_ROOT/../home-network-k3s, which from a worktree is the wrong
   directory; it exists beside the MAIN checkout. Mounting it RO through the ENGINE
   shim turned the one tolerated skip into a real green, so R-TEST-06 is genuinely
   verified. The CLAUDE.md section 3 shim should mount two paths, not one.

2. CI's green tick does NOT mean rc 0. gate/conformance is `if: false` and reports
   success with "Has been skipped"; gate/gate runs
   `./ci/gate.sh || [ "$?" -eq 3 ]`, so CI tolerates exit 3 by declared design and
   would report the same green for the rc 3 above.

Branch protection and the runner were measured against the API, not assumed:
protected=true, user_can_push=false, required_approvals=1, context
'gate / gate (pull_request)', runner incluster-dind online. PR #23 left unmerged.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 21:53:56 -05:00
jakman85andClaude Opus 5 d29a73cad8 docs(crafty): 19132/udp IS bound and answering -- the ERROR banner was advisory
gate / gate (pull_request) Successful in 1m35s
gate / conformance (pull_request) Has been skipped
The prior entry's operator fix (transport=raknet) was labelled REASONED, NOT
OBSERVED. The operator applied it, BDS 1.26.52 printed
"================ TRANSPORT TYPE ERROR ===================
 NetherNet is the only supported transport type", and that was read as a refusal.

It was not a refusal. The process kept running and bound the RakNet sockets:
/proc/net/udp 00000000:4ABC and /proc/net/udp6 ...:4ABD, both owned by
bedrock_server pid 47006 (fd/25, fd/26). The earlier TCP bind on 4ABC is gone.
Crafty's raknet_ping warning changed kind at 21:34 -- last "Got timeout"
21:34:04,306, first "too few fields" 21:34:09,854 -- and per raknet_ping.py the
latter is reachable only after a 0x1C Unconnected Pong arrives with valid magic.

Proven end-to-end from off-cluster: a RakNet Unconnected Ping to the prod VIP
192.168.2.22:19132 returns a 139-byte 0x1c pong in ~12ms:
  MCPE;Dedicated Server;2193;1.26.52;0;10;...;Bedrock level;Survival;1;19132;19133;0;0;0;

The vendor's own shipped bedrock_server_how_to.html documents raknet as a
supported value ("the server listens directly on server-port"), and RakNet is
still compiled in (RakNetConnector, createUniqueRakPeer, RakNetSocket2.cpp).
"Only supported transport type" is a support statement, not a capability one.

Also explains why nethernet could never have worked here: it signals over TCP on
server-port, and this repo exposes no TCP 19132 -- crafty-l4-tcp is 25565-25567
and crafty-allow-ingress allows 19132 UDP only. packet-statistics.txt for that
run: Total Received: 0, 0B over 2293s.

Decision: keep transport=raknet, NO manifest change. The 19132-19134/UDP
exposure is correct and live, not inert. Geyser+Floodgate documented as the
reserve migration if RakNet is ever really removed (also zero manifest change).

Other corrections: the "too few fields" warning is the success signature, not a
failure; enable-lan-visibility upgraded to OBSERVED (19133 answers with the same
server_guid, so slot 1 is NOT free -- previous text said it was unreachable);
Game VIP corrected from 192.168.1.55 (the base/dev value) to the live prod
192.168.2.22; noted transport=raknet is PVC state that silently regresses to
nethernet on a server recreate. K3s-Bedrock auto_start=0/crash_detection=0
re-confirmed from crafty.sqlite and still outstanding.

Read-only throughout: no kubectl mutation, no flux reconcile, no write inside
the container. server.properties was read, never touched.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 21:46:57 -05:00
jakman85andClaude Opus 5 07f03536b3 docs(crafty): Bedrock 19132 refuses nothing -- BDS ships transport=nethernet
gate / gate (pull_request) Successful in 1m35s
gate / conformance (pull_request) Has been skipped
Reported as "users cannot connect to Bedrock on UDP 19132". Measured read-only
on prod; the platform delivered the packets and the server never opened the
socket.

Server `K3s-Bedrock` (c5837301-..., type=minecraft-bedrock, server_port=19132)
exists and `./bedrock_server` is alive, but /proc/net/udp and /proc/net/udp6 in
the crafty container hold NO socket on 19132 (0x4ABC) in either family -- port
19132 is bound as TCP (/proc/net/tcp6 ...:4ABC state 0A). Cause, from the
server's own server.properties: transport=nethernet, the value Microsoft's BDS
archive ships. Its inline docs: `Allowed values: "raknet" or "nethernet"`, and
against server-portv6, `This property is ignored when transport=nethernet`. NetherNet is the
WebRTC/Xbox-signalled transport and opens no RakNet UDP listener. Crafty does
not manage the property (no `transport` anywhere under /crafty/app/classes/) and
its only Bedrock probe is raknet_ping.py, hence commander.log spamming
"Got timeout while issuing bedrock ping to 127.0.0.1:19132" -- over loopback,
which indicts the server, not the network.

The path below it is sound and was measured as such: crafty-l4-udp live as
externalTrafficPolicy: Cluster with the VIP in status.loadBalancer, a non-empty
EndpointSlice, `cilium-dbg service list` showing 192.168.2.22:19132/UDP =>
10.0.0.130:19132/UDP (active) on BOTH announcing nodes, the datapath policy map
carrying `Allow Ingress reserved:world 19132/UDP` with 26 packets / 27660 bytes
already counted through it, pihole answering crafty-mc -> 192.168.2.22, and Flux
at main@sha1:20ea70d == the GitRepository artifact.

So this is not a platform or manifest defect. No manifest is touched: adding
ports or widening policy for this would be manufacturing work. The repo-side
defect is documentation -- the README told the operator that creating a server
on 19132 was sufficient.

README: add "Bedrock on UDP 19132 -- the transport trap" (the measurement chain,
and the raknet_ping warning as its signature) and "Bedrock realities that are NOT
platform defects" (no LAN/Friends discovery behind a VIP, consoles cannot enter
an address, enable-lan-visibility collides with slots 1-2, server-portv6 is not
exposed). Correct three stale claims: "the operator has not created any servers
yet" (two exist), the human test implying a started Bedrock server is reachable,
and the single-server Runtime state section (now a two-row table recording
K3s-Bedrock's auto_start=0 / crash_detection=0 -- the create-API trap, live
again, so it will not survive a pod restart).

Also measured, reported, and deliberately NOT changed: the shared VIP's two
leases sit on different nodes (crafty-l4-tcp on k3s-prod-3, crafty-l4-udp on
k3s-prod-1), so two nodes ARP-announce 192.168.2.22. Benign under `Cluster` --
both nodes carry the full active frontend set -- and the existing L0 check
l2-announced-vips-serve-from-any-node already forbids the `Local` that would make
it fatal. A "shared VIPs must co-locate" check would assert runtime lease
placement a static check cannot see.

Read-only throughout: no apply/edit/patch, no flux reconcile, no write inside
the container. Operator actions (set transport=raknet and verify the UDP bind;
PATCH auto_start; tell players to Add Server by hand) are in the DECISIONS-LOG
entry, along with what is UNVERIFIED -- chiefly that `raknet` fixes it is
reasoned from the shipped docs, not observed.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 21:20:14 -05:00
jakman85andClaude Opus 5 e993d0f7d1 docs(ci): the wiring comment named a secret Gitea cannot accept
gate / gate (pull_request) Successful in 1m36s
gate / conformance (pull_request) Has been skipped
ci/gitea-settings.sh told the reader to "create a Gitea repo secret
GITEA_SETTINGS_TOKEN". That instruction is IMPOSSIBLE to follow: Gitea reserves the
GITEA_ prefix for secret names and rejects the PUT with HTTP 400 "invalid secret name".

Same defect as the one fixed in PR #15 and in the README in PR #19 -- this was the third
file carrying it, found by an agent reading the file rather than trusting my description
of it. A comment that instructs an impossible action is not cosmetic: this exact wrong
name is why the drift detector sat inert while a token existed and no secret did.

Now names SETTINGS_API_TOKEN, and states explicitly that GITEA_SETTINGS_TOKEN remains
honoured ONLY as a local-run env var -- it can never be the CI wiring, because the secret
that would feed it cannot exist.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 19:30:35 -05:00
jakman85 20ea70d221 Merge pull request 'fix(sec): the R-SEC-02 admission policy was never applied' (#16) from fix/kyverno-policy-was-never-applied into main
gate / gate (push) Successful in 1m36s
gate / conformance (push) Has been skipped
Reviewed-on: #16
Reviewed-by: Keith Jackson <[email protected]>
2026-09-29 18:28:54 +00:00
jakman85andClaude Opus 5 8c7a770de2 fix(sec): the R-SEC-02 admission policy was never applied -- and nothing could tell
gate / gate (pull_request) Successful in 1m37s
gate / conformance (pull_request) Has been skipped
PR #13 added conformance-trigger-job-shape.yaml to platform/kyverno/admission/
kustomization.yaml. That file applies NOTHING. Its own sibling says so: "kept as
reference; THIS file is the buildable render root" -- and that render root
(platform/kyverno/kustomization.yaml) ships only the namespace and the HelmRelease,
noting that ClusterPolicies are applied a tier later, by platform-config, because they
need the Kyverno CRDs first.

So three lists exist and only platform-config/kustomization.yaml applies anything. The
policy went into the wrong one and EVERYTHING WENT GREEN: the file parsed, kyverno test
exercised it by path and passed, the merge gate passed, and Flux reported `platform`
Ready at the new revision. Measured on prod afterwards:

    kubectl get clusterpolicy conformance-trigger-job-shape
    Error from server (NotFound)

while ns conformance, the conformance-trigger ServiceAccount, its Role and its
RoleBinding were all live. That is exactly the state that identity's own RBAC header
says must never exist -- the trigger identity present with nothing bounding it. Blast
radius was small only because no trigger kubeconfig exists yet, which is luck, not design.

Wires the policy into platform-config (verified: the render goes from 7 ClusterPolicies to
8 and contains conformance-trigger-job-shape).

CLOSES THE CLASS, which matters more than the instance. Adds the L0 check
admission-policies-actually-applied: every ClusterPolicy file under platform/kyverno/ is
either referenced by platform-config/kustomization.yaml or named in a CI_ONLY list with a
reason -- and CI_ONLY may not name a policy that IS applied, so the exemption cannot rot in
either direction. Declared scope whole-repo deliberately: a policy wired nowhere touches no
requirement the diff attributes, so a path-scoped run would have skipped it on the exact
commit that introduced the defect.

Proven both ways, not assumed:
  with the fix reverted -> FAIL, naming conformance-trigger-job-shape as never applied
  with the fix          -> PASS, 9 policy files, 8 applied, 1 CI-only with a reason

`kyverno test` proves a policy is CORRECT. This proves it is DEPLOYED. Nothing did before.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 13:23:09 -05:00
jakman85 1c5974013e Merge pull request 'docs(decisions-log): end-of-session PM handoff' (#14) from docs/handoff-2026-09-29 into main
gate / gate (push) Successful in 1m35s
gate / conformance (push) Has been skipped
Reviewed-on: #14
Reviewed-by: Keith Jackson <[email protected]>
2026-09-29 18:16:22 +00:00
jakman85 4791d4a43a Merge pull request 'feat(sec): constrain the conformance CI trigger at admission' (#13) from feat/conformance-trigger-admission-boundary into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
Reviewed-on: #13
Reviewed-by: Keith Jackson <[email protected]>
2026-09-29 18:16:00 +00:00
jakman85 dfd608ac32 Merge pull request 'fix(ci): CLUSTER_AVAILABLE=true would have faked a green' (#12) from fix/cluster-available-would-fake-a-green into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
Reviewed-on: #12
Reviewed-by: Keith Jackson <[email protected]>
2026-09-29 18:14:55 +00:00
jakman85 1d2d9d6549 Merge pull request 'fix(ci): the settings token could never have been stored' (#15) from fix/secret-name-and-require-gitea-settings into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
Reviewed-on: #15
Reviewed-by: Keith Jackson <[email protected]>
2026-09-29 18:14:30 +00:00
jakman85andClaude Opus 5 4b31f97486 docs(decisions-log): addendum — detector armed, and why an agent cannot merge here
gate / gate (pull_request) Successful in 1m34s
gate / conformance (pull_request) Has been skipped
Records two findings from the unattended session.

The drift detector was INERT: set-ci-token.sh stored the secret under a name Gitea
reserves (GITEA_ prefix, HTTP 400 "invalid secret name"), sent the PUT to /dev/null,
and printed success anyway. A token existed; no secret did. Enabling
REQUIRE_GITEA_SETTINGS on top of that would have blocked every PR including its own
fix. PR #15 fixes it and its CI is green WITH require active, which is the only
proof that counts.

And: merging as gitea-admin returns HTTP 405 "Does not have enough approvals".
required_approvals:1 + block_admin_merge_override:true + an author who cannot
self-approve means "merge on green CI" is not executable by an agent. Recorded as
the control working rather than a fault, and deliberately not circumvented --
self-approval would hollow out the boundary just built. The choice of what to do
about it is the operator s, and whatever is chosen must also change the declaration
in expected-branch-protection.json or the detector will correctly fail on the drift.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 12:12:33 -05:00
jakman85andClaude Opus 5 50a48e6f07 fix(ci): the settings token could NEVER have been stored -- Gitea reserves the GITEA_ prefix
gate / gate (pull_request) Successful in 1m36s
gate / conformance (pull_request) Has been skipped
The drift detector merged in #11 was inert and would have stayed inert silently.

WHAT HAPPENED. infra/gitea/set-ci-token.sh minted a token and PUT it to the repo Actions
secret GITEA_SETTINGS_TOKEN. Gitea RESERVES the `GITEA_` prefix for secret names -- exactly
as GitHub reserves `GITHUB_` -- and rejects that PUT with:

    HTTP 400  {"message":"invalid secret name"}

The script sent the PUT to /dev/null and printed "==> stored." unconditionally, and
`curl -sS` exits 0 on an HTTP error, so it reported success for a store that never happened.
Measured afterwards: the token existed in the user's token list, and the repo's secret list
was EMPTY. ci/gitea-settings.sh would have skipped on every CI run forever, and with
REQUIRE_GITEA_SETTINGS=true it would instead have BLOCKED EVERY PR -- including the one
fixing it.

Three changes:
  * the secret is SETTINGS_API_TOKEN; ci/gitea-settings.sh reads it, keeping
    GITEA_SETTINGS_TOKEN as a local-run fallback;
  * set-ci-token.sh checks the PUT's HTTP status AND reads the secret back, refusing to claim
    success otherwise. That missing verification is the actual defect -- the wrong name was
    just what it hid;
  * gate.yml maps the secret and ENABLES REQUIRE_GITEA_SETTINGS.

WHY ENABLING IT HERE IS SAFE: this PR's own CI run is the proof. If the stage were still
skipping, REQUIRE turns that into a failure and this PR goes RED rather than merging. Green
means the token resolved and live Gitea matched infra/gitea/expected-branch-protection.json.

Verified locally before pushing:
  REQUIRE=true, token present -> MERGE GATE: PASS
  REQUIRE=true, no token      -> gitea-settings SKIPPED -> BLOCKED (not a silent skip)

The secret is already stored and read back as present, so CI has what it needs.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 12:08:12 -05:00
jakman85andClaude Opus 5 911ead5fbd docs(decisions-log): end-of-session PM handoff — six PRs open, and what a new machine needs
gate / gate (pull_request) Successful in 1m33s
gate / conformance (pull_request) Has been skipped
Written for a different session on a different computer. Lists the three open gated
PRs with their risk, the one command that must follow #11 (the drift detector skips
until its token exists, and a skipped detector detects nothing), and the three
credentials that are deliberately NOT in git and without which the toolbox and
cluster are unreachable: the age private key, prod.kubeconfig, and a Gitea
credential.

Records the measured platform state while telling the reader to re-measure it rather
than trust this log -- which is the whole point of CLAUDE.md section 2b, and which
this log earned by being confidently wrong for a month.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 03:28:33 -05:00
jakman85andClaude Opus 5 8eb5baffa4 feat(sec): constrain the conformance CI trigger at admission (R-SEC-02)
gate / gate (pull_request) Successful in 1m38s
gate / conformance (pull_request) Has been skipped
Implements the credential model the operator chose for wiring L2/L3 into CI: a narrow
trigger SA bounded by a fail-closed Kyverno admission policy, rather than the
zero-credential GitOps-native alternative (plan section 4, primary model).

THE ESCALATION THIS CLOSES. Kubernetes places NO `use` check on a Job's
serviceAccountName. Any principal that can `create jobs` in ns conformance can create a Job
running as conformance-runner -- nodes/patch, pods/exec, namespaces/patch -- with an
arbitrary command. A CI token with plain `create jobs` IS that privileged identity no matter
how narrow its own Role looks. RBAC cannot fix this; only constraining the SHAPE of what the
trigger may create can.

Adds:
  * SA/Role/RoleBinding conformance-trigger -- create+get+list+watch jobs, get pods and
    pods/log, nothing else. Namespaced. No delete (a trigger that can delete can hide a
    failed run), no exec, no secrets. Results come back as a tarball on stdout read via
    pods/log, so no execution channel into the runner pod is ever granted.
  * ClusterPolicy conformance-trigger-job-shape -- Enforce, failurePolicy: Fail. It does NOT
    use ${KYVERNO_VALIDATION_ACTION}: a security boundary an overlay can downgrade to Audit
    is not a boundary. It constrains ONLY conformance-trigger, so a mistake here cannot wedge
    the rest of the cluster.
  * Flux Kustomization `conformance` -> ./conformance/incluster, wait:false and non-gating
    like apps-vms, so a test identity can never stall the platform or app tiers.
  * A kyverno test with seven cases, six of them real escalation paths.

VERIFIED by `kyverno apply` (pass: 10, fail: 6, error: 0), not assumed:
  evil-command    DENIED at /spec/template/spec/containers/0/command/   <- THE attack
  evil-image      DENIED at .../image/
  tag-not-digest  DENIED at .../image/          (mutable tag is republishable)
  host-namespaces DENIED at /spec/template/spec/hostPID/
  extra-volume    DENIED by the volumes foreach
  injected-args   DENIED by the args foreach
  good-run        ADMITTED

ALSO FIXES A FAIL-OPEN IN THE GATE ITSELF. `kyverno test` does NOT fail when a policy cannot
LOAD: every row becomes "Skip / Invalid Policy" and it still prints "N tests passed" and
exits 0. Measured while writing this policy -- a schema error (subjects in a separate
match.all entry from resources) produced exactly that, and ci/l0.sh would have certified a
security policy that does not load as green. This is not specific to this policy: a typo in
restrict-host-namespaces or secret-provenance would silently disable it with the gate still
green. ci/l0.sh now scans for "Invalid Policy" and fails. Proven: broken policy -> l0 rc=1,
restored -> rc=0.

IMAGE PINNING, HONEST LIMIT: the plan specifies pinning the exact runner image digest. That
image does not exist yet, so this enforces the enforceable half -- the image must be under
the conformance repository path AND digest-pinned, never a mutable tag. Residual surface:
anyone who can PUSH to that path could publish a different digest. Tighten to the literal
digest when the image is built; it is a one-line change to the pattern, reviewed in a PR,
which is where a change to a security boundary belongs.

NOT YET WIRED, deliberately: no runner image, no trigger kubeconfig, no CLUSTER_AVAILABLE.
The identities here are inert until those exist, and PR #12 makes CLUSTER_AVAILABLE=true
block rather than fake a green in the meantime.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 03:22:41 -05:00
jakman85 4634b44cc7 Merge pull request 'feat(ci): detect Gitea settings drift' (#11) from feat/gitea-settings-drift-check into main
gate / gate (push) Successful in 1m34s
gate / conformance (push) Has been skipped
Reviewed-on: #11
Reviewed-by: Keith Jackson <[email protected]>
2026-09-29 07:59:29 +00:00
jakman85andClaude Opus 5 6586919132 fix(ci): CLUSTER_AVAILABLE=true would have FAKED A GREEN over 54 skipped tests
gate / gate (pull_request) Successful in 1m36s
gate / conformance (pull_request) Has been skipped
Wiring L2/L3 into CI was a tracked next step, described in the plan as needing only
vars.CLUSTER_AVAILABLE. Attempting it surfaced that the variable does not enable L2/L3 at
all -- it only changes which way the stage fails, and before this commit one of those ways
was a false pass.

MEASURED, before the guard:
  CLUSTER_AVAILABLE=true ./ci/conformance.sh   ->  rc=0
  "[junit] wrote skipped JUnit for 54 scaffolded test(s)"
  "[ ok ] L2/L3 conformance passed for touched subset"

run.sh exits 0 in scaffold mode when it cannot reach a cluster, marking every test
<skipped>. That is correct and honest FOR run.sh. It is catastrophic in ci/conformance.sh,
which reaches that code only because the environment DECLARED a cluster available -- so it
reported a pass over 54 tests that never executed. One Gitea variable away from a fake green
on a gate that is now an ENFORCING required status check, and nothing caught it because
nobody had ever set the variable.

The CI runner holds no cluster credential by design (automountServiceAccountToken: false),
so the local run this stage attempts can never succeed. The in-cluster trigger path that
would make L2/L3 actually run is still unimplemented, and is blocked on a privilege-
escalation design decision that is the operator's to make (plan section 4: any principal
that can `create jobs` in ns conformance can run arbitrary code as the privileged
conformance-runner SA, because Kubernetes places no `use` check on serviceAccountName).

The stage now BLOCKS (exit 1) when a cluster is declared but every test skipped, which is
what the merge-gate decision matrix already said should happen ("conformance skipped while
CLUSTER_AVAILABLE=true -> BLOCK"). Verified: rc=1, naming the count and the reason.

The plan document claimed CLUSTER_AVAILABLE was "the only remaining gate". Corrected in
place with the measurement and an explicit do-not-set-this-yet.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 02:58:26 -05:00
jakman85andClaude Opus 5 e6f6e4febb feat(ci): detect Gitea settings drift -- the blind spot that hid a month of outage
gate / gate (pull_request) Successful in 1m34s
gate / conformance (pull_request) Has been skipped
Every check in this repo reads the repository, the rendered manifests, or the cluster.
NONE could see Gitea's own database, and that is where branch protection and runner
registration live. Three failures came out of that blind spot on 2026-09-29, all the same
shape -- a setting that READ as enforced while not being, with nothing able to notice:

  * main had NO branch protection for ~a month while DECISIONS-LOG recorded it as a
    required status check;
  * ZERO Actions runners were registered, so PR #1 -- the first PR this repo ever had --
    sat pending for 30.6 minutes with nothing to execute it;
  * after enable_status_check and required_approvals were turned on they STILL did not
    bite, because block_admin_merge_override was false: PRs #5-#9 merged with zero
    approvals recorded against required_approvals: 1.

infra/gitea/*.sh ASSERT the desired state. Nothing DETECTED drift from it. This is the
detector.

ci/gitea-settings.sh runs as a merge-gate stage: it diffs the live rule at
/branch_protections/main against infra/gitea/expected-branch-protection.json and asserts at
least one runner is online. It reads the RULE object, not /branches/main -- the branch view
does not expose block_admin_merge_override and showed all-green while the rule was
overridable by the only account that merges.

Verified by test, not assumed:
  no token            -> exit 3, skips loudly, does NOT block
  live == declaration -> exit 0
  injected drift      -> exit 1, blocks, naming both the drifted key and the runner count

The skip is DELIBERATELY not in gate.sh's strict-skips loop. Making an absent credential
block every PR is precisely the lock-yourself-out failure this repo keeps walking into, so
a skip denies a PASS (exit 3) and says loudly what is unverified. REQUIRE_GITEA_SETTINGS=true
upgrades it to a block once the token is wired.

gate.yml's `|| [ "$?" -eq 3 ]` tolerance now covers a second stage. That widening is
declared in the file rather than left to be discovered, and it narrows back the moment
REQUIRE_GITEA_SETTINGS is set.

infra/gitea/set-ci-token.sh mints the token and stores it as the repo Actions secret without
ever printing the value.

UNTIL THAT TOKEN EXISTS THIS STAGE SKIPS, AND A SKIPPED DETECTOR DETECTS NOTHING.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 02:52:28 -05:00
jakman85 e8e6b9c681 Merge pull request 'fix(gitea): admins could override the gate' (#10) from fix/gitea-admins-could-override-the-gate into main
gate / gate (push) Successful in 1m34s
gate / conformance (push) Has been skipped
Reviewed-on: #10
Reviewed-by: Keith Jackson <[email protected]>
2026-09-29 07:43:52 +00:00
jakman85andClaude Opus 5 fcc66c7858 fix(gitea): admins could override the gate -- the enforcement was not enforcing
gate / gate (pull_request) Successful in 1m34s
gate / conformance (pull_request) Has been skipped
enable_status_check and required_approvals were set earlier today and recorded as done.
They did not bite. block_admin_merge_override was false, and Gitea lets repo admins
override merge requirements without it. jakman85 holds owner, so both requirements were
bypassable by exactly the account that performs every merge.

The evidence was already on the record: PRs #5-#9 all merged while required_approvals: 1
was in force, with ZERO approvals recorded on any of them.

Two process failures, both the same shape as the defect this whole session started with:

1. Verified with the wrong view. /branches/main reports protected, enable_status_check and
   required_approvals but NOT block_admin_merge_override -- it showed everything green while
   the rule was overridable. /branch_protections/main is authoritative.
2. Probed for a field name assumed to exist. This Gitea (1.24.3) has no `apply_to_admins`;
   asking jq for it returns null, which reads like "set to false" but means "key absent", so
   a PATCH that silently did nothing produced a convincing before/after. jq returning null
   is not a value -- it may be an absence. Enumerate with to_entries.

require-gate.sh now sets block_admin_merge_override: true; block-admin-override.sh fixes it
in isolation. Both carry the field-name warning.

This PR is the test. If it merges WITHOUT an approval, the fix did not work.

Also records the governance question: with only jakman85 (human) and gitea-admin (agent),
a human-authored PR now needs the automation account to approve it. That is ceremony, not
review, and the choice between keeping it, dropping approvals in favour of the status
check, or adding a second human is the operator's to make.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 02:36:38 -05:00
jakman85 b9e8a4f9e8 Merge pull request 'docs(ci): close out the hardening; correct a claim that shipped wrong' (#9) from docs/close-out-ci-hardening into main
gate / gate (push) Successful in 1m34s
gate / conformance (push) Has been skipped
Reviewed-on: #9
2026-09-29 07:32:52 +00:00
jakman85andClaude Opus 5 b2d9883958 docs(ci): close out the hardening -- land the scripts, correct a claim that shipped wrong
gate / gate (pull_request) Successful in 2m2s
gate / conformance (pull_request) Has been skipped
The branch-protection + PR-gate + sidecar sequence is complete and verified. This records
the finished state and fixes what the sequence itself proved wrong.

CORRECTS A FALSE CLAIM I SHIPPED. PR #7 said the sidecar conversion would remove the
one-per-start dockerd line "http: TLS handshake error from 127.0.0.1: EOF". It does NOT.
Exactly one still appears, immediately after "API listen on [::]:2376" -- it is now the
startupProbe's FIRST attempt, landing microseconds after dockerd begins listening and cut
off by timeoutSeconds: 1; the next poll succeeds. Cosmetic, and deliberately not chased:
raising the timeout to hide one benign startup line would slow every subsequent poll. The
manifest comment now states this instead of the wrong prediction.

Lands infra/gitea/require-gate.sh and verify-then-enforce.sh, which drove the rollout from
a session temp dir. verify-then-enforce.sh merges nothing -- merging is a human's reviewed
act -- but forces the reconcile and ABORTS unless the pod is healthy, a runner is online,
and a real workflow goes green on it, before enabling enforcement and probing the push.
Re-run it after any future act-runner change.

Trims the now-obsolete feat/ rationale from claim-ship.sh: per-branch push triggers were
deleted, so no prefix produces a duplicate run any more.

CLAUDE.md's enforcement table flips from "weaker than this rule" to machine-enforced, with
all four rows marked verified and the push rejection noted as PROVOKED rather than assumed.
It also now states what a green PR does NOT prove: CLUSTER_AVAILABLE is unset, so L2/L3
never runs in CI and only the L0/L1 offline subset is covered.

infra/gitea/README.md marks the sequence done and names the root cause that remains
unfixed: Gitea repo settings live in its own database, invisible to Flux and unreadable by
the conformance gate, so NOTHING detects drift from what this directory declares. These
scripts assert desired state; they are not a detector. The check that reads the live API
and fails on divergence is the highest-value work left, and it is the difference between
this being fixed and this being unable to recur.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 02:28:55 -05:00
jakman85 00efd4b828 Merge pull request 'fix(ci): convert dind to a native sidecar; delete both wait loops' (#7) from fix/dind-native-sidecar into main
gate / gate (push) Successful in 1m35s
gate / conformance (push) Has been skipped
Reviewed-on: #7
2026-09-29 07:22:50 +00:00
jakman85 67a54a33a4 Merge pull request 'fix(ci): delete the per-branch push triggers instead of adding feat/**' (#6) from fix/gate-push-trigger-matches-feat-prefix into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
Reviewed-on: #6
2026-09-29 07:22:20 +00:00
jakman85 db1a623073 Merge pull request 'docs(runbooks): the PR flow, not git push origin main' (#5) from docs/runbooks-pr-flow into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
Reviewed-on: #5
2026-09-29 07:21:28 +00:00
jakman85 ce271225de Merge pull request 'docs: correct CI/platform documentation that was stale or wrong' (#8) from docs/ci-reality-vs-phase-0 into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
Reviewed-on: #8
2026-09-29 07:20:48 +00:00
jakman85andClaude Opus 5 e72050c690 docs: correct CI/platform documentation that was stale, wrong, or load-bearing
gate / gate (push) Successful in 1m37s
gate / gate (pull_request) Successful in 1m36s
gate / conformance (push) Has been skipped
gate / conformance (pull_request) Has been skipped
The 2026-09-29 session found the merge gate did not exist -- no branch protection, no
runner -- while the tree asserted both. This corrects the documents that carried, and in
several cases CAUSED, that belief.

CLAUDE.md gains section 2b: never infer platform state from this repository. It carries
the specific API and kubectl checks for protection, runners, CI status, workloads and
Flux revisions, and the rule that a green tick is not proof anything ran (a skipped Gitea
job reports success). This is the rule whose absence produced every other finding.

.gitea/workflows/README.md and merge-gate.yaml both said "there is no remote git host,
this file is dormant". Gitea had been live for weeks. Both were written as plans and never
revisited once the plan completed. The Transition section is now marked COMPLETE and says
what is genuinely still outstanding (CLUSTER_AVAILABLE unset, so L2/L3 never runs in CI).

DECISIONS-LOG 2026-08-31 is annotated IN PLACE. Anyone reading chronologically previously
hit "required status check; runner incluster-dind online" with nothing marking it false
until a month later. The correction now sits on the bullet itself.

ci/README.md, clusters/README.md and both conformance READMEs carry a status banner: they
speak as though no cluster exists, which makes a live L2/L3 coverage gap read as an
expected phase. That framing is why the gap went unexamined.

apps/wave-1/gitea-actions-runner-secret.sops.yaml: fixed a wrong path, replaced advice to
`sops -d` into a terminal (which CLAUDE.md forbids) with the rotation script, and recorded
that a fresh token alone does not re-register a runner holding a stale /data/.runner.

apps/wave-0/plex-claim-secret.sops.yaml: the header claimed the file ships the placeholder.
It has not since the claim -- it holds the spent token, which is inert and deliberately
left in place.

catalog/plex.yaml and the plex secret both asserted a "~4-minute TTL" as fact. It was
uncited, upstream documents no expiry, and it was the basis for a since-disproven claim
that the pipeline was too slow to fit inside it. Measured: ~2m45s.

Both .sops.yaml files were verified to still decrypt after the comment edits
(`sops -d > /dev/null`), and render==golden confirms the catalog comment does not
propagate into the rendered manifest.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 02:14:26 -05:00
jakman85andClaude Opus 5 761e25cc3f docs(runbooks): two more uncited "4-minute" claims the first pass missed
gate / gate (push) Successful in 1m34s
gate / gate (pull_request) Successful in 1m35s
gate / conformance (push) Has been skipped
gate / conformance (pull_request) Has been skipped
A systematic grep for the TTL assertion found it still in runbooks/README.md and in
runbooks/kustomization.yaml, outside the page being rewritten. Same correction: the
countdown on plex.tv/claim is the authority, not a number in our tree.

README.md also now records that the server is CLAIMED and points at claim-ship.sh with
the measured ~2m45s loop, rather than describing a push-based sequence that no longer
exists.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 01:57:47 -05:00
jakman85andClaude Opus 5 e7508931ca fix(ci): convert dind to a native sidecar; delete both hand-rolled wait loops
gate / gate (push) Successful in 1m34s
gate / gate (pull_request) Successful in 1m35s
gate / conformance (push) Has been skipped
gate / conformance (pull_request) Has been skipped
Executes the FOLLOW-UP that has been carried in this manifest since the readiness fix,
now that k3s is v1.36 and sidecar containers are GA.

dind moves into initContainers as the first entry with restartPolicy: Always -- the one
field that makes an initContainer a sidecar: it starts before the regular containers,
runs alongside them, and is terminated after them. kubelet therefore enforces the
ordering that the runner used to hand-roll, so the runner drops its command: override
and runs the image's own entrypoint.

The startupProbe execs a full TLS-verified `docker ... info`, NOT a tcpSocket probe. A
tcpSocket probe would re-open the exact gap the old Stage 1 existed to close: dockerd
writes the TLS certs and only THEN starts listening, so the port can accept a connection
while the certs are absent, and act_runner hard-exits on a missing
/certs/client/ca.pem. `docker info` proves both that the certs are usable and that the
daemon answers. periodSeconds 1 x failureThreshold 60 preserves the previous 60s cap.

It also removes the one-per-start dockerd log line
  http: TLS handshake error from 127.0.0.1: EOF
which was the old `nc -z` probe opening a TCP connection and closing it without a TLS
handshake -- an error we were generating ourselves and then reading in the logs.

SECONDARY WIN: deleting the shell deletes an envsubst hazard. apps-wave-1 runs postBuild
substitution over this ENTIRE manifest, comments included, and those loops depended on
bare dollar-names passing through untouched -- a braced or double-dollar spelling would
have silently unbounded the waits or failed the whole Kustomization. The only dollar left
in the file is the intended ${BASE_DOMAIN}.

LANDS ALONE, DELIBERATELY. This Deployment is the merge gate for the entire repository:
if the conversion is wrong, nothing can merge. Merge this BEFORE enabling the required
status check, and watch a real workflow run green on the converted pod before doing so --
otherwise a bad conversion locks the repo with no way to revert through the gate.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 01:54:47 -05:00
jakman85andClaude Opus 5 bf98a99af6 fix(ci): delete the per-branch push triggers instead of adding feat/** to them
gate / gate (pull_request) Successful in 1m34s
gate / conformance (pull_request) Has been skipped
The tracked next-step said "align gate.yml's push trigger with this repo's feat/**
prefix". Doing that literally would have made things worse, so this does the opposite.

act_runner runs at capacity 1. A branch push and its PR fire the SAME gate over the SAME
tree, and the redundant run SERIALISES AHEAD of the PR run -- doubling the wall clock on
the one result that gates the merge. Adding another prefix widens that, it does not fix
it. infra/plex/claim-ship.sh currently has to pick a feat/ branch name specifically to
DODGE this, during a loop that is racing a short-lived token.

The prefix list also silently failed to match this repo's own convention: branches here
are feat/**, which was absent, so the "fast feedback" never fired for the most common
prefix. Nobody noticed for a month because the only symptom of a wrong entry is a MISSING
run. Any hand-maintained prefix list has that failure mode. Deleting the list removes the
whole class rather than correcting one entry.

It was redundant anyway: CLAUDE.md already requires ./ci/gate.sh in the pinned toolbox
before pushing (74s locally), and the PR gate re-runs it authoritatively.

push: [main] is KEPT -- post-merge main must still be verified, and it is the only run for
which no PR context exists.

FOLLOW-UP once this and #5 are both merged: the feat/ rationale in
infra/plex/claim-ship.sh becomes obsolete and its comment should be trimmed. The script
keeps working either way.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 01:50:59 -05:00
jakman85andClaude Opus 5 8e017fad91 docs(runbooks): the PR flow, not git push origin main -- that instruction is now blocked
gate / gate (pull_request) Successful in 1m34s
gate / gate (push) Successful in 1m34s
gate / conformance (pull_request) Has been skipped
gate / conformance (push) Has been skipped
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks
instructed is REJECTED. The wording was merely stale before; it is now actively wrong,
and it is the line an operator reads mid-incident.

Every occurrence now describes the real path: push a branch, open a PR, merge once the
gate is green -- that merge is the deploy. 35 files shared one identical block;
flux-reconcile.md, README.md and plex-claim.md each said it their own way and were
corrected individually.

plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW
CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the
page only applies after a /config wipe.

Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate ->
merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose
under time pressure and two are SILENT when skipped: --with-source (without it the
Kustomization applies the previously-fetched revision), the mandatory rollout restart
(only the ciphertext changes, so the pod template is byte-identical and nothing restarts
itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside
gate.yml's push trigger fires a redundant run that serialises ahead of the PR run).

Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream
pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority.
Measured loop: ~2m45s.

DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false,
the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction
that a skipped Gitea job reports success rather than hanging.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 01:42:35 -05:00
gitea-admin 27d0ef3324 Merge pull request 'chore(plex): fresh plex.tv claim token' (#4) from feat/plex-claim-token-013124 into main
gate / gate (push) Successful in 1m33s
gate / conformance (push) Has been skipped
2026-09-29 06:33:04 +00:00
jakman85 bbd4f682a6 chore(plex): fresh plex.tv claim token
gate / gate (pull_request) Successful in 1m35s
gate / conformance (pull_request) Has been skipped
Injected inside the plex.tv claim window per runbooks/plex-claim.md.
2026-09-29 01:31:24 -05:00
jakman85 0391139240 Merge pull request 'feat(plex): wire PLEX_CLAIM from a SOPS Secret (placeholder value)' (#1) from feat/plex-claim-sops-secret into main
gate / gate (push) Successful in 1m34s
gate / conformance (push) Has been skipped
Reviewed-on: #1
2026-09-29 06:18:14 +00:00
jakman85 e9f441b5ce Merge pull request 'docs(gitea): a skipped job reports SUCCESS, not pending' (#3) from fix/conformance-context-is-fail-open into main
gate / gate (push) Successful in 1m34s
gate / conformance (push) Has been skipped
Reviewed-on: #3
2026-09-29 06:12:30 +00:00
jakman85andClaude Opus 5 234cc6a71a docs(gitea): a skipped job reports SUCCESS, not pending -- correct the deadlock claim
gate / gate (push) Successful in 1m33s
gate / gate (pull_request) Successful in 1m32s
gate / conformance (push) Has been skipped
gate / conformance (pull_request) Has been skipped
The runner came up on 2026-09-29 and the first CI jobs this repo has ever run went
green, which falsified a claim merged in #2.

MEASURED:
  gate / gate (pull_request)          success   1m58s (PR 1), 1m34s (PR 2)
  gate / gate (push)                  success   1m32s (PR 2 only)
  gate / conformance (pull_request)   success   "Has been skipped"

#2 said listing the conformance context as a required check would deadlock every PR,
because the job is `if: false` and never runs. That is wrong. Gitea reports a skipped
job as SUCCESS. Requiring it would not deadlock anything -- it would pass VACUOUSLY,
forever, while nothing ran.

The conclusion is unchanged (never require it) but the reason inverts, and the corrected
reason matters more: under HARD RULE 4 a permanently-green check that verifies nothing is
worse than a deadlock. A deadlock is loud. This is the exact fail-open shape the
2026-09-18 audit and this one both keep finding.

Also records that the (push) contexts only appear for branches matching gate.yml's push
trigger -- feature/**, fix/**, docs/**, chore/** -- and NOT feat/**, which is why PR 1's
feat/ branch produced only pull_request contexts. Require the pull_request context alone,
or PRs from non-matching prefixes never satisfy the rule.

CLAUDE.md's enforcement table is updated: the runner half is genuinely fixed and observed
green, the branch-protection half is not. Adds the warning that a green tick is not by
itself evidence that anything ran.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 01:07:13 -05:00
jakman85 69a0d1a1c7 Merge pull request 'docs: delivery contract for agents + record that the merge gate does not exist' (#2) from docs/ci-delivery-flow-for-agents into main
gate / gate (push) Successful in 1m34s
gate / conformance (push) Has been skipped
Reviewed-on: #2
2026-09-29 05:50:31 +00:00
jakman85andClaude Opus 5 4fd75ccdaa feat(plex): wire PLEX_CLAIM from a SOPS Secret so the server can be claimed
gate / gate (pull_request) Successful in 1m58s
gate / conformance (pull_request) Has been skipped
Plex cannot create libraries over the already-mounted 8.33T NFS bulk share while
the server is UNCLAIMED. catalog/plex.yaml carried a long argument for leaving
PLEX_CLAIM unwired; the operator has decided to override it. This implements the
wiring in the way that respects the half of that argument which is still true.

WHAT WAS TRUE AND STILL IS: a https://plex.tv/claim token has a ~4-minute TTL,
far shorter than commit -> push -> Flux reconcile -> pod-roll on its own timers.
So the committed value is a documented PLACEHOLDER and the live token is injected
by the operator inside that window, with the reconcile FORCED rather than waited
for. runbooks/plex-claim.md is that timed procedure.

WHAT WAS NOT TRUE: the old comment claimed "a placeholder/absent claim Secret
would crashloop the pod". Half of that is wrong, and the difference is the whole
design. Established from plexinc/pms-docker's own first-run script (branch
`master`, root/etc/cont-init.d/40-plex-first-run, s6-overlay v2), cross-checked
byte-identical against the same file inside the running :latest image:

  * the claim exchange (line 106) is gated on PLEX_CLAIM being non-empty AND the
    PlexOnlineToken pref in /config being empty. Once the server is claimed the
    block is skipped entirely, so the env is a NO-OP on every later start
    whatever it holds ("If server is already logged in, this parameter is
    ignored" -- upstream README).
  * with an invalid/expired token on a still-unclaimed /config, plex.tv answers
    403 with an empty body; curl's status is never checked, the token sed yields
    empty, the success branch at line 120 does not fire, there is no else branch
    and no non-zero exit, and the script (no `set -e`) exits 0. The pod comes up
    UNCLAIMED AND SERVING. It does not crashloop.
  * the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS (default 0 = continue even
    if a cont-init script failed), so even a non-zero exit would not halt
    startup.

An ABSENT Secret is the real hard failure (secretKeyRef has no `optional`, so it
is CreateContainerConfigError), which is why the Secret is committed, encrypted
and listed in the wave kustomization rather than hand-created at claim time.

NO SCHEMA CHANGE. `optional` on secretKeyRef would only be needed if the pod had
to tolerate a missing Secret or could not start without a VALID token. Neither
holds: the Secret always exists in the Flux graph, and an invalid token is
non-fatal. schemas/servicedefinition.schema.json is untouched, and so are
catalog/render.py's env emission and the renderer contract.

Changes:
  * apps/wave-0/plex-claim-secret.sops.yaml -- new Opaque Secret plex-claim/plex,
    key PLEX_CLAIM, SOPS-encrypted to the committed age recipient. Its header
    carries NO dollar-brace expressions on purpose: kustomize preserves body
    comments into the stream envsubst reads (per
    conformance/tests/l0-static/flux-postbuild-substitution), so quoting the
    upstream shell conditions verbatim would have been a live substitution
    hazard on apps-wave-0, not prose.
  * catalog/plex.yaml -- the PLEX_CLAIM env via valueFrom.secretKeyRef, the
    "deliberately NOT set" block rewritten around the TTL + one-shot design, and
    the stale crashloop claim in the /config storage comment corrected.
  * apps/wave-0/plex.yaml -- re-rendered with the canonical command
    (`--all catalog -o apps --wave-layout --no-image-automation`). No L1 golden
    under conformance/tests/l1-rendering covers plex, so this is the only
    rendered surface that moves.
  * catalog/render.py -- plan_apps_waves places `<svc>-claim-secret.sops.yaml`
    in the app's OWN wave, like the bootstrap secret, plus a selftest assertion.
    Without this, the next `--emit-waves` would have silently dropped the Secret
    from apps/wave-0/kustomization.yaml and put the pod into
    CreateContainerConfigError. apps/wave-0/kustomization.yaml is the hand-pruned
    subset (vm-demo/keycloak are rendered but deliberately unlisted), so its one
    new line was applied without re-emitting the rest.
  * runbooks/plex-claim.md + its kustomization.yaml generator entry + README
    index -- the operator procedure that fits in 4 minutes, including the single
    forced-reconcile command and the mandatory pod roll (a changed Secret does
    NOT restart anything: PLEX_CLAIM is read only at container start and the pod
    template is byte-identical across this change).

Touched requirements: R-SEC-01 (secrets encrypted at rest), R-GITOPS-02
(ServiceDefinition onboarding contract).

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-28 22:26:44 -05:00
87 changed files with 3983 additions and 255 deletions

No files matched your search

+29 -15
View File
@@ -15,7 +15,7 @@ it touches is green** (R-TEST-03).
## The pipeline
```
lint ─▶ l0 (schema + Kyverno) ─▶ render-golden ─▶ l1 (rendering) ─▶ conformance (L2/L3) ─▶ merge-gate decision
decisions-log ─▶ lint ─▶ l0 (schema + Kyverno) ─▶ render-golden ─▶ l1 (rendering) ─▶ conformance (L2/L3) ─▶ merge-gate decision
└──────────────── run in CI, NO cluster ────────────────┘ └── in-cluster Chainsaw Job ──┘
live now cluster-gated (SKIPS in Phase 0)
```
@@ -40,9 +40,15 @@ and ADR-0011: **CI holds no broad kubeconfig.** L2/L3 run as an in-cluster **Cha
*collects* its artifacts, using a narrow trigger credential referenced **by secret name only**
(`secrets.CONFORMANCE_TRIGGER_KUBECONFIG`) — never a plaintext secret, never a cluster-admin kubeconfig.
## Honest SKIP — never fake green (Phase 0)
## Honest SKIP — never fake green
**No cluster exists in Phase 0.** The gate is deliberately built so unavailable checks **SKIP loudly,
**NOTE (2026-09-29):** this section is written in the past's voice — "Phase 0", "no cluster
exists". **A cluster has existed for weeks** and Flux has been reconciling against it. The
skip logic below is still correct and still honest, but read it as describing a **live
coverage gap** (L2/L3 is not wired into CI) rather than an expected phase. Its being
phrased as a phase is part of why the gap went unexamined.
Originally: The gate is deliberately built so unavailable checks **SKIP loudly,
they do not pass**:
- The `conformance` job is `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}` — with no cluster it is
@@ -77,21 +83,29 @@ Both dirs are git-ignored (see [`.gitignore`](../../.gitignore)).
## Triggers & branch model (CONTRIBUTING.md)
- `pull_request` → `main`: **PRs are the enforced merge gate into `main`.**
- `push` → `feature/**`, `fix/**`, `docs/**`, `chore/**`: fast feedback on the branch. **No work on `main`.**
- `push` → `main` only. The per-branch prefix list was removed: `act_runner` runs at capacity 1,
so a branch push fires a redundant run that serialises *ahead* of the PR run for the same tree.
**No work on `main`** — and since 2026-09-29 that is enforced, not conventional: `main` is a
protected branch and a direct push is rejected by the pre-receive hook (verified, not assumed).
## Transition: generic-CI-now → Gitea-Actions-after-Gitea-is-up
## Transition: COMPLETE (2026-09-29)
Per [`CONTRIBUTING.md`](../../../home-network-k3s/CONTRIBUTING.md) and
[`docs/05` §"local bootstrap → self-hosted Gitea"](../../../home-network-k3s/docs/05-gitops-and-delivery.md):
This section used to describe a migration as pending. It has happened. Gitea is live at
`gitea.home.keithjackson.dev`, the runner `incluster-dind` is registered and online, and
`gate.yml` runs on every PR.
1. **Now (interim):** there is **no remote git host** on this network. This workflow file is dormant —
but the **identical gate runs locally today** via `make -C ci gate` (R-TEST-02). Development is
local feature-branch → `main`, the gate enforced by convention + the local runner.
2. **Gitea comes online early** (one of the first services onboarded) at `gitea.home.keithjackson.dev`.
3. **CI moves to Gitea Actions:** this workflow activates unchanged. The only operational steps are
enabling a Gitea Actions runner and setting `vars.CLUSTER_AVAILABLE=true` +
`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` once a dev/test cluster exists. PRs then become the
**platform-enforced** merge gate — the same no-work-in-main rule, now enforced by machine.
**What took a month to notice:** the migration was recorded as done on 2026-08-31 without
anyone checking the platform. In reality `main` had **no branch protection** and **zero
runners were registered** until 2026-09-29. Every "the gate enforces this" statement in
between rested on an assertion nobody verified. PR #1 — the first pull request this
repository ever had — sat `pending` for 30.6 minutes because nothing existed to run it.
So: **do not read this file, or the DECISIONS-LOG, as evidence about the live platform.**
Query the API. `infra/gitea/` holds the scripts and the measured state.
Still genuinely outstanding: `vars.CLUSTER_AVAILABLE` and
`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` are unset, so L2/L3 conformance does not run in
CI. That is a real coverage gap, not a phase.
## Configuration (set in Gitea; referenced by name only)
+37 -4
View File
@@ -44,16 +44,31 @@
---
name: gate
# TRIGGERS — deliberately only `pull_request` into main, plus `push` to main itself.
#
# There is NO per-branch push trigger. There used to be a prefix list (feature/** fix/**
# docs/** chore/**) for "fast feedback on the branch", and it was a net negative here:
#
# * `act_runner` runs at capacity 1. A branch push and its PR fire the SAME gate over the
# SAME tree, and the redundant run SERIALISES AHEAD of the PR run — doubling the wall
# clock on the one result that actually gates the merge. That cost real time during the
# 2026-09-29 plex claim, where the loop is racing a short-lived token.
# * The list silently did not match this repo's own convention. Branches here are `feat/**`,
# which was absent, so the "fast feedback" never fired for the most common prefix anyway —
# a mismatch nobody noticed for a month because its only symptom was a MISSING run.
# Any hand-maintained prefix list has that failure mode; deleting it removes the class.
# * It is redundant. CLAUDE.md already requires `./ci/gate.sh` in the pinned toolbox before
# you push (74s locally), and the PR gate re-runs it authoritatively. A third run of the
# same checks buys nothing.
#
# `push: [main]` is KEPT: post-merge main must still be verified, and it is the only run for
# which no PR context exists.
on:
pull_request:
branches: [main]
push:
branches:
- main
- 'feature/**'
- 'fix/**'
- 'docs/**'
- 'chore/**'
concurrency:
group: gate-${{ github.ref }}
@@ -95,9 +110,27 @@ jobs:
# UNVERIFIED and NAMED in the PASS headline, never faked green. GATE_STRICT_SKIPS + the
# tolerated exit 3 are explained in the file header; the tolerance covers exactly one
# thing, the unverifiable spec repo, and nothing else can produce a 3 here.
# GITEA_SETTINGS_TOKEN feeds ci/gitea-settings.sh, the only check that can see Gitea's
# OWN database (branch protection + runner registration) — the blind spot that let `main`
# sit unprotected with zero runners for a month while this repo recorded both as working.
# Referenced BY NAME only; create it with infra/gitea/set-ci-token.sh.
#
# WIDENED TOLERANCE, DECLARED: `|| [ "$?" -eq 3 ]` used to cover exactly one skip
# (spec-immutability). It now also covers gitea-settings skipping when that secret is
# absent. That is a deliberate, temporary weakening: making an unwired credential BLOCK
# every PR is the lock-yourself-out failure this repo keeps walking into. Once the secret
# exists and the stage has been SEEN green, set REQUIRE_GITEA_SETTINGS=true below and the
# tolerance narrows back to spec-immutability alone.
- name: Merge gate — ci/gate.sh (what `make -C ci gate` runs)
env:
GATE_STRICT_SKIPS: "true"
SETTINGS_API_TOKEN: ${{ secrets.SETTINGS_API_TOKEN }}
# ENABLED 2026-09-29. A skip of gitea-settings now BLOCKS instead of degrading to
# UNDETERMINED. Safe because THIS PR's own CI run is the proof: if the stage were
# skipping, this line turns that into a failure and the PR goes red instead of
# merging. Green here means the token resolved and the live Gitea matched the
# declaration in infra/gitea/expected-branch-protection.json.
REQUIRE_GITEA_SETTINGS: "true"
run: ./ci/gate.sh || [ "$?" -eq 3 ]
# NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4
+8 -5
View File
@@ -11,11 +11,14 @@
# v
# merge-gate decision [R-TEST-03: touched subset green]
#
# Transition (CONTRIBUTING.md / docs/05 §"local bootstrap -> self-hosted Gitea"):
# Today there is no Gitea host, so this file is dormant. The SAME gate is runnable
# now via `make -C ci gate` (R-TEST-02). Once Gitea is self-hosted and Gitea
# Actions is enabled, this workflow becomes the ENFORCED PR merge gate into main —
# no code change required, only turning on the runner and setting CLUSTER_AVAILABLE.
# STATUS (corrected 2026-09-29): Gitea IS hosted and Gitea Actions IS enabled; the runner
# `incluster-dind` is online and gate.yml is the live PR gate. The text that used to sit here
# ("Today there is no Gitea host, so this file is dormant") was stale by weeks -- it was written
# as a plan and never revisited once the plan completed.
#
# THIS file is still dormant, but for the reason in the DORMANT block below -- its spec checkout
# and L2/L3 seams are unwired -- NOT because there is no host. Do not read a stale plan as a
# statement about the live platform; query the API instead (see infra/gitea/).
#
# CI holds NO broad kubeconfig. L2/L3 run as an in-cluster Chainsaw Job whose own
# scoped ServiceAccount holds cluster access (docs/14, R-SEC-02); CI uses only a
+74 -18
View File
@@ -1,7 +1,7 @@
# CLAUDE.md — how work lands in this repo
Read this before doing anything durable. It is the delivery contract for AI coding
sessions and agents. `DECISIONS-LOG.md`'s last entry is the current PM handoff and
sessions and agents. The newest entry in `decisions/` is the current PM handoff and
takes precedence on *what* to work on; this file governs *how* it lands.
## 1. Never push to `main`. Ever.
@@ -23,25 +23,58 @@ There is no other path. Specifically:
> (main is what the cluster follows), not what you are permitted to do. Honour this
> file, not that line. Fixing those runbooks is tracked work.
## 2. Enforcement is currently WEAKER than this rule. Follow the rule anyway.
## 2. Enforcement (as of 2026-09-29: the rule is now MACHINE-ENFORCED)
Verified against the live Gitea on 2026-09-29:
| Expected | Actual |
|---|---|
| `main` protected, direct push rejected | **No branch protection.** `protected: false`, `user_can_push: true` |
| PR blocked until the gate passes | **`enable_status_check: false`**, `status_check_contexts: []` |
| Gate runs on every PR | **No Actions runner is registered** (repo, org and site level all report zero) |
| Approval required before merge | **`required_approvals: 0`** |
| `main` protected, direct push rejected | ✅ `protected: true`, `user_can_push: false` — **verified by a real rejected push**, not by the API's word |
| PR blocked until the gate passes | ✅ `enable_status_check: true`, context `gate / gate (pull_request)` |
| Gate runs on every PR | ✅ runner `incluster-dind` online; ~1m35s |
| Approval required before merge | ✅ `required_approvals: 1` |
So nothing mechanically stops you from pushing to `main` right now. **That is a gap
being closed, not permission.** `DECISIONS-LOG.md` (2026-08-31) claims the gate is a
"required status check" with a runner online; that claim is not true of the live
system, which is how this went unnoticed.
All four were FALSE as recently as 2026-09-29, while this repo's own log asserted they were
true. They are true now because each was measured, and the push rejection was provoked rather
than assumed. **Re-measure before relying on any of them** — see §2b.
**Consequence for you:** because CI cannot run, the **local toolbox gate is the only
real gate**. Run it and quote its numbers in your report. Never describe a PR as
"green" on the strength of CI that did not execute.
Still not covered: `vars.CLUSTER_AVAILABLE` is unset, so **L2/L3 conformance does not run in
CI**. A green PR therefore proves the L0/L1 offline subset, not runtime behaviour.
**Consequence for you:** CI now runs, but it does not yet *block* anything. Always run
the toolbox gate yourself and quote its numbers. Never describe a PR as "green" on the
strength of a check you have not seen report — and note that a **skipped** Gitea job
reports as `success`, so a green tick is not by itself evidence that anything ran.
## 2b. Never infer platform state from this repository
**This repo is not evidence about the live platform.** Neither is the handoff log
(`decisions/`, and the frozen `DECISIONS-LOG.md` archive). Both describe what someone
believed at the time of writing, and both have been confidently wrong for a month at a
stretch.
On 2026-09-29 the log stated the merge gate was a required status check with a runner
online. The API said `branch_protections: []` and **zero** runners. Everything downstream
of that belief — "the gate enforces this", "your push would be rejected", "CI covers it" —
was wrong, and nothing in the tree could have revealed it, because the tree was the source
of the error.
**So, before asserting that any gate, protection, runner, secret, or workload is in a
given state: query the thing itself.**
| Claim about... | Check |
|---|---|
| branch protection / required checks | `GET /api/v1/repos/{owner}/{repo}/branches/main` |
| a runner exists | `GET /api/v1/admin/actions/runners` |
| CI actually ran | the commit's `/statuses` — and read the **description**; a *skipped* Gitea job reports `success` with "Has been skipped" |
| a workload / Secret / env | `kubectl --context k3s-prod ...` (read-only is fine for agents) |
| Flux applied your commit | compare `.status.lastAppliedRevision` against the GitRepository's artifact revision |
Quote the measured field, not the prose. If you cannot measure it, say so explicitly —
"unverified" is a legitimate report, and a far better one than a confident guess that
matches the docs.
**A green tick is not proof anything ran.** Read what actually executed.
## 3. Gate before you push
@@ -107,9 +140,32 @@ invocations concurrently.
flow. Do not design around committing one — read the expiry from the issuing page and
treat a burnt token as free to retry.
## 7. Before you finish
## 7. Before you finish — write a handoff entry, in its OWN file
Append a handoff entry to `DECISIONS-LOG.md` describing what you changed, what you
verified, and what you did **not** verify. State gaps plainly — an honest
"UNVERIFIED" is worth more than an optimistic "green", and this repo has already been
burned by the difference.
Add **one new file**:
decisions/YYYY-MM-DD-<slug>.md <slug> = your branch's last segment
with frontmatter (`date`, `branch`, `title`) and exactly these three sections:
## What changed
## What I verified
## What I did NOT verify
State gaps plainly — an honest "UNVERIFIED" is worth more than an optimistic "green",
and this repo has already been burned by the difference. All three headings are
**enforced** by `ci/decisions-log.sh`, because the third is the one that gets dropped.
Template and full rationale: `decisions/README.md`.
**Do NOT append to `DECISIONS-LOG.md`.** It is a frozen archive (pre-2026-09-29), its
SHA-256 is pinned in `decisions/ARCHIVE.sha256`, and the merge gate rejects any change
to it. Appending to its tail is what every agent used to do, and because every branch
appended to the *same last hunk*, no two concurrent branches could auto-merge. Measured
on 2026-09-29: **four of the five open PRs were `mergeable: false`, with
`DECISIONS-LOG.md` the only conflicting path in all four** — none of those changes had
anything to do with each other. Separate files have nothing to merge, which is the
entire fix. Note that the damage from getting this wrong lands on *other people's*
branches, later, which is why it is a gate stage and not a request.
Many older handoff entries and runbooks say "append to `DECISIONS-LOG.md`". That
wording is **stale**, in the same way §1's `git push origin main` wording is stale.
+950
View File
@@ -1,3 +1,16 @@
> **FROZEN ARCHIVE — 2026-08-16 … 2026-09-29. Do not append to this file.**
>
> New handoff entries go in **`decisions/`**, one file per entry
> (`decisions/YYYY-MM-DD-<slug>.md`). See [`decisions/README.md`](decisions/README.md).
>
> Why: every agent appended to this file's tail, so any two concurrent branches
> conflicted on the same last hunk. On 2026-09-29 that made four of the five open PRs
> unmergeable, with this file as the *only* conflicting path in all four. Separate files
> have nothing to merge.
>
> This file's SHA-256 is pinned in `decisions/ARCHIVE.sha256` and the merge gate
> (`ci/decisions-log.sh`) rejects any change to it.
# Autonomous Session Decision Log (PM)
Program Manager working independently while the operator is AFK (started 2026-08-16). Every
@@ -1192,6 +1205,13 @@ fresh agent should read this entry + `infra/PHASE1-STATUS.md` + `docs/plans/` an
Platform-admins require TOTP MFA.
- **Gitea Actions CI is LIVE + enforcing**: `.gitea/workflows/gate.yml` runs `make -C ci gate` on push/PR to main;
required status check; runner `incluster-dind` online (ns gitea-actions).
> **[CORRECTION, 2026-09-29 — THIS BULLET WAS FALSE WHEN WRITTEN AND STAYED FALSE FOR A MONTH.]**
> Measured against the live Gitea API on 2026-09-29: `branch_protections` returned `[]` (`main`
> `protected:false`, `enable_status_check:false`, `user_can_push:true`) and **zero** Actions runners
> were registered at repo, org or site level. The gate was neither enforced nor able to run. PR #1
> — the first pull request this repo ever had — was opened on 2026-09-29 and sat `pending` for
> 30.6 minutes with nothing to execute it. Both halves of this bullet were asserted, not verified.
> See the 2026-09-29 entries for the root cause and the fix.
### What changed this session (durable)
1. **SSO cluster-wide break→fix — root cause Cilium CVE-2025-30162.** The 1.16→1.20 security fix makes the shared
@@ -4781,3 +4801,933 @@ accurate before the PR flow and are stale now. Correcting them is tracked, not d
3. Land `feat/plex-claim-sops-secret` (PR #1) once the gate can actually run.
4. Claim plex, after checking it is not already claimed.
5. Correct the 37 runbooks' `git push origin main` wording.
## 2026-09-29 — Plex is claimed; the CI/CD gate that "enforced" it never existed. **PM HANDOFF.**
Began as "inject a plex.tv claim token". Ended by discovering the delivery pipeline was fiction,
rebuilding it, and shipping the token through it. Plex is **claimed** — `PlexOnlineToken` is
present in `/config` on the `plex-config` PVC, so the exchange is now a permanent no-op and the
library scan is running over the 8.33T NFS share.
### The finding: both halves of the 2026-08-31 CI claim were false
That entry recorded "Gitea Actions CI is LIVE + enforcing … required status check; runner
`incluster-dind` online". Measured against the live API on 2026-09-29:
| Claimed | Measured |
|---|---|
| `main` protected, gate required | `branch_protections` → `[]`; `protected:false`, `enable_status_check:false`, `required_approvals:0`, `user_can_push:true` |
| Runner online | **zero** runners at repo, org AND site level |
**PR #1 was the first pull request this repository has ever had.** Its contexts sat `pending`
for 30.6 minutes across 122 polls and never reported. For a month, every "the gate enforces this"
statement rested on an assertion nobody had checked — the same fail-open class as 2026-09-18,
and the reason it hid so long is that the log itself was the evidence.
### Root cause of the dead runner
`act_runner` registers **once** and persists its identity in `/data/.runner` on the
`act-runner-data` PVC. Gitea no longer held a matching row, so the pod booted, saw `.runner`,
**skipped registration**, and polled with an identity Gitea did not recognise. Symptom: zero
runners, pod Running, nothing crashlooping, no error anyone reads. Deleting the PVC forced a
genuine re-registration (`.runner is missing` → `Runner registered successfully`).
It then did not come back for several minutes, because Flux could not recreate the PVC:
`apps-wave-1` was blocked four levels deep on a **stale** `DependencyNotReady` for
`platform-config`, which itself read `Ready=True`. Each wave re-evaluates only on its own
`interval: 10m`, so a resource deleted at the bottom of the chain can take ~40 min to return.
**Force it** with `flux reconcile kustomization apps-wave-1 --with-source` rather than waiting.
### Corrections to things this log and the runbooks asserted
- **A skipped Gitea job reports `success`, not pending.** Measured: `gate / conformance` returned
`success` / "Has been skipped". An earlier entry (#2) said requiring it would deadlock every PR.
Wrong — it would pass **vacuously**, forever, while nothing ran. Under HARD RULE 4 that is worse
than a deadlock, because a deadlock is loud. Never require it; require
`gate / gate (pull_request)` alone. Corrected in #3.
- **`gate.yml`'s push trigger omits `feat/**`** (it lists `feature/** fix/** docs/** chore/**`).
With `act_runner` capacity 1, a branch prefix that IS in that list fires a second redundant run
which serialises ahead of the PR run and doubles wall-clock. Exploited deliberately in
`infra/plex/claim-ship.sh`; the honest fix is to align the trigger list.
- **The "~4-minute TTL"** asserted in four places was **uncited**, and upstream `pms-docker`
documents no expiry at all. The authoritative reading is the countdown plex.tv/claim renders at
issue time. Removed from `runbooks/plex-claim.md`; the same claim still sits in
`catalog/plex.yaml` and the Secret's header comment (left alone — editing a `.sops.yaml`
comment risks the MAC, and touching `catalog/plex.yaml` re-renders a golden).
### Measured, once the pipeline actually worked
`gate / gate (pull_request)` 1m34s–1m58s. Full claim loop end to end **~2m45s** — comfortably
inside the window, which retires the theory that a short-lived token cannot survive a PR flow.
That theory was mine and it was wrong: the pipeline was never too slow, it simply did not exist.
### State now
- `main` **protected**, `user_can_push:false`. Runner `incluster-dind` online.
- `CLAUDE.md` is the agent delivery contract; `infra/gitea/` holds the protection + runner scripts.
- **All 37 runbooks** that instructed `git push origin main` now describe the PR flow — that
instruction had become actively blocked, not merely stale.
- `infra/plex/claim-ship.sh` drives the claim loop end to end.
### Next steps
1. **`enable_status_check: true`** with `gate / gate (pull_request)` as the only context. The
prerequisite is met: the gate has been observed green on real PRs.
2. Add `jakman85` as a repo **collaborator**, then `required_approvals: 1`. (Currently the only
org member is `gitea-admin`, which agents authenticate as — requiring approval before that is
fixed would leave nobody able to approve.)
3. Run the **push probe** against protection: `gitea-admin` is a site admin and Gitea may let it
bypass. `user_can_push:false` is the API's opinion, not proof.
4. Align `gate.yml`'s push trigger with this repo's `feat/**` prefix.
5. Convert the `dind` container to a **native sidecar** with a `startupProbe`, per the standing
FOLLOW-UP note in `apps/wave-1/gitea-actions.yaml`. It deletes both hand-rolled wait loops and
the envsubst hazard. Land it ALONE — it is the merge gate for the whole repo.
6. Create the Plex libraries over `/media/movies`, `/media/tv`, `/media/photos`.
## 2026-09-29 (later) — the merge gate is now real, enforced, and proven. **PM HANDOFF.**
Closes the gap opened by the earlier entry today. Eight PRs, all gated; every claim below was
measured against the live API or cluster, not inferred from this repository.
### Final state
| | Verified |
|---|---|
| `main` protected | `protected:true`, `user_can_push:false` — and a direct push as site-admin `gitea-admin` was **provoked and rejected** (`pre-receive hook declined`) |
| Gate required | `enable_status_check:true`, contexts `["gate / gate (pull_request)"]` |
| Approvals | `required_approvals:1`; `jakman85` holds `owner` |
| Runner | `incluster-dind` online, now a **native sidecar** (`initContainers`, `restartPolicy: Always`, TLS-verified `docker info` startupProbe) |
| Gate time | ~1m35s |
| Plex | claimed 2026-09-29; `PlexOnlineToken` present on the `plex-config` PVC |
### What was corrected in the documentation
`CLAUDE.md` §2b — *never infer platform state from this repository* — is the durable outcome.
The 2026-08-31 entry asserting an enforced gate is now annotated **in place**. The "Phase 0 /
no cluster exists" framing across `ci/`, `clusters/` and `conformance/` READMEs is banner-marked
as describing a live gap rather than a stage. All 37 runbooks now describe the PR flow. Two
`.sops.yaml` headers and `catalog/plex.yaml` were corrected (comment edits verified not to break
the SOPS MAC, and not to move the rendered golden).
### A claim of mine that shipped wrong, and is now corrected in place
PR #7 stated the sidecar conversion would remove the one-per-start dockerd line
`http: TLS handshake error from 127.0.0.1: EOF`. **It does not.** Exactly one still appears,
immediately after `API listen on [::]:2376` — it is now the startupProbe's first attempt,
landing microseconds after dockerd begins listening and cut off by `timeoutSeconds: 1`; the
next poll succeeds. Cosmetic, and deliberately not chased: raising the timeout to hide one
benign startup line would slow every subsequent poll. The manifest comment now says so.
### The root cause nobody has fixed yet
Gitea's repo settings live in its own database — invisible to Flux and unreadable by the
conformance gate. **No check exists that could have caught "protection off, runner gone."**
`infra/gitea/` declares and can re-assert the desired state, but it is not a detector. A
conformance check that reads the live Gitea API and fails on divergence is the highest-value
remaining work in this repo, and it is the difference between this having been fixed and this
being unable to recur.
### Next steps
1. **Write that Gitea-settings conformance check.** Everything else here is a symptom.
2. Wire `vars.CLUSTER_AVAILABLE` + `secrets.CONFORMANCE_TRIGGER_KUBECONFIG` so **L2/L3 runs in
CI**. A green PR currently proves the L0/L1 offline subset only — runtime behaviour is
unverified on every merge.
3. Re-run `infra/gitea/verify-then-enforce.sh` after any future act-runner change.
4. `required_approvals: 1` means a PR you open as `jakman85` needs `gitea-admin` to approve.
If that friction bites, either open agent work as `gitea-admin` and review as `jakman85`, or
drop back to relying on the status check alone.
## 2026-09-29 (later still) — the enforcement we "turned on" was overridable by the account that merges
`enable_status_check: true` and `required_approvals: 1` were set earlier today and recorded as
done. **They did not bite.** `block_admin_merge_override` was `false`, and Gitea lets repository
admins override merge requirements without it. `jakman85` holds `owner`, so both requirements
were bypassable by exactly the account that performs every merge.
**The evidence was already on the record and nobody read it:** PRs #5, #6, #7, #8 and #9 all
merged while `required_approvals: 1` was in force, with **zero approvals recorded on any of
them**. A one-line check (`/pulls/{n}/reviews`) would have shown it at any point.
Two process failures, both mine, both the same shape as the original defect:
1. **I verified with the wrong view.** `/branches/main` reports `protected`,
`enable_status_check` and `required_approvals` but NOT `block_admin_merge_override`. It
showed everything green while the rule was overridable. The authoritative object is
`/branch_protections/main` — read it and enumerate its keys.
2. **I probed for a field name I assumed existed.** This Gitea (1.24.3) has no
`apply_to_admins`. Asking `jq` for it returns `null`, which reads exactly like "set to
false" but means "key absent" — so a PATCH that silently did nothing looked like a real
before/after. **`jq` returning null is not a value; it may be an absence.** Enumerate with
`to_entries`.
Now `block_admin_merge_override: true`, applied and read back. `infra/gitea/require-gate.sh`
sets it, and `infra/gitea/block-admin-override.sh` fixes it in isolation.
**This is still only a setting.** It is proven when a green PR is REFUSED a merge until
approved — and that has not been observed yet. This PR is the test: if it merges without an
approval, the fix did not work and the finding stands.
### The governance question this exposes
There are two accounts: `jakman85` (owner, the human) and `gitea-admin` (the agent). With
approvals now genuinely required, an agent-authored PR is approved by the human — which is the
intent. But a PR authored by `jakman85` needs `gitea-admin` to approve it, i.e. the automation
account rubber-stamping the human. That is ceremony, not review. Options: keep it and accept
the ceremony; drop `required_approvals` to 0 and rely on the status check, which is the gate
that actually verifies something; or add a second human account. **This is a decision for the
operator, not a default to be chosen silently.**
## 2026-09-29 (end of session) — **PM HANDOFF.** Six PRs open; read this before touching anything.
The session that began "set a SOPS secret" ended having rebuilt the delivery pipeline. Plex is
claimed and scanning. `main` is genuinely protected and the gate genuinely enforces. What is
open is listed below with the reason each is not merged.
### MERGE THESE — both gated green
| PR | What | Risk |
|---|---|---|
| #12 | `CLUSTER_AVAILABLE=true` would have FAKED A GREEN over 54 skipped tests — now blocks | low; makes a wrong state loud |
| #13 | Admission boundary constraining the conformance CI trigger (R-SEC-02) + a fail-open fix in `ci/l0.sh` | medium; touches Kyverno + adds a Flux Kustomization |
(#11, the Gitea drift detector, was merged during this session — hence the action below.)
Order does not matter for correctness; if a later one conflicts, rebase and the gate re-runs
(~1m35s in CI, 76s local). **Each was gated in the pinned toolbox, not on a host.**
### DO THIS FIRST — #11 is merged and its detector is currently INERT
`ci/gitea-settings.sh` SKIPS until its token exists, and **a skipped detector detects nothing**:
./infra/gitea/set-ci-token.sh # mints + stores GITEA_SETTINGS_TOKEN, never prints it
Then open any PR and confirm the gate prints `gitea-settings: green`. ONLY THEN uncomment
`REQUIRE_GITEA_SETTINGS: "true"` in `.gitea/workflows/gate.yml`, so a future missing token blocks
instead of skipping. Doing that before seeing it green locks the repo.
Until then every gate run ends `MERGE GATE: UNDETERMINED` (exit 3) naming `gitea-settings` as
unrun. That is BY DESIGN and `gate.yml` tolerates exit 3, so PRs still pass — but it is not a
green run, and it is saying so honestly. The token is what turns it green.
### A DIFFERENT COMPUTER NEEDS THREE THINGS THAT ARE NOT IN GIT
This is the part that will silently waste an hour otherwise. None of these are in the repo, by
design, and without them the toolbox and the cluster are unreachable:
1. **`~/.config/sops/age/keys.txt`** — the age PRIVATE key. Without it NO `*.sops.yaml` can be
read or edited, and `sops` will fail in a way that looks like a broken file. Copy it from this
workstation; the public recipient in `.sops.yaml` is only half the pair.
2. **`infra/ansible/.cache/prod.kubeconfig`** — untracked, so it exists only in the main checkout
on this machine. Without it `dev.sh` refuses to start. In a git WORKTREE it is absent even
here: use `DEV_KUBECONFIG=/dev/null` for anything that needs no cluster (see `CLAUDE.md` §3).
3. **A Gitea credential** for `gitea.home.keithjackson.dev` (this session used the stored
`~/.git-credentials`). `git push` and every `infra/gitea/*.sh` script needs it.
### STATE, MEASURED — do not re-derive it from this log, re-measure it
`CLAUDE.md` §2b exists because this log was confidently wrong for a month. Verify before relying:
protected:true user_can_push:false enable_status_check:true
status_check_contexts:["gate / gate (pull_request)"] required_approvals:1
block_admin_merge_override:true runner incluster-dind: online gate ~1m35s
### NEXT WORK, in priority order
1. **Finish the L2/L3 wiring** now that the credential model is decided and #13 lands the
boundary. Still missing (`docs/plans/l2-l3-conformance-ci-wiring.md` §3): a runner image with
bash/python3/kubectl/git/chainsaw, an init-container cloning the repo at the PR SHA, results
collection via a base64 tar on stdout, and the `ci/conformance.sh` rewrite to drive the Job.
**Tighten the admission policy's image pattern to the literal digest when that image exists.**
2. **Do NOT set `vars.CLUSTER_AVAILABLE=true`** until step 1 is done. It does not enable L2/L3; it
only changes which way the stage fails (see #12).
3. The `gitea-settings` detector covers branch protection + runners. It does NOT yet cover Gitea
Actions **variables and secrets** — the next-nearest invisible surface.
4. `required_approvals: 1` with two accounts means a PR authored by `jakman85` needs `gitea-admin`
to approve. The operator holds ultimate authority and does not require that review; revisit if
it becomes friction.
### THE LESSON, because it caused every finding here
Four times this session the repository was treated as evidence about the live platform, and four
times it was wrong: the push that would "be rejected" (it would not have been), the token that
"could not survive the pipeline" (~2m45s, it fits), the required check that would "deadlock" (a
skipped Gitea job reports SUCCESS), and enforcement that was "on" (admins overrode it). Two
fail-opens in the gate itself were found the same way — `kyverno test` certifying policies that do
not load, and `CLUSTER_AVAILABLE` certifying 54 skipped tests as passed.
**Measure, then assert. `CLAUDE.md` §2b is the rule; this entry is the evidence for it.**
## 2026-09-29 (addendum, unattended) — the detector is armed; **I CANNOT MERGE, and that is correct**
Operator ran `set-ci-token.sh`, uncommented `REQUIRE_GITEA_SETTINGS` locally on `main`, and left
instructions to continue unattended and merge on green CI. Two things came out of that.
### 1. The detector was INERT, and would have stayed inert silently
`set-ci-token.sh` PUT the secret as `GITEA_SETTINGS_TOKEN`. **Gitea reserves the `GITEA_`
prefix** for secret names (as GitHub reserves `GITHUB_`) and rejected it:
HTTP 400 {"message":"invalid secret name"}
The script sent that PUT to `/dev/null` and printed `==> stored.` unconditionally; `curl -sS`
exits 0 on an HTTP error. **A token existed; no secret did.** So `ci/gitea-settings.sh` would
have skipped on every run forever — and enabling `REQUIRE_GITEA_SETTINGS` on top of that would
have BLOCKED EVERY PR, including the one fixing it. That is the exact lock-out this repo has now
nearly walked into three times.
Fixed in **PR #15**: the secret is `SETTINGS_API_TOKEN`, and `set-ci-token.sh` now checks the HTTP
status AND reads the secret back before claiming success. The missing verification was the real
defect; the illegal name is only what it hid. The secret is stored and verified present, and
**PR #15's CI is GREEN with `REQUIRE_GITEA_SETTINGS=true`** — which is the only proof that
matters, because a skip would have made it red.
The operator's local edit could not be pushed to `main` (protected). It is carried in PR #15.
### 2. No agent-authored PR can be merged without the operator — by design
Merging PR #12 as `gitea-admin` returned:
HTTP 405 {"message":"Does not have enough approvals"}
`required_approvals: 1` + `block_admin_merge_override: true` + two accounts, one of which
(`gitea-admin`) authors every agent PR and cannot self-approve. **So "merge on green CI" is not
something an agent can execute here.** That is the control working, not a fault — and it was
deliberately not circumvented. Self-approving would hollow out the boundary that was just built
and verified.
**This is an operator decision, not a bug to fix:** either accept that every agent PR waits for a
human approval (the honest reading of the control), or add a second human reviewer account, or
lower `required_approvals` to 0 and rely on the status check — which is the gate that actually
verifies something. Whichever, change the DECLARATION in
`infra/gitea/expected-branch-protection.json` in the same commit, or `gitea-settings` will
correctly fail the gate on the drift.
### Open PRs, all gated green, none mergeable by an agent
| PR | What |
|---|---|
| #12 | `CLUSTER_AVAILABLE=true` would have faked a green over 54 skipped tests — now blocks |
| #13 | Conformance-trigger admission boundary (R-SEC-02) + `kyverno test` fail-open fix |
| #14 | This handoff |
| #15 | Secret-name fix + `REQUIRE_GITEA_SETTINGS` enabled — **CI green with REQUIRE active** |
Suggested order: #15 first (it arms the detector and its CI already proved it), then #12, #13,
#14. If a later one conflicts, rebase; the gate re-runs in ~1m36s.
### Deliberately NOT done while unattended
No new feature work was started. Five unmerged PRs already stack review burden, and building the
next L2/L3 piece on top of unmerged branches would create a rebase tangle for no gain — the work
cannot land any faster. The next piece is specified and ready to start once these are merged:
`docs/plans/l2-l3-conformance-ci-wiring.md` §3, with §4's credential model now decided and its
boundary landed in #13.
## 2026-09-29 — Bedrock "cannot connect on 19132": the platform delivered the packets, the server never opened the socket
**Branch:** `fix/crafty-bedrock-19132` → PR to `main`. **Read-only diagnosis; nothing was applied.**
**Gate:** see "Gate" below — quote it, do not round it up.
### The report, and what actually refuses
Reported as "users are unable to connect to Crafty Minecraft servers listening on UDP port 19132
(Bedrock, slot 0)". No client type, no error text, no history.
**Nothing was listening on UDP 19132.** A Bedrock server exists and its process is alive, but it is
not speaking RakNet, so it never binds a UDP port. The entire network path below it is correct and was
measured as such, including *counted packets that arrived and were allowed*.
The chain, each line a measured field with the command that produced it:
1. **The server exists and runs.** `kubectl -n crafty exec … -c crafty -- ps -ef`:
`crafty 46689 1 ./bedrock_server`, started 20:54, cwd
`/crafty/servers/c5837301-12b1-404e-b50e-70eafb37226b`. Crafty's DB (`select * from servers` via
`/crafty/.venv/bin/python`, read-only URI) row 2: `server_name='K3s-Bedrock'`,
`type='minecraft-bedrock'`, `server_port=19132`, `created='2026-09-29 20:51:53'`.
So hypothesis "the operator never created a Bedrock server" — the PM's prior favourite, and mine —
is **false**. It was created ~20 minutes before I looked.
2. **Nothing is bound to UDP 19132.** `cat /proc/net/udp` and `/proc/net/udp6` in the crafty
container: **no `4ABC` local port in either family**. IPv4 UDP held nothing but an ephemeral
(`BBBF`=48063); IPv6 UDP nothing but `1D7F`=7551. Meanwhile `/proc/net/tcp6` line 0 is
`…:4ABC … 0A` — port 19132 is bound, as **TCP**, LISTEN. (Java's 25565 = `63DD` and Crafty's
8443 = `20FB` are both there in `tcp6`, so the Java server is genuinely up.)
3. **Why: `transport=nethernet`.** `grep -v ^# server.properties` on the Bedrock server dir:
`server-port=19132`, `server-portv6=19133`, **`transport=nethernet`**, `enable-lan-visibility=true`.
The file's own shipped documentation, quoted verbatim: `# Allowed values: "raknet" or "nethernet"`,
and above `server-portv6`: `# This property is ignored when transport=nethernet and a dual-stack
socket will be opened on server-port instead`. NetherNet is the WebRTC/Xbox-signalled transport —
it opens no RakNet UDP listener. Consistent with the pod's outbound `tcp` entries to
`150.171.110.177:443` / `.178:443` (Microsoft) and the churning ephemeral UDP sockets.
4. **Nobody set that; it is the vendor default.** `server.properties` mtime `Sep 29 20:52` equals the
`bedrock_server` binary's — untouched since extraction — and `grep -rn transport
/crafty/app/classes/minecraft/` returns **nothing**: Crafty has no code that reads or writes this
property. So a freshly-downloaded BDS of this build **never** listens on `19132/udp`, and no amount
of Crafty-side configuration changes that.
5. **The failure is observable from inside the pod, from localhost.**
`grep raknet /crafty/logs/commander.log`, every 5–30 s since 21:02:
`WARNING - app.classes.remote_stats.raknet_ping - Got timeout while issuing bedrock ping to
127.0.0.1:19132`. Crafty's only Bedrock health probe is `raknet_ping.py` (`ls
/crafty/app/classes/remote_stats/`), so the panel is asking the right question and getting silence
**over loopback**. That single line eliminates every network hypothesis at once.
6. **Real player traffic reached the pod and was permitted.** `cilium-dbg bpf policy get <crafty ep>`
on `cilium-wdh82` (k3s-prod-1):
`Allow Ingress reserved:world 19132/UDP … BYTES 27660 PACKETS 26`.
**26 packets of world traffic were allowed in on 19132/UDP.** The policy did not drop them; the
application had no socket to receive them. `reserved:remote-node 19132/UDP` is also an Allow (0
bytes — the packets landed on the node that both announces the VIP and runs the pod, so no second
hop), which re-confirms the 2026-09-14 finding that `cluster` subsumes `remote-node` here.
**Verdict: this is not a platform defect and not a manifest defect. It is the Bedrock Dedicated
Server's default `transport=nethernet`, plus a README that told the operator creating a server on
19132 was sufficient.** The repo-side defect is documentation, and that is what this branch changes.
### The platform path, measured sound (so the operator can rule it out for good)
| Claim | Measured |
|---|---|
| `crafty-l4-udp` really is `Cluster` | `kubectl get svc crafty-l4-udp -o yaml` → `spec.externalTrafficPolicy: Cluster` (the 2026-09-14 fix IS live, not just in git) |
| VIP allocated | `status.loadBalancer.ingress[0].ip: 192.168.2.22`, `ipMode: VIP`, condition `cilium.io/IPAMRequestSatisfied=True reason=satisfied` |
| Backends exist | EndpointSlice `crafty-l4-udp-68kws` → `10.0.0.130`, ports `19132,19133,19134` |
| Datapath frontend present | `cilium-dbg service list` on **both** k3s-prod-1 and k3s-prod-3: `192.168.2.22:19132/UDP LoadBalancer 1 => 10.0.0.130:19132/UDP (active)` — i.e. any node can serve it, which is exactly what `Cluster` is for |
| Policy admits it | datapath policy map, `Allow Ingress reserved:world 19132/UDP` (+ `host`, `remote-node`, `unmanaged`, `init`, `health`, `kube-apiserver`) |
| LAN DNS resolves | `nslookup crafty-mc.home.keithjackson.dev 127.0.0.1` in the pihole pod → `Address: 192.168.2.22` |
| Flux applied current `main` | Kustomization `crafty` `.status.lastAppliedRevision = main@sha1:20ea70d…` == GitRepository `flux-system` `.status.artifact.revision = main@sha1:20ea70d…` |
### A real second finding: the shared VIP's two leases sit on DIFFERENT nodes
`kubectl -n kube-system get lease … | grep l2announce`:
cilium-l2announce-crafty-crafty-l4-tcp k3s-prod-3
cilium-l2announce-crafty-crafty-l4-udp k3s-prod-1
cilium-l2announce-unifi-unifi-l4-tcp k3s-prod-2
cilium-l2announce-unifi-unifi-l4-udp k3s-prod-2
cilium-l2announce-pihole-pihole-dns-tcp k3s-prod-3
cilium-l2announce-pihole-pihole-dns-udp k3s-prod-3
Crafty is the **only** shared-VIP pair that is split: **two nodes ARP-announce `192.168.2.22`
simultaneously.** unifi and pihole happen to be co-located. This is the per-Service, backend-blind
election described in `service-l4.yaml` doing exactly what that comment says it does — and it is a
*coincidence* that the other two pairs agree, not a property.
**I am not calling this the bug, and I am not changing anything for it**, because under
`externalTrafficPolicy: Cluster` it is benign by construction: `cilium-dbg service list` shows the
full frontend set with an active backend on **both** announcing nodes (table above), so whichever
node's ARP reply a client caches, the packet is forwarded to the pod. The cost is ARP-cache churn on
LAN clients, not a black hole — and this is why the 2026-09-14 `Local`→`Cluster` flip was the right
fix rather than a lucky one. **No new L0 check is warranted:**
`conformance/tests/l0-static/l2-announced-vips-serve-from-any-node/check.sh` already forbids
`Local` on any announced Service tree-wide, which is precisely the invariant that makes a split
harmless. An additional "shared-VIP pairs must be co-located" check would assert a *runtime* lease
placement a static check cannot see, and Cilium offers no field to pin it.
### Hypotheses tested and discarded, so nobody re-runs them
- **Wrong port / IPv6-only bind (h2).** Partly right in shape, wrong in mechanism: `server-port=19132`
is correct and the process is not v6-only — it binds *no* UDP port at all. Measured, not assumed.
- **Shared-VIP ARP split (h3).** Real (above), not causal.
- **`Cluster` not actually live on the UDP Service (h4).** False; it is live, with datapath to match.
- **Loaded policy ≠ git policy (h5).** False; dumped the BPF map, 19132/UDP is an Allow and has
counted traffic.
- **SNAT collapsing RakNet sessions (h6).** **UNTESTED AND UNTESTABLE TODAY** — there is no RakNet
session to collapse. It remains a live theory for *after* the transport is fixed: Cilium is in SNAT
mode (no `loadBalancer.mode: dsr`), so every LAN player will reach the server as one address, and
RakNet keys sessions on source `ip:port`. Nothing was measured about masquerade port-reuse or UDP
conntrack here. Do not treat this paragraph as evidence of anything.
- **DNS (h8).** False; pihole answers `192.168.2.22` (table above).
- **Client-side realities (h7).** Not the *cause* of a dead socket, but they will be the next wall:
LAN/Friends discovery cannot work behind a VIP, and consoles cannot type a server address. Both are
now documented in `platform/crafty/README.md` rather than left for the next person to rediscover.
### What changed in this repo
`platform/crafty/README.md` only. **No manifest was touched** — the manifests are provably correct,
and adding ports or widening policy for this would have been manufacturing work to look productive.
Added: *"Bedrock on UDP 19132 — the transport trap"* (the measurement chain above, and the
`raknet_ping` warning as its signature), *"Bedrock realities that are NOT platform defects"* (no LAN
discovery, consoles, the `enable-lan-visibility` collision with slots 1–2, `server-portv6`). Corrected
three stale claims: "the operator has **not created any servers yet**" (two exist), the human test
that implied a started Bedrock server on 19132 is reachable, and the single-server *Runtime state*
section (now a two-row table).
### What I did NOT verify — read this before acting
- **That `transport=raknet` fixes it is REASONED, NOT OBSERVED.** It follows from the shipped property
docs (`Allowed values: "raknet" or "nethernet"`; `server-portv6` "ignored when
transport=nethernet") and from the absence of any UDP bind under `nethernet`. Editing
`server.properties` is a write to application state on the PVC, which is an operator action, so I
did not try it. **Nobody has seen a packet answered on 19132/udp on this cluster.**
- **No end-to-end LAN test.** I never sent a RakNet unconnected-ping from a LAN host to
`192.168.2.22:19132`. I did not need to: the same ping fails from `127.0.0.1` inside the pod.
- **Nothing was applied.** No `kubectl apply/edit/patch/scale`, no `flux reconcile`, no file written
inside the container. The running pod is still `crafty-64f7b6d78-pkpgp` (2/2, 15d, 0 restarts) on
the **pre-`fix/crafty-oauth2-proxy-signin`** spec — that PR is still unmerged.
- **Who those 26 packets came from is unknown.** Byte/packet counters prove arrival and permission,
not identity. I did not run a live `cilium monitor` or `tcpdump`, so "users were trying and being
ignored" is a strong inference, not a captured flow.
- **The Java server's reachability was not tested.** `25565` is bound (`tcp6 63DD LISTEN`) and its
datapath frontend is active on both nodes, but no client connected during this session. The TCP
lease being on k3s-prod-3 while the pod is on k3s-prod-1 is `Cluster` working as designed — asserted
from the service table, not from a connection.
- **The NetherNet side was not explored.** Whether this server is currently joinable *at all* through
Xbox/Friends signalling (its actual transport) is unknown and untested.
- **Bedrock slots 1 and 2 (19133/19134) have never had a server.** The `enable-lan-visibility`
collision I documented for them is read from the shipped comment, not provoked.
### Operator actions required (I cannot do any of these)
1. **Make the Bedrock server speak RakNet.** In Crafty: stop `K3s-Bedrock`, edit
`/crafty/servers/c5837301-12b1-404e-b50e-70eafb37226b/server.properties` →
`transport=raknet`, start it. Then confirm the socket exists before telling anyone it works:
`kubectl --context k3s-prod -n crafty exec <pod> -c crafty -- grep 4ABC /proc/net/udp /proc/net/udp6`
should now show a wildcard bind, and the `raknet_ping` timeout warnings in
`/crafty/logs/commander.log` should stop. **Both checks, not one.**
2. **Then tell the players how to connect.** Windows/mobile client → **Add Server** →
`crafty-mc.home.keithjackson.dev`, port **19132**. It will *not* appear in the LAN/Friends tab, and
a Switch/Xbox/PlayStation client cannot reach it at all. If the reported users are on consoles,
step 1 will not help them and the answer is "unsupported without a DNS redirect".
3. **Fix `auto_start` on the new server.** DB row `K3s-Bedrock` has `auto_start=0` **and**
`crash_detection=0` — the create-API trap already documented in this repo, live again.
`PATCH /api/v2/servers/c5837301-…` setting both true, **and read them back**. Until then the
Bedrock server does not survive a pod restart — and merging
`fix/crafty-oauth2-proxy-signin` restarts the pod (`strategy: Recreate`, RWO volume), stopping
*both* Minecraft servers.
4. **Optional, cosmetic:** to stop two nodes ARP-announcing `192.168.2.22`, delete
`lease/cilium-l2announce-crafty-crafty-l4-tcp` in `kube-system` and let it re-elect — a 1-in-3
chance of landing on k3s-prod-1. Not worth doing: under `Cluster` the split costs nothing but ARP
churn, and a redraw is equally random.
5. **After step 1, watch for the SNAT/RakNet question (h6 above)** — if a *second* simultaneous player
evicts the first, that is the SNAT theory becoming real and it needs its own measured
investigation (masquerade port reuse, UDP conntrack), not a guess.
## 2026-09-29 (later) — Bedrock IS reachable on 19132/udp. The `TRANSPORT TYPE ERROR` banner is advisory, and the last entry's "REASONED, NOT OBSERVED" call has now been OBSERVED — correct
**Do not read this as overturning the previous entry's recommendation. It confirms it.** What it
overturns is the conclusion the operator and I both drew from the server's *reaction* to it.
### The sequence
The entry above ended with operator action 1: stop `K3s-Bedrock`, set `transport=raknet`, start it —
explicitly flagged **"REASONED, NOT OBSERVED … Nobody has seen a packet answered on 19132/udp on this
cluster."** The operator did it. `/crafty/logs/audit.log`, measured:
21:32:32 issued command stop_server for server K3s-Bedrock
21:33:31 Edited file /crafty/servers/c5837301-…/server.properties
21:33:58 issued command start_server for server K3s-Bedrock
BDS then printed, at ERROR level, `================ TRANSPORT TYPE ERROR ===================` /
`Your current connection type is not set to NetherNet. In this release, NetherNet is the only supported
transport type.` / `Players will not be able to connect to your game without NetherNet.` That was
reported to me as *"the server rejected it"*, and I was briefed to treat the shipped property comment
(`Allowed values: "raknet" or "nethernet"`) as stale relative to the binary and to write up an
architecture finding: that BDS is NetherNet-only, can never bind `19132/udp`, and that the whole
`crafty-l4-udp` exposure is therefore inert.
**That brief was wrong, and so was the inference behind it. The server did not reject anything.**
### What is actually true (all OBSERVED, 2026-09-29 21:38–21:41)
- `./bedrock_server` pid **47006** is running, started 21:34 and still up — it printed that banner and
carried on. It did not exit and Crafty did not restart it.
- It **bound the RakNet sockets**: `/proc/net/udp` `00000000:4ABC` (19132/IPv4) and `/proc/net/udp6`
`…:4ABD` (19133/IPv6), both owned by that pid (`/proc/47006/fd/25`, `fd/26` → those inodes). The
**TCP** bind on `4ABC` that the last entry measured under nethernet is **gone**.
- Crafty's warning changed kind at that exact moment: last `Got timeout while issuing bedrock ping` at
**`21:34:04,306`**, first `Bedrock ping returned a string with too few fields` at **`21:34:09,854`**
(155 timeout lines, then 86 answered and counting). Per `raknet_ping.py`, `Got timeout` is emitted
*only* when `sock.recv` raises `TimeoutError`; `too few fields` is reachable *only after* a `0x1C`
Unconnected Pong arrives **and its magic validates**. The listener answers.
- **The end-to-end test the last entry admitted it had never run, run:** a RakNet Unconnected Ping from
the toolbox on the LAN (host network, off-cluster) to the prod VIP **`192.168.2.22:19132`** returns a
139-byte `0x1c` pong in ~12 ms, magic valid:
MCPE;Dedicated Server;2193;1.26.52;0;10;11676629557768398479;Bedrock level;Survival;1;19132;19133;0;0;0;
BDS **1.26.52**, protocol 2193, 0/10 players. `:19133` returns the **same** pong from the same
`server_guid` (that is `enable-lan-visibility=true`, now observed rather than read off a comment);
`:19134` times out, correctly. **A Bedrock player on this LAN can connect right now.**
- The vendor's own doc on the PVC, `bedrock_server_how_to.html` (shipped in the same archive as the
binary, same mtime 20:52), documents `transport` as *"raknet, nethernet"* and: *"**raknet** — The
traditional Bedrock UDP transport. The server listens directly on `server-port` (IPv4) and
`server-portv6` (IPv6), and clients connect to that address and port."*
- RakNet is still fully compiled in: `strings bedrock_server` yields `RakNetConnector`,
`createUniqueRakPeer`, `RakPeerInterface`, and the vendored build path
`/mnt/vss/_work/1/s/handheld/src-deps/raknet/raknet/RakNetSocket2.cpp`.
So `NetherNet is the only supported transport type` is a **support** statement, not a **capability**
one. Microsoft declines to support RakNet and ships it working, at ERROR log level, in 1.26.52.
### Why nethernet could never have worked here anyway — a finding the last entry missed
The same vendor doc explains the last entry's oddest measurement (19132 bound **TCP**): nethernet
*"listens on a single dual-stack **TCP** socket on `server-port` for an HTTP-based signaling handshake,
then negotiates a UDP connection with each client."* And this repo exposes **no TCP 19132**:
`crafty-l4-tcp` carries 25565-25567 only, and `crafty-allow-ingress` allows `19132 UDP` and not
`19132 TCP`. The signaling port was unreachable at the Service layer *and* the policy layer. The
server's own `packet-statistics.txt` for that 2293-second nethernet run agrees: `Total Received: 0, 0B`.
The 26 packets / 27660 bytes the last entry found on the `19132/UDP` policy map were players' RakNet
pings arriving at a port where nethernet had opened no UDP socket. Consistent, and now closed.
### Where the reasoning failed — the part worth keeping
The previous entry's reasoning was **sound and its recommendation was right**; it was correctly
labelled as unobserved, and the observation vindicated it. The failure was downstream, in reading the
server's response:
1. **An ERROR-level log line was read as a refusal.** It was a deprecation notice. Nothing checked
whether the process survived it or what it bound — and both were one command away.
2. **"The vendor comment is stale relative to the binary" was asserted from the binary's *complaint*,
never from the binary's *behaviour*.** The binary had two other testimonies on disk — the shipped
how-to, and its own symbol table — and both said `raknet` works. Neither was consulted.
3. The tiebreak between two contradicting vendor statements is never a third piece of prose. It is
`/proc/net/udp` and a pong on the wire. That is what settled it, in about four minutes.
This is CLAUDE.md §2b applied one level further in: don't infer platform state from the repo, **and
don't infer application state from the application's own error message.** Query the socket.
### Decision and recommendation
**Keep `transport=raknet`. No manifest change.** `service-l4.yaml`, `networkpolicy.yaml` and the
`containerPort`s were already shaped for exactly RakNet and are now proven end-to-end rather than
assumed. Ranked alternatives, all rejected for today and written up in the README:
1. **Keep `transport=raknet`** — zero manifest change, works now, vanilla parity, real Bedrock world.
2. **GeyserMC + Floodgate** on the existing `homelab` Paper server — also zero manifest change (Geyser
itself binds `19132/udp`, so the existing Service is already correct), held in reserve for the day
RakNet is genuinely removed. It **cannot run alongside** the live BDS: both want 19132. Cost is that
Bedrock players join the **Java** world. Geyser's current Bedrock-version support is
**external and unmeasured**.
3. **Pin/downgrade BDS** — unnecessary (the current build speaks RakNet) and fragile.
4. **PocketMine-MP / Nukkit** — no; loses vanilla parity and world format.
5. **Accept NetherNet** — worst option: it needs *more* manifest work than the one that already works
(TCP 19132 in `crafty-l4-tcp` **and** in `crafty-allow-ingress`, plus `server-udp-ports` pinned and
exposed), and WebRTC media behind a shared VIP and SNAT is unproven here.
The 19132-19134/UDP exposure **stays** and is **not** inert. That part of the brief is refuted.
### What changed in this repo
`platform/crafty/README.md` only — again **no manifest touched**, and this time that is a measured
conclusion rather than a cautious one.
- Replaced *"the transport trap"* with a two-trap section: nethernet hides the UDP socket, and then the
`TRANSPORT TYPE ERROR` banner frightens you into reverting the fix. Records the full evidence chain,
the vendor how-to quotes, the binary symbols, and the off-cluster pong.
- Documented that Crafty's perpetual `too few fields - missing values [None]` warning is the **success**
signature, not a failure: BDS 1.26.52's pong has 16 `;`-fields, Crafty 4.10.8's
`pingpong_string_fields` knows 14 and wants an empty terminator, gets `'0'`. Cosmetic. The failure
signature is the *other* message, `Got timeout`.
- Added the ranked options table with each option's manifest consequence, and an *"If RakNet is ever
really removed"* section naming Geyser+Floodgate as the migration with the exact PVC/panel steps.
- Upgraded the `enable-lan-visibility` note from "read off a comment" to OBSERVED, and **corrected** the
claim that `19133` is unreachable — it is reachable, on IPv4, and slot 1 genuinely is occupied.
- Corrected the **Game VIP**: the README stated `192.168.1.55` as fact. That is the base/dev value;
`clusters/prod/kustomization.yaml` patches `CRAFTY_L4_VIP` to `192.168.2.22`, which is the live
`EXTERNAL-IP` on both `crafty-l4-tcp` and `crafty-l4-udp`. Rewrote as the variable plus the measured
prod value.
- Noted that `transport=raknet` is PVC state Flux cannot restore, and that a recreated or reinstalled
Bedrock server silently regresses to the vendor default `nethernet`.
- Re-confirmed `K3s-Bedrock` `auto_start=0` / `crash_detection=0` straight out of `crafty.sqlite`.
### What I did NOT verify — read this before acting
- **No human Bedrock client has connected.** I proved the RakNet *ping/pong* handshake end-to-end from
off-cluster; I did not complete a login, which additionally requires Xbox Live auth
(`online-mode=true`) from the client. **UNVERIFIED that a player reaches the world**, and it is the
one test only a human with a Minecraft client can run.
- **The multi-player SNAT question from the last entry is still open and still untested.** Whether a
*second* simultaneous Bedrock player evicts the first (masquerade port reuse / UDP conntrack behind a
shared VIP) is unmeasured. One ping proves one flow.
- **Whether a console can reach it is still no** — unchanged and still not tested; consoles have no
server-address field.
- **Geyser is entirely unmeasured here.** Nothing was installed, downloaded, or run. Its Bedrock-version
support, its jar names and its config keys as written in the README are **web/documentation-sourced,
not observed on this cluster**. Verify before acting on that section.
- **Whether Crafty can pin a BDS version is UNVERIFIED** — I did not read its Bedrock installer.
- **A web search on this error was attempted and was low-quality** — results largely echoed the error
string back. Nothing in the README or this entry rests on it. The load-bearing evidence is all local:
`/proc/net/*`, `/proc/47006/fd/*`, `audit.log`, `commander.log`, `raknet_ping.py`, the shipped
`bedrock_server_how_to.html`, `strings bedrock_server`, and the pong.
- **Whether a future BDS removes RakNet for real is unknowable from here.** Treat the banner as a
deprecation notice with an unknown expiry.
- **Nothing was applied or written.** No `kubectl apply/edit/patch/scale/delete`, no `flux reconcile`,
no write anywhere inside the container — `server.properties` was read, never touched. The running pod
is still `crafty-64f7b6d78-pkpgp` (2/2, 15d, 0 restarts). All probes were reads plus outbound UDP
pings from the toolbox.
- **`fix/crafty-oauth2-proxy-signin` is still unmerged**, so the pod still runs the pre-that-PR spec.
### Gate and CI — rc 3, and why CI's green does not contradict that
Local, pinned toolbox, from the worktree (`ENGINE` shim + `DEV_KUBECONFIG=/dev/null`), diff base
`origin/main @ 20ea70d` → `HEAD @ d29a73c`:
| Stage | Result |
|---|---|
| `lint` | green |
| `l0` | green — 19 cases, 0 failures, 0 skips |
| `render-golden` | green — all 10 onboarded goldens reproduce |
| `l1` | green — 3 cases, 0 failures, 0 skips |
| `spec-immutability` | **green** (see below) |
| `gitea-settings` | **SKIPPED** — `SETTINGS_API_TOKEN` unset |
| `conformance` (L2/L3) | **SKIPPED** — `CLUSTER_AVAILABLE != true` |
**Exit code 3 — UNDETERMINED, not a PASS.** Nothing failed; nothing outside the green rows is
certified. Touched requirements `R-DNS-01 R-NET-01 R-PLAT-02 R-STATE-01` remain L2/L3 UNVERIFIED.
`spec-immutability` — normally *the* tolerated skip — was closed rather than accepted. It had skipped
**spuriously**: `dev.sh` resolves the spec repo at `$REPO_ROOT/../home-network-k3s`, which from a
*worktree* is the wrong directory; the repo exists beside the **main** checkout. Mounting it read-only
through the `ENGINE` shim turned the skip into a real green, so R-TEST-06 is genuinely verified here.
Worth folding into the CLAUDE.md §3 worktree workarounds: the documented shim should mount **two**
paths, the main `.git` *and* the sibling spec repo.
CI on `d29a73c`, statuses read by description and not by tick:
context='gate / gate (pull_request)' status='success' description='Successful in 1m35s'
context='gate / conformance (pull_request)' status='success' description='Has been skipped'
**Neither green means rc 0.** `gate / conformance` is `if: false` in `.gitea/workflows/gate.yml` — it
never runs and reports `success` with "Has been skipped", the exact trap CLAUDE.md §2 names. And
`gate / gate`, the single required context, runs `./ci/gate.sh || [ "$?" -eq 3 ]` (line 134): **CI
tolerates exit 3 by declared design**, so its green means "exited 0 *or* 3". It would report this
identical green for the rc 3 above. Enforcement itself was measured directly rather than assumed
(§2b): `branches/main` → `protected: true`, `user_can_push: false`, `required_approvals: 1`,
`enable_status_check: true`, `status_check_contexts: ['gate / gate (pull_request)']`;
`admin/actions/runners` → 1 runner, `incluster-dind`, `online`. **PR #23 is open, unmerged, and I did
not merge it.**
### Operator actions required
1. **Nothing to do about the transport.** `transport=raknet` is correct, live, and answering. **Do not
revert it because of the `TRANSPORT TYPE ERROR` banner, and do not chase Crafty's `too few fields`
warning** — both appear on a fully working server.
2. **Get a human to actually join**, from a Windows/mobile client: **Add Server** →
`crafty-mc.home.keithjackson.dev`, port **19132**. This is the only remaining unproven step. It will
never appear in the LAN/Friends tab; a console cannot reach it at all.
3. **Fix `auto_start` — still outstanding, and now more urgent.** `K3s-Bedrock` is still
`auto_start=0`, `crash_detection=0`. `PATCH /api/v2/servers/c5837301-12b1-404e-b50e-70eafb37226b`
setting both true **and read them back** (the create API silently drops them). Until then the
Bedrock server does not survive a pod restart — and merging `fix/crafty-oauth2-proxy-signin`
restarts the pod (`strategy: Recreate`, RWO volume), stopping **both** Minecraft servers **and**
leaving Bedrock down until someone starts it by hand.
4. **Set `enable-lan-visibility=false` on `K3s-Bedrock`** if a second Bedrock server is ever wanted.
It currently occupies **both** 19132 and 19133 (measured), so slot 1 is not actually free. LAN
discovery cannot work behind the VIP anyway, so nothing is lost.
5. **Re-check after any Bedrock server recreate or BDS update:** confirm `transport=raknet` survived,
then re-ping. The vendor default is `nethernet`, and the regression is silent.
6. **If a second simultaneous player evicts the first**, that is the SNAT/UDP-conntrack theory becoming
real; it needs its own measured investigation, not a guess.
## 2026-09-29 (later still) — "You are not invited": the join gate is an empty allowlist. RakNet is confirmed working *by the refusal itself*
A Bedrock client reported **"Unable to connect / You are not invited to play on this server"**. This was
put to me as evidence that NetherNet signalling is working, that the refusal might have come from
Microsoft rather than our server, and that BDS players therefore never traverse this repo's
`crafty-l4-udp` path. **Measured, all three of those are wrong.** The refusal is ours, it came back over
`19132/udp`, and it has a one-line cause.
### Measured 21:56 (read-only)
| Property / file | Value |
|---|---|
| `allow-list` | **`true`** (line 37 of `server.properties`) |
| `allowlist.json` | **`[]`** — 3 bytes, mtime **20:51**, never edited since the server was created |
| `online-mode` | `true` |
| `transport` | **`raknet`** (line 71); `server.properties` mtime **still 21:33:31** |
| UDP 19132 | bound — `/proc/net/udp` `00000000:4ABC`, pid 47006 |
| **TCP 19132** | **absent** — `grep 4ABC /proc/net/tcp /proc/net/tcp6` → no match |
| `bedrock_server` | pid **47006**, etime 21:58 — the *same* process since 21:34, never restarted |
| `K3s-Bedrock.log_path` | **`''`** (empty string) in `crafty.sqlite`; `Dedicated_Server.txt` is **0 bytes** |
### What that settles
1. **`transport` is still `raknet`. Nobody reverted it and the binary did not rewrite it.** The file's
mtime is unchanged from the operator's 21:33:31 edit, `audit.log` records no later edit, and pid
47006 has been the same process since 21:34. **So NetherNet is NOT in use and cannot be what served
this client.** The premise that "something is evidently set to nethernet" is refuted.
2. **NetherNet could not have served it even in principle:** nethernet signals over a **TCP** socket on
`server-port`, and there is **no TCP 19132 listener in the pod at all**. No signalling endpoint
exists, and none is exposed by any Service.
3. **Therefore the refusal came from our server, over RakNet, over `19132/udp`, through the VIP.** A
refusal authored by the server requires the client to have reached the server; the only listener is
the RakNet UDP socket. **The 19132-19134/UDP exposure is the opposite of inert — it is the path that
carried both the earlier pong and this refusal.** *(Observed: the listeners, the missing TCP socket,
`transport=raknet`, the process identity. Reasoned from those: that the refusal traversed that path.)*
4. **The cause is trivial and certain: `allow-list=true` with `allowlist.json == []` refuses every
player.** The shipped `bedrock_server_how_to.html` states it — *"If the `allow-list` property is
enabled … the server will only allow selected users to connect. To allow a user to connect you need
to know their Xbox Live Gamertag."* An empty array selects nobody. **This configuration cannot admit
anyone, so the refusal needed no other explanation.**
### The correlation I was asked for, and why it cannot be done from disk
I was asked to correlate the client error against the BDS server log and `commander.log` at 21:40–22:00,
on the basis that our log showing nothing would mean the refusal came from Microsoft. **That inference
does not hold here, because there is no BDS log on disk to be silent.** ``K3s-Bedrock``'s `log_path` is
the empty string and `Dedicated_Server.txt` is 0 bytes: **Crafty persists no Bedrock stdout anywhere on
the PVC** — the console is panel-UI-only. `commander.log` 21:39–21:56 holds nothing but `homelab`
crash-detection lines and `raknet_ping` chatter: no join, no gamertag, no XUID, no allowlist denial.
That silence is evidence about Crafty's logging, **not** about the refusal's origin. **UNVERIFIED and
only observable live in the panel console.** The point stands: absence of a log here is not absence of
an event.
I also did **not** capture the player's packet (no `tcpdump`, no `cilium monitor`), so the flow's
identity is inferred from the listener topology, not seen. What removes the doubt is that the
configuration is *unconditionally* refusing: whichever path a client takes, `allow-list=true` plus `[]`
denies it. The operator action is required either way.
### Correction to the earlier entry's loose end, now firmer
The previous entries left "who sent those 26 packets / 27660 bytes on the `19132/UDP` policy map"
UNVERIFIED, and it stays UNVERIFIED — but note it was measured *before* 21:34, i.e. during the nethernet
window when **no UDP listener existed**. Those packets are consistent with clients RakNet-pinging a port
nothing was listening on. They should **not** be attributed to a successful Bedrock session, and equally
should not now be re-attributed to this refused client, which arrived later.
### Ranking after this measurement — unchanged at the top
**Keep `transport=raknet`, add the players to the allowlist. Still zero manifest change.** The allowlist
refusal is orthogonal to transport: it would refuse identically under NetherNet. Switching to NetherNet
would *add* manifest work (TCP 19132 in `crafty-l4-tcp` **and** `crafty-allow-ingress`, plus
`server-udp-ports` pinned and exposed) and would admit **not one extra player**. "Accept NetherNet" does
**not** outrank the status quo on this evidence, because NetherNet is not what is working — RakNet is.
Geyser stays at #2, reserve-only.
### What changed in this repo
`platform/crafty/README.md` — added *"You are not invited to play on this server — the join gate is the
allowlist, not the network"*: the measured table, the proof that the refusal is ours and rode the UDP
path, the "BDS console is not on disk" caveat, the gamertag/XUID identity model, the three operator
routes (`allowlist add` in the console; edit-then-`allowlist reload`; or `allow-list=false`, discouraged
because the game port is deliberately un-gated so the allowlist is the *only* join control), and a note
that `allowlist.json` is PVC state that resets to `[]` on a server recreate. Updated the options table
and human test 4 to put the allowlist check before blaming the network. **No manifest touched.**
### Still UNVERIFIED
- **No successful join.** Reachability and an authorization refusal are both proven; a completed login is
not. Only a human with a client, after an allowlist entry, can close it.
- **The exact client-side string was not observed by me** — it is the operator's report, and its mapping
to the allowlist denial is **reasoned** from the vendor doc, not read out of a BDS log (there is none).
- **Whether the client used Add-Server-by-IP or an Xbox/Friends route is unknown**, so a *concurrent*
Xbox-side "not invited" cannot be fully excluded. It does not matter operationally: the allowlist must
be fixed regardless, and after that a retry distinguishes the two immediately.
- **The multi-player SNAT question** remains open and untested.
- **Nothing was written.** `allowlist.json`, `server.properties` and the DB were read only; no
`allowlist add` was issued, no console command sent, no `kubectl` mutation, no `flux reconcile`.
### Operator actions — revised, allowlist first
1. **Add each player's Xbox Gamertag to the allowlist.** Crafty → `K3s-Bedrock` → Console:
`allowlist add <Gamertag>` (quote if it contains a space). Takes effect immediately, no restart.
Verify with `allowlist list`. This is **the** fix for "You are not invited".
2. **Then retry the join** from a Windows/mobile client: **Add Server** →
`crafty-mc.home.keithjackson.dev`, port **19132**. If it now connects, everything in these three
entries is closed. If it still refuses, capture the **panel console** output during the attempt —
that is the only place BDS's own reason is visible.
3. **Do not set `allow-list=false`** as the fix. The game port is deliberately un-gated at the network
layer, so the allowlist is the only thing deciding who may join. (And console `allowlist off` does not
change `server.properties`, so it silently reverts on restart.)
4. **Leave `transport=raknet` alone**, and keep ignoring both the `TRANSPORT TYPE ERROR` banner and the
`too few fields` warning — the server is reachable, proven by pong and by this refusal.
5. **`auto_start=0` / `crash_detection=0` on `K3s-Bedrock` — still outstanding.** `PATCH
/api/v2/servers/c5837301-12b1-404e-b50e-70eafb37226b`, both true, **read them back**.
6. **Treat `allowlist.json` and `transport=raknet` as backup-relevant PVC state.** A server recreate
resets the allowlist to `[]` and the transport to `nethernet` — both silently.
## 2026-09-29 (closing) — a player has joined. The chain is closed: `allow-list=true` + an empty allowlist was the wall, and nothing else was
The operator added one gamertag to the allowlist and the Bedrock client connected and joined. **The
three-entry investigation above is now closed by a working server**, with no manifest change, no
transport change, and no network change.
### What is now OBSERVED (moved out of UNVERIFIED)
- **A successful join.** *Operator-reported, 2026-09-29* — they added the player's gamertag and the
client connected and joined. **I did not watch the join**, and that distinction is kept deliberately;
this repo has been burned by collapsing it.
- **Corroborated directly on disk, though**, which is stronger than the report alone:
`allowlist.json` went `[]` → **one entry**, 75 bytes, and that entry's **`xuid` is populated**
(`keys=['ignoresPlayerLimit','name','xuid']`, mtime **22:03:17**). `audit.log` shows the operator wrote
only the name — `Edited file …/allowlist.json` at **22:02:13**. The vendor doc: `xuid` is *"Optional.
If it's not set then it will be populated when someone with a matching name **connects**."* So the XUID
was written **by the server, on connect**, ~64 s after the operator's edit. That is server-authored
evidence of an Xbox-Live-authenticated join. *(Observed: file state, both timestamps, the audit line.
Reasoned from the vendor doc: populated XUID ⇒ a connect happened.)*
- **The reasoned mapping is confirmed.** The client's *"You are not invited to play on this server"* was
mapped to the empty allowlist **by reasoning** from the vendor doc, because Crafty persists no BDS
stdout and there was no log to read it from. Adding **exactly one** gamertag and getting **exactly one**
successful join confirms that mapping. **`allow-list=true` + `allowlist.json == []` was the wall, and
nothing else was.**
- **`transport=raknet` survives a restart.** The fix involved one: `stop_server` 22:02:37 →
`start_server` 22:02:41, pid **47006 → 47875**. Afterwards `server.properties` mtime is **still
21:33:31** with `transport=raknet`, UDP `4ABC` is bound by the new pid, there is still **no TCP 4ABC**,
and `192.168.2.22:19132` answered a fresh pong at 22:06 (`MCPE;…;1.26.52;0;10;…`, new `server_guid`
matching the new pid, `players=0` — the player had left).
- **The route the operator used was file-edit + restart**, not the console `allowlist add` I put first:
`audit.log` edit 22:02:13 → stop → start. A restart is a valid substitute for `allowlist reload`.
Both routes now documented as proven; the console route is still preferred because it needs no restart.
### The recommendation is now PROVEN, not provisional
**Keep `transport=raknet` and enumerate players in the allowlist. Zero manifest change.** This is no
longer the best-reasoned option — it is the configuration a player has demonstrably joined over
`192.168.2.22:19132`. `service-l4.yaml`, `networkpolicy.yaml` and the `containerPort`s needed nothing.
Ranking otherwise unchanged: Geyser #2 (reserve, and it cannot share 19132), pinning BDS #3,
PocketMine/Nukkit #4, accept-NetherNet last — NetherNet is still **not in use**, would still need TCP
19132 exposed in two places, and would still admit not one extra player.
### Still open — NOT weakened by the success
- **`K3s-Bedrock` still has `auto_start=0` and `crash_detection=0`.** Re-measured from `crafty.sqlite` at
22:06, *after* the successful join and after the operator's 22:02:28 `modified the server` action —
which did **not** change them. **The operator now has a working server that will silently not come back
after any pod restart**, and merging `fix/crafty-oauth2-proxy-signin` **is** such a restart
(`strategy: Recreate`, RWO volume) and would stop **both** Minecraft servers. This matters *more* now,
not less: the failure mode has changed from "never worked" to "worked, then vanished".
- **The silent-reset exposure is unchanged, and now precisely characterised:**
a **restart** preserves `transport=raknet` and `allowlist.json` (observed above) but does not restart
the server itself (`auto_start=0`); a **server recreate or reinstall** resets `server.properties` to
`transport=nethernet` (nothing bound on 19132/udp) **and** `allowlist.json` to `[]` (everyone refused).
Neither reset announces itself, and the panel will show the server healthy and started. Both files are
PVC state Flux cannot restore. **Re-run the 5-step connect procedure after any recreate or BDS update.**
- **L2/L3 conformance is still uncertified.** `CLUSTER_AVAILABLE` is unset, so the gate's `conformance`
stage does not run; `R-DNS-01 R-NET-01 R-PLAT-02 R-STATE-01` remain UNVERIFIED in CI. A player joining
is not a conformance test.
- **Multi-player SNAT/UDP-conntrack is still untested.** Exactly **one** player has joined. Whether a
*second simultaneous* Bedrock player evicts the first behind the shared VIP and SNAT is unmeasured, and
a single successful session is precisely the evidence that does *not* settle it.
- **I still never saw BDS's own console output** — `log_path` is `''`, `Dedicated_Server.txt` is 0 bytes.
Anything BDS itself said about the join is visible only live in the panel.
- **Consoles still cannot reach this**, and LAN/Friends discovery still cannot work behind the VIP.
- **Geyser remains entirely unmeasured**; the pinnability of a BDS version remains UNVERIFIED.
### What changed in this repo
`platform/crafty/README.md` and `DECISIONS-LOG.md`. **No manifest touched, in any of the three passes.**
- Marked the join-gate section **RESOLVED and confirmed**, with the XUID-population evidence and the
operator-reported join labelled as exactly that.
- Promoted the decision heading and option-1 row from provisional to **PROVEN**, and recorded that the
operator's actual route was file-edit + restart (route 2), with route 1 still preferred.
- Rewrote Bedrock human-test 4 as a **5-step verified connect procedure**, each step with its distinct
failure signature, and stated plainly that the `TRANSPORT TYPE ERROR` banner and the `too few fields`
warning both appear on a fully working server — that pair misled two consecutive sessions.
- Sharpened the PVC warning to distinguish **restart** (preserves both files; server stays down because
`auto_start=0`) from **recreate/reinstall** (silently resets transport *and* allowlist).
### Operator actions — one is now urgent
1. **`PATCH /api/v2/servers/c5837301-12b1-404e-b50e-70eafb37226b`** setting `auto_start` **and**
`crash_detection` true, **and read them back**. Both are still `0`. This is the only thing standing
between the working server and it disappearing at the next pod restart — including the restart that
merging `fix/crafty-oauth2-proxy-signin` will cause.
2. **Leave `transport=raknet` and the allowlist alone.** Keep ignoring the `TRANSPORT TYPE ERROR` banner
and the `too few fields` warning.
3. **Add a gamertag per additional player** — console `allowlist add <Gamertag>` (no restart needed).
Do **not** switch to `allow-list=false`: the game port is un-gated by design, so the allowlist is the
only join control.
4. **After any Bedrock recreate or BDS update**, re-run the 5-step connect procedure — transport and
allowlist both reset silently.
5. **Before announcing this to more than one player**, note that only one simultaneous session has been
proven; if a second player evicts the first, that is the SNAT/conntrack question and needs its own
measured investigation.
+1
View File
@@ -17,6 +17,7 @@ resources:
- pihole-dns.yaml
- pihole-oauth2-proxy-credentials.sops.yaml
- pihole.yaml
- plex-claim-secret.sops.yaml
- plex.yaml
- qbittorrent-bootstrap-secret.sops.yaml
- qbittorrent-oauth2-proxy-credentials.sops.yaml
+77
View File
@@ -0,0 +1,77 @@
# Plex plex.tv server-claim token (WS-APPS). SOPS-encrypted (age).
# Consumed by catalog/plex.yaml as env PLEX_CLAIM via valueFrom.secretKeyRef.
#
# NOTE ON THESE COMMENTS: they carry NO dollar-brace expressions, deliberately. This file is applied
# by the apps-wave-0 Flux Kustomization, which runs envsubst over the built output, and kustomize
# preserves body comments into that stream (see conformance/tests/l0-static/flux-postbuild-
# substitution). A braced token quoted from the upstream shell script would be a live hazard here,
# not prose, so the upstream conditions below are described in words.
#
# THIS FILE NO LONGER HOLDS THE PLACEHOLDER. Corrected 2026-09-29: the server was CLAIMED on that
# date, and the value here is the SPENT token from that claim. It is inert and safe to leave --
# upstream gates the exchange on PlexOnlineToken being ABSENT from /config, and it is now present on
# the plex-config PVC, so this value is a NO-OP on every start whatever it holds. Deleting the Secret
# would NOT be safe (see below), which is why a spent token stays here rather than being blanked.
#
# The claim token is SHORT-LIVED -- minutes. Read the countdown that plex.tv/claim renders when it
# issues one; do NOT budget against a number written in this tree. Earlier revisions of this comment
# asserted a "~4-minute TTL" as established fact with no citation, and upstream plexinc/pms-docker
# documents no expiry at all. That uncited number was load-bearing for the whole procedure design.
#
# This file exists to make the Secret — and therefore the env reference — ALWAYS RESOLVABLE IN THE
# FLUX GRAPH, not to carry a working token. Re-claiming (only needed after a /config wipe) is
# scripted: infra/plex/claim-ship.sh, measured at ~2m45s end to end.
#
# WHY SHIPPING A PLACEHOLDER IS SAFE (verified against plexinc/pms-docker branch `master`, the
# s6-overlay v2 script root/etc/cont-init.d/40-plex-first-run, cross-checked byte-identical against
# the same file extracted from the running `plexinc/pms-docker:latest` image):
# - line 106 guards the claim exchange on TWO conditions: PLEX_CLAIM non-empty AND the
# PlexOnlineToken pref read out of /config empty. Once the server is claimed that block is
# skipped entirely, so whatever this Secret holds is a NO-OP on every later start. Upstream
# README: "If server is already logged in, this parameter is ignored."
# - on a still-UNCLAIMED /config, an invalid or expired token makes plex.tv answer 403 with an
# empty body. curl's exit status is never checked, the sed for the authentication-token element
# yields the empty string, and the success branch at line 120 simply does not fire. There is no
# else branch and no non-zero exit; the script has no `set -e` and ends by touching
# /.firstRunComplete and echoing, so it exits 0. The server comes up UNCLAIMED and serving.
# It does NOT crashloop.
# - the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS, whose default is 0 ("continue silently even
# if a cont-init script failed"), so even a hypothetical non-zero exit would not halt startup.
# A MISSING Secret would NOT be safe: the ServiceDefinition schema's valueFrom.secretKeyRef shape
# has no `optional` field, so an absent Secret is a hard CreateContainerConfigError. That is exactly
# why this file is committed and listed in apps/wave-0/kustomization.yaml instead of being created
# by hand at claim time.
#
# INJECTION / ROTATION (operator, not agents) — runbooks/plex-claim.md has the timed procedure:
# infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
# Never decrypt this file into a shell, a log, or a shared terminal.
apiVersion: v1
kind: Secret
metadata:
name: plex-claim
namespace: plex
labels:
app.kubernetes.io/name: plex
type: Opaque
stringData:
PLEX_CLAIM: ENC[AES256_GCM,data:nLmzeUBWRhRjUakrGiGzQpQuqZNcIYvRH6M=,iv:z0on/dArsNejNLrp3V1nSiNLFzKcREhPuMO1634CwQw=,tag:Ih23CT1JYna/5lfYQUb1Dw==,type:str]
sops:
kms: []
gcp_kms: []
azure_kv: []
hc_vault: []
age:
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2dWtqOVNQalZ2QlRxOXJa
cDc1MTdtWENkdWIxS1JId3dIU1JDSmk0ODNrCk1YUzhFWXR3bUxyV2NNcmFOakdz
THl1Q284dFJxTy9Rd1h0eXJJaVZtdmcKLS0tIERuN0ZFeW9zdi9aNFRYNWxmUS9s
QUtYdHEvN042ajNoNktOLzlYQmQrc1kK4NHozMbXVzNGYbUBRbK6a9Uw/sVPAfe6
RW5KWzZ4I/hloqjN3zzVISoSMieMkBGDnF5qZS5N6FqIc6/HrsDD8A==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-09-29T06:31:16Z"
mac: ENC[AES256_GCM,data:62BmBAq45CD2g6LDyx+Ho4bTfrhJcaro/RB/iM+OaRc4a/D83MhjbcBL11Uz/O726inqhkvCKDJ/fFemoc50JDbtC79cYMaxrGJNjFM0sdckWrowPvh6ie7HHHUBRBaefl+gNfnc+pbvcncvRzwA4RyzcZse1wbxKfHN8FjEopE=,iv:h0cMShs3tIgnY6eruUYymYrcLFzQZLk/uXQJ/Jsd5ek=,tag:nZ9Pau7p4xWT1QCN1vv2vA==,type:str]
pgp: []
encrypted_regex: ^(data|stringData)$
version: 3.9.4
+5
View File
@@ -132,6 +132,11 @@ spec:
value: Etc/UTC
- name: ALLOWED_NETWORKS
value: 10.0.0.0/8,192.168.0.0/16
- name: PLEX_CLAIM
valueFrom:
secretKeyRef:
name: plex-claim
key: PLEX_CLAIM
volumeMounts:
- name: config
mountPath: /config
@@ -4,7 +4,13 @@
# against Gitea. Flux decrypts it via the sops-age key at reconcile (apps Kustomization
# decryption is wired in clusters/dev/kustomization.yaml). Rotate by resetting the token
# in Gitea (admin -> actions -> runners) and re-encrypting a fresh value here.
# Recover/inspect: infra/devcontainer/dev.sh run sops -d apps/gitea-actions-runner-secret.sops.yaml
# ROTATE: ./infra/gitea/runner-token-rotate.sh -- it mints a fresh instance-level token from the
# Gitea API and writes it here via `sops set`, without the value ever being printed.
# Do NOT `sops -d` this file into a terminal, a log, or an agent transcript. To check only that
# it still decrypts, redirect: `sops -d <file> > /dev/null`.
# NOTE: a fresh token alone may not re-register the runner. act_runner registers ONCE and keeps
# its identity in /data/.runner on the act-runner-data PVC; if Gitea no longer holds a matching
# row it skips registration silently. Clearing that PVC is what forces a real re-registration.
apiVersion: v1
kind: Secret
metadata:
+69 -106
View File
@@ -99,70 +99,79 @@ spec:
# CI holds no cluster credentials — do not mount a SA token into the runner or
# (via the shared netns) the job containers (R-SEC-02, docs/14).
automountServiceAccountToken: false
# ---- Docker-in-Docker: the job executor daemon (privileged) -------------------
#
# NATIVE SIDECAR (k3s v1.36; sidecar containers are GA). It is an initContainer with
# `restartPolicy: Always`, which means: it starts BEFORE the regular containers, keeps
# running alongside them, and is terminated after them. kubelet holds `runner` until
# this container's startupProbe passes, so the ordering is enforced by the kubelet
# instead of hand-rolled shell in the runner.
#
# This REPLACES two bounded wait-loops that used to live in the runner's `command:`.
# Deleting them also deletes an envsubst hazard: clusters/base/apps-wave-1.yaml runs
# postBuild substitution over this ENTIRE manifest, comments included, and those loops
# depended on bare dollar-names surviving it untouched. There is now no shell here at
# all, so there is nothing to escape or mis-escape.
#
# WHY THE PROBE IS `exec docker ... info` AND NOT `tcpSocket`. A tcpSocket probe would
# re-open the exact gap the old Stage 1 existed to close: dockerd writes the TLS certs
# and only THEN begins listening, so the port can accept a connection while the certs
# are absent or incomplete -- and act_runner hard-exits on a missing
# /certs/client/ca.pem. A full TLS-verified `docker info` proves BOTH that the certs
# are usable and that the daemon answers.
#
# CORRECTION (measured 2026-09-29, after this conversion shipped): the commit that
# introduced this claimed the conversion would REMOVE the one-per-start dockerd line
# `http: TLS handshake error from 127.0.0.1: EOF`. IT DOES NOT. Exactly one still
# appears, immediately after "API listen on [::]:2376" -- it is now this probe's FIRST
# attempt, landing microseconds after dockerd begins listening and cut off by
# timeoutSeconds: 1. The next poll succeeds. It is cosmetic, exactly as cosmetic as it
# was before, and deliberately NOT chased: raising timeoutSeconds to hide one benign
# startup line would slow every subsequent poll. Left documented rather than silenced.
#
# periodSeconds 1 x failureThreshold 60 preserves the previous 60s cap.
initContainers:
- name: dind
restartPolicy: Always # <- this field is what makes it a SIDECAR
image: docker:27-dind
securityContext:
privileged: true
env:
# Generate + publish TLS certs into the shared emptyDir; dockerd then listens
# on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
- name: DOCKER_TLS_CERTDIR
value: /certs
volumeMounts:
- name: certs
mountPath: /certs
- name: dind-storage
mountPath: /var/lib/docker
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
memory: 2Gi
startupProbe:
exec:
command:
- docker
- --host=tcp://127.0.0.1:2376
- --tlsverify
- --tlscacert=/certs/client/ca.pem
- --tlscert=/certs/client/cert.pem
- --tlskey=/certs/client/key.pem
- info
periodSeconds: 1
failureThreshold: 60
containers:
# ---- act_runner: registers with Gitea, dispatches jobs to the DinD daemon ----
- name: runner
image: gitea/act_runner:0.2.11
# TWO-STAGE BOUNDED WAIT on the DinD sidecar, then exec the image's normal
# entrypoint (/sbin/tini -- /opt/act/run.sh), which registers-from-env on first
# boot and runs `act_runner daemon`.
#
# STAGE 1 — the certs must EXIST: act_runner reads /certs/client/ca.pem at startup
# and hard-exits if it is missing (a startup race against the sidecar).
# STAGE 2 — the daemon must actually ACCEPT a connection: the cert FILE appearing is
# necessary but NOT sufficient. dockerd writes the certs and only THEN begins
# listening on :2376, so a runner that proceeds on the file alone still hard-exits
# with "cannot ping the docker daemon at tcp://localhost:2376" — observed
# crashlooping 11 times in ~8 minutes on a fresh pod after a node bounce.
# Both loops are bounded (60 x 1s) and exit 1 on timeout, so a genuinely dead sidecar
# surfaces as a CrashLoopBackOff with a clear message instead of hanging forever.
#
# ENVSUBST SAFETY — the BARE $i / $j below are CORRECT. Do not "tidy" them into the
# braced form or the double-dollar form; both are wrong here, for different reasons.
# clusters/base/apps-wave-1.yaml carries postBuild.substituteFrom cluster-vars, so
# Flux envsubst runs over this ENTIRE manifest — YAML comments very much included.
# Its contract (drone/envsubst; the same one spelled out in clusters/base/apps-vms.yaml,
# and re-verified against this exact file with `flux envsubst`, flux v2.4.0) is:
# - A bare $name is NOT a reference to envsubst: it passes through untouched, and
# that holds even when the name IS set in cluster-vars. So $i and $j reach the
# cluster verbatim and both 60s bounds below are real, enforced bounds.
# - $((i + 1)) is likewise untouched: "(" cannot begin a variable name.
# - A BRACED dollar-name IS a reference and gets replaced — with the empty string
# when the name is unset. One of those here would make a bound read [ "" -ge 60 ],
# which errors, short-circuits the &&, and silently turns the wait UNBOUNDED.
# - A DOUBLE-dollar braced name is envsubst's escape and survives as the braced
# form — still wrong for us, because on any path where envsubst does NOT run
# (a plain `kustomize build` or `kubectl apply -k` of this directory) the shell
# reads that leading double-dollar as its own PID, so a bound would compare a
# string like 1234{i} against 60. The bare form is the only spelling that is
# correct on BOTH paths.
# - A braced name containing punctuation is a HARD ERROR ("unable to parse variable
# name") that fails this whole Kustomization — and this Kustomization gates every
# merge in the repo. So never write one anywhere in this file, INCLUDING in a
# comment. That is why this note names the three forms in words rather than
# showing them: an earlier draft of it spelled them out literally and broke
# `flux envsubst` on the file. (Same trap the indexed-expansion warning in
# clusters/base/apps-vms.yaml describes.)
command:
- /bin/sh
- -c
- |
# Stage 1: the sidecar's TLS client certs must appear in the shared emptyDir.
i=0
until [ -f /certs/client/ca.pem ]; do
i=$((i + 1))
[ "$i" -ge 60 ] && { echo "dind certs never appeared" >&2; exit 1; }
echo "waiting for dind TLS certs ($i)"; sleep 1
done
# Stage 2: dockerd must actually accept a connection on :2376. busybox nc in this
# Alpine-based image (gitea/act_runner:0.2.11) supports -z (scan) and -w (timeout).
j=0
until nc -z -w1 127.0.0.1 2376; do
j=$((j + 1))
[ "$j" -ge 60 ] && { echo "dind daemon never accepted a connection on 2376" >&2; exit 1; }
echo "waiting for dind daemon on :2376 ($j)"; sleep 1
done
exec /sbin/tini -- /opt/act/run.sh
# No `command:` override. The image's own entrypoint (/sbin/tini -- /opt/act/run.sh)
# registers from env on first boot and runs `act_runner daemon`. The two wait-loops
# that used to live here are gone: kubelet now holds this container until the dind
# sidecar's startupProbe passes (see initContainers above).
env:
- name: GITEA_INSTANCE_URL
value: https://gitea.${BASE_DOMAIN}
@@ -198,52 +207,6 @@ spec:
memory: 128Mi
limits:
memory: 512Mi
# ---- Docker-in-Docker: the job executor daemon (privileged) ----
#
# FOLLOW-UP (deliberately NOT bundled into the readiness fix above — own change):
# Convert this container into a NATIVE SIDECAR and delete both shell wait-loops.
# k3s is now v1.36, where sidecar containers are GA, so kubelet can enforce the
# ordering that the loops above hand-roll — which also removes the envsubst hazard
# from this file entirely (no shell, no $i/$j, nothing to escape or mis-escape).
# Shape to build:
# 1. Move this whole container into `initContainers:` as the FIRST entry.
# 2. Add `restartPolicy: Always` to it. That single field is what makes an
# initContainer a sidecar: it starts before the regular containers, keeps
# running alongside them, and is terminated after them.
# 3. Give it a `startupProbe` that succeeds only when dockerd accepts a TLS
# connection — e.g. `exec` a `docker -H tcp://127.0.0.1:2376 --tlsverify
# --tlscacert /certs/client/ca.pem --tlscert /certs/client/cert.pem
# --tlskey /certs/client/key.pem info`. Do NOT use a bare `tcpSocket` probe:
# it would re-open the same "port accepting but TLS/certs not ready" gap that
# Stage 1 above exists to close. `periodSeconds: 1` + `failureThreshold: 60`
# preserves today's 60s cap.
# 4. kubelet then holds the `runner` container until that startupProbe passes, so
# `runner` can drop its `command:` override completely and run the image's
# default entrypoint.
# Why it is a separate change: this Deployment is the ENFORCED MERGE GATE for this
# entire repository. If the conversion is wrong, nothing can merge — so it must land
# alone, where the failure is unambiguous and not entangled with anything else, and
# only after a real workflow run has been observed green on the converted pod.
- name: dind
image: docker:27-dind
securityContext:
privileged: true
env:
# Generate + publish TLS certs into the shared emptyDir; dockerd then listens
# on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
- name: DOCKER_TLS_CERTDIR
value: /certs
volumeMounts:
- name: certs
mountPath: /certs
- name: dind-storage
mountPath: /var/lib/docker
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
memory: 2Gi
volumes:
- name: data
persistentVolumeClaim:
+46 -9
View File
@@ -32,11 +32,46 @@ env:
# (10.43.0.0/16); 192.168.0.0/16 covers the home LAN.
- name: ALLOWED_NETWORKS
value: 10.0.0.0/8,192.168.0.0/16
# PLEX_CLAIM is deliberately NOT set here: a https://plex.tv/claim token has a ~4-min TTL, far
# shorter than the commit -> push -> Flux reconcile -> pod-roll cycle, so any token committed to git
# is already expired by the time the pod starts. Claiming is a one-time LIVE step (UI sign-in, or a
# single API POST) against the running server; the claim then persists on the /config PVC and
# survives pod rolls (including the roll this env change causes), so it never belongs in git.
# PLEX_CLAIM — the ONE-SHOT plex.tv server claim, sourced from the SOPS Secret `plex-claim`
# (apps/wave-0/plex-claim-secret.sops.yaml, same wave => applied in the same reconcile as this
# workload). Plex cannot create libraries over the mounted bulk share while the server is
# UNCLAIMED, so this is wired; it was previously left out on the reasoning below, and only one
# half of that reasoning survived contact with the upstream source.
#
# A SHORT TTL IS THE OPERATIONAL CONSTRAINT, not a reason to abstain. A https://plex.tv/claim token
# expires in MINUTES — read the countdown the claim page renders; do not budget against a number
# written in this tree. (This comment previously asserted "~4 minutes" as established fact. It was
# UNCITED, upstream plexinc/pms-docker documents no expiry at all, and that number was load-bearing
# for the whole procedure design — including a since-disproven claim that the delivery pipeline was
# too slow to fit inside it. Measured 2026-09-29: the full loop is ~2m45s and fits.)
#
# The live token is injected by the operator inside that window, with an explicit `flux reconcile`
# to collapse the wait. Scripted end to end as infra/plex/claim-ship.sh; runbooks/plex-claim.md is
# the procedure. SERVER CLAIMED 2026-09-29 — this env is now a permanent no-op (see below), and the
# committed value is the spent token from that claim rather than the original placeholder.
#
# ONE-SHOT, AND SAFE TO SIT HERE EXPIRED. Verified from plexinc/pms-docker's own first-run script
# (branch `master`, root/etc/cont-init.d/40-plex-first-run, confirmed byte-identical to the file in
# the running `:latest` image), NOT assumed:
# - the exchange (line 106) is guarded on TWO conditions: PLEX_CLAIM non-empty AND the
# PlexOnlineToken pref in /config empty. Once claimed, the block is SKIPPED — this env is a
# no-op on every subsequent start, whatever it holds. (Upstream README: "If server is already
# logged in, this parameter is ignored.") The claim itself persists on the /config PVC as before.
# - an invalid/expired token on a still-unclaimed /config gets a 403 with an empty body; curl's
# status is unchecked, the token `sed` yields "", the success branch at line 120 does not fire,
# there is no `else` and no `exit 1`, and the script (no `set -e`) exits 0. It comes up UNCLAIMED and
# SERVING — it does NOT crashloop. The image also leaves S6_BEHAVIOUR_IF_STAGE2_FAILS at its
# default 0, which continues startup even when a cont-init script fails.
# So the earlier "a placeholder claim Secret would crashloop the pod" note was WRONG about the
# placeholder; it was right only about an ABSENT Secret. secretKeyRef here has no `optional` field,
# so the Secret must always exist in the Flux graph — which is why it is committed, encrypted, and
# listed in apps/wave-0/kustomization.yaml rather than hand-created at claim time. No schema change
# is needed: `optional` would only matter if a missing Secret had to be tolerated, and it doesn't.
- name: PLEX_CLAIM
valueFrom:
secretKeyRef:
name: plex-claim
key: PLEX_CLAIM
# ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex
# which URL to advertise to plex.tv for REMOTE clients. The only candidate is
# https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL
@@ -56,10 +91,12 @@ storage:
# container's EPHEMERAL layer -> unclaimed + all state lost on restart. The image inits as
# root (s6-svscan PID 1), chowns /config to the plex user, then drops -> no fsGroup needed
# (verified in-cluster: PID1=root, "Plex Media Server"=plex). Persisting /config here also
# makes the claim survive restarts once claimed. To auto-claim on first boot instead of the
# UI sign-in, the operator adds a PLEX_CLAIM env from a fresh https://plex.tv/claim token
# (4-min TTL) — deliberately NOT wired here: the env secretKeyRef pattern has no `optional`,
# so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
# makes the claim survive restarts once claimed — which is what makes the PLEX_CLAIM env
# above a ONE-SHOT: the token is exchanged for a PlexOnlineToken written here, and the
# exchange is then skipped on every later start. (The older note here claimed a
# placeholder claim Secret "would crashloop the pod"; upstream's first-run script says
# otherwise — see the PLEX_CLAIM comment in `env` above. An ABSENT Secret is the hard
# failure, and that is why one is committed.)
# THE LIBRARY (docs/04 bulk tier). One READ-ONLY mount of the export ROOT, giving /media/movies,
# /media/tv and /media/photos — the whole tree the layout actually has (`ls /mnt/Storage/plex`:
# downloads/ movies/ photos/ tv/).
+15 -1
View File
@@ -1844,6 +1844,10 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
convention:
- `<svc>.yaml` -> wave(svc)
- `<svc>-bootstrap-secret.sops.yaml` -> wave(svc) (consumed by svc's env, same apply)
- `<svc>-claim-secret.sops.yaml` -> wave(svc) (a one-shot upstream-service claim token
consumed by svc's env — plex's plex.tv claim. Same wave as the app for the same reason the
bootstrap secret is: an env `secretKeyRef` has no `optional`, so the Secret must be applied
in the SAME reconcile as the workload or the pod is CreateContainerConfigError.)
- `<svc>-oauth2-proxy-credentials.sops.yaml` -> wave(svc) (forward-auth sidecar OIDC creds)
- `<svc>-db.yaml` -> wave(svc-db) (a strictly-lower wave; CNPG DB)
- `<svc>-db-app.sops.yaml` -> wave(svc-db) (CNPG initdb owner secret)
@@ -1870,6 +1874,11 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
w = waves[name]
add(w, f"{name}.yaml")
add(w, f"{name}-bootstrap-secret.sops.yaml")
# A one-shot claim token for an UPSTREAM service (plex's plex.tv claim). Same wave as the
# app: it is read by the app's env at pod start, and an env secretKeyRef has no `optional`,
# so a Secret applied in a later wave would leave the pod in CreateContainerConfigError.
# Placed only when the file exists, so services without one are unaffected.
add(w, f"{name}-claim-secret.sops.yaml")
# forward-auth (Option B): the per-app oauth2-proxy sidecar's OIDC client + cookie secret.
# Same wave as the app (the sidecar reads it at pod start); placed only when the file exists,
# so non-forward-auth apps are unaffected. Mirrors the bootstrap-secret naming convention.
@@ -2548,7 +2557,8 @@ def _selftest() -> int:
# 7a) plan_apps_waves: sidecars inherit their unit's wave; not-onboarded SDs are skipped;
# EXTRA_UNITS anchor strictly above; a file absent from the tree is never placed.
present = {
"a.yaml", "svc.yaml", "svc-bootstrap-secret.sops.yaml",
"a.yaml", "a-claim-secret.sops.yaml",
"svc.yaml", "svc-bootstrap-secret.sops.yaml",
"svc-db.yaml", "svc-db-app.sops.yaml", "svc-db-backup-credentials.sops.yaml",
"svc-db-backup-credentials-prod.sops.yaml",
"gitea.yaml",
@@ -2559,6 +2569,10 @@ def _selftest() -> int:
plan = plan_apps_waves(plan_sds, present)
flat = {f: wv for wv, fs in plan.items() for f in fs}
check(flat.get("a.yaml") == 0, "leaf app must land in wave 0")
check(flat.get("a-claim-secret.sops.yaml") == 0,
"a `<svc>-claim-secret.sops.yaml` (plex's plex.tv claim) must be placed in the SAME wave "
"as its app — an env secretKeyRef has no `optional`, so a later wave means "
"CreateContainerConfigError")
check(flat.get("svc-db.yaml") == 0 and flat.get("svc-db-app.sops.yaml") == 0
and flat.get("svc-db-backup-credentials.sops.yaml") == 0
and flat.get("svc-db-backup-credentials-prod.sops.yaml") == 0,
+7 -1
View File
@@ -12,7 +12,7 @@ CI_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
.DEFAULT_GOAL := help
.PHONY: help spec-immutability lint schema l0 render-golden l1 conformance gate all touched versions tools test
.PHONY: help spec-immutability decisions-log lint schema l0 render-golden l1 conformance gate all touched versions tools test
help: ## Show this help
@awk 'BEGIN{FS=":.*##"} /^[a-zA-Z0-9_-]+:.*##/{printf " \033[36m%-14s\033[0m %s\n",$$1,$$2}' $(MAKEFILE_LIST)
@@ -29,9 +29,15 @@ schema l0: ## ServiceDefinition schema validation + Kyverno L0 policy gate (no c
render-golden: ## render==golden: canonical render reproduces committed apps/ (no cluster; live now)
@"$(CI_DIR)render-golden.sh"
decisions-log: ## handoff log: DECISIONS-LOG.md frozen, decisions/ entries well-formed (no cluster)
@"$(CI_DIR)decisions-log.sh"
l1: ## L1 rendering checks (no cluster; live now)
@"$(CI_DIR)l1.sh"
gitea-settings: ## Gitea branch protection + runners match infra/gitea/expected-branch-protection.json (SKIPS w/o GITEA_SETTINGS_TOKEN)
@./gitea-settings.sh
conformance: ## L2/L3 in-cluster Chainsaw Job (SKIPS w/o CLUSTER_AVAILABLE=true)
@"$(CI_DIR)conformance.sh"
+15
View File
@@ -1,5 +1,19 @@
# `ci/` — portable merge gate (runnable today, without Gitea)
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
> This document speaks as though no cluster exists and the platform is scaffolding. **A
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
> coverage hole, not a phase.
>
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
> documentation rather than measurement, and `main` was unprotected with zero runners
> registered the whole time. **Query the live API before asserting platform state.**
**Task T-0.2 · R-TEST-02 (reproducible, single-command) · Owner WS-GITOPS (CI).**
The same merge gate the Gitea Actions workflow runs
@@ -87,6 +101,7 @@ caller that must distinguish them invokes `./ci/gate.sh` directly.
|---|---|---|
| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is installed for us by `.gitea/workflows/scripts/install-tools.sh`, so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
| `spec-immutability` (R-TEST-06) | the sibling spec repo is not checked out, so the spec **could not be verified** | **UNDETERMINED** — unverified is unverified. Deliberately *not* upgraded to BLOCKED by strict skips: the live runner does not check the spec repo out at all, so blocking would leave the only enforced status check permanently red for a reason no PR could clear. It is therefore the **one** skip whose non-answer a caller may knowingly tolerate, and `.gitea/workflows/gate.yml` does so in one visible line. `REQUIRE_SPEC_VERIFICATION=true` makes it a hard block once that checkout is wired. Locally: check the spec repo out beside this one, or set `SPEC_REPO` (`dev.sh` mounts a sibling automatically). |
| `decisions-log` | **nothing** — this stage has no skip path. It reads only files in the tree under test and needs no tool and no cluster, so there is no environment in which it "could not run" | a reported skip is a **defect in the stage** and BLOCKS. The stage fails when the frozen `DECISIONS-LOG.md` is modified (which silently re-creates, for every *other* open branch, the tail-hunk conflict that made four of five PRs unmergeable on 2026-09-29) or when a `decisions/` entry is malformed. There is deliberately **no** `REQUIRE_DECISIONS_LOG` knob: a guard with an off switch is not a guard. See `decisions/README.md`. |
| `conformance` (L2/L3) | `CLUSTER_AVAILABLE!=true` — the environment **declared** the capability absent (documented Phase-0 behaviour) | **does not deny a PASS**: a declared-inapplicable stage was never *supposed* to run. It is NAMED in the PASS headline and step summary, the touched requirements it would have covered are reported UNVERIFIED, and `REQUIRE_CLUSTER_TESTS=true` (Phase 1+) turns that into a block. A skip **despite** `CLUSTER_AVAILABLE=true` is a defect and BLOCKS. |
That asymmetry is the whole rule: **declared-not-applicable may pass; could-not-run may not.**
+40
View File
@@ -61,5 +61,45 @@ if [ -d "${CONF_ART}/coverage" ]; then
cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
fi
# ---------------------------------------------------------------------------------------
# DID IT ACTUALLY RUN? -- the guard without which this stage FAKES A GREEN.
#
# run.sh exits 0 in "scaffold mode" when it cannot reach a cluster: it emits a JUnit file
# whose every testcase is <skipped> and returns success. That is correct FOR run.sh -- an
# honest skip is its documented no-cluster behaviour. It is catastrophic HERE, because
# reaching this point means the environment DECLARED CLUSTER_AVAILABLE=true, and this stage
# would otherwise print "L2/L3 conformance passed" over 54 tests that never executed.
#
# MEASURED 2026-09-29, before this guard existed:
# CLUSTER_AVAILABLE=true ./ci/conformance.sh -> rc=0
# "[junit] wrote skipped JUnit for 54 scaffolded test(s)"
# "[ ok ] L2/L3 conformance passed for touched subset"
# One Gitea variable away from a fake green on an enforcing merge gate. That is exactly the
# HARD RULE 4 violation this repo's whole gate design exists to prevent, and nothing caught
# it because nobody had ever set the variable.
#
# A skip while a cluster is DECLARED is a contradiction: either the declaration is wrong or
# the cluster is unreachable. The merge-gate decision matrix already says this case is BLOCK
# ("conformance skipped while CLUSTER_AVAILABLE=true -> BLOCK (it should have run)"). This
# makes the stage itself honour that instead of reporting a pass.
JUNIT="${CONF_ART}/report.junit.xml"
if [ -f "$JUNIT" ]; then
total="$(grep -o '<testcase' "$JUNIT" | wc -l | tr -d ' ')"
skipped="$(grep -o '<skipped' "$JUNIT" | wc -l | tr -d ' ')"
if [ "${total:-0}" -gt 0 ] && [ "${skipped:-0}" -ge "${total:-0}" ]; then
err "CLUSTER_AVAILABLE=true, but the L2/L3 suite SKIPPED every one of its ${total} tests"
err " (scaffold mode -- run.sh could not reach a cluster and exited 0)."
err " This stage will NOT report that as a pass. Either the cluster is unreachable from"
err " this runner, or CLUSTER_AVAILABLE is set when it should not be."
err " NOTE: the in-cluster trigger path is still UNIMPLEMENTED -- see"
err " docs/plans/l2-l3-conformance-ci-wiring.md. Setting CLUSTER_AVAILABLE=true does not"
err " by itself make L2/L3 run; it only makes this stage attempt a LOCAL run that the CI"
err " runner has no credential for (automountServiceAccountToken: false, by design)."
summary "- conformance-l2-l3: **BLOCKED — declared available but every test skipped** (not a pass)"
exit 1
fi
log "L2/L3 executed: ${total} testcase(s), ${skipped} skipped"
fi
ok "L2/L3 conformance passed for touched subset"
summary "- conformance-l2-l3: **passed**"
+195
View File
@@ -0,0 +1,195 @@
#!/usr/bin/env bash
# ci/decisions-log.sh — the handoff log is a DIRECTORY, and the old single file is frozen.
#
# WHY THIS STAGE EXISTS
# ---------------------
# CLAUDE.md §7 tells every agent to record a handoff before finishing. Until 2026-09-29 that
# meant appending to the tail of one 5,228-line file, DECISIONS-LOG.md. Every branch appended
# to the SAME last hunk, so git could not auto-merge any two of them. Measured that day against
# the live Gitea: four of the five open PRs were `mergeable: false`, and `git merge-tree`
# against `main` named DECISIONS-LOG.md as the ONLY conflicting path in all four:
#
# #22 fix/crafty-oauth2-proxy-signin CONFLICT (content): DECISIONS-LOG.md
# #20 feat/conformance-job-template CONFLICT (content): DECISIONS-LOG.md
# #19 feat/conformance-trigger-kubeconfig CONFLICT (content): DECISIONS-LOG.md
# #18 feat/conformance-runner-image CONFLICT (content): DECISIONS-LOG.md
# #17 feat/ci-is-the-approver clean — it wrote no handoff entry
#
# Nothing was wrong with any of those changes. The log's storage format serialised work that
# was otherwise parallel, and the cost grew with every concurrent branch (N branches -> N-1
# rebases, each invalidating the gate run before it).
#
# The fix is that handoffs now live one-per-file under decisions/, so concurrent branches touch
# disjoint paths and there is nothing to merge. See decisions/README.md.
#
# WHY IT NEEDS A CHECK AND NOT JUST A DOCUMENT
# --------------------------------------------
# The old habit is written into this repo in many places — the runbooks, the muscle memory of
# every agent that has read the log, and (until this change) CLAUDE.md itself. An agent that
# appends to DECISIONS-LOG.md anyway does not fail: it produces a perfectly good-looking commit
# that quietly re-creates the conflict for every other open branch, and the damage is only
# visible later, to somebody else, in a PR that is not theirs. That is the exact shape this
# repo keeps getting burned by — a silent regression with a delayed, misattributed symptom — so
# it gets a guard rather than a paragraph.
#
# WHAT IS ASSERTED
# (1) DECISIONS-LOG.md is byte-identical to the SHA-256 pinned in decisions/ARCHIVE.sha256.
# Re-pinning is a deliberate, reviewable line in a diff; appending is not.
# (2) Every decisions/ entry is named YYYY-MM-DD-<slug>.md with a real calendar date, so the
# directory sorts chronologically and two agents cannot collide on a filename.
# (3) Every entry carries frontmatter (date, branch, title) whose `date` agrees with its
# filename — a disagreement means a copied template nobody edited.
# (4) Every entry has all three of CLAUDE.md §7's sections with a real body under each:
# "What changed", "What I verified", "What I did NOT verify". This is the quality floor
# that stops an entry degrading into "done ✅". The third heading is the load-bearing one:
# this repo's recurring failure is a confident report of something nobody measured, and an
# entry with nowhere to write "UNVERIFIED" does not get one written.
#
# EXIT CODES (ci/README.md contract): 0 pass, 1 BLOCKS the merge. This stage has NO skip path —
# it reads only files that are in the tree it is run against, needs no tool beyond coreutils and
# no cluster, so there is no environment in which it "could not run". Adding an rc 3 here would
# add a fail-open surface to a guard whose whole job is to close one.
set -uo pipefail
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck disable=SC1091
source "${CI_DIR}/lib.sh"
set +e # aggregate every problem in one run; do not stop at the first.
cd "$REPO_ROOT"
ARCHIVE="DECISIONS-LOG.md"
PIN="decisions/ARCHIVE.sha256"
rc=0
# ---- (1) the archive is frozen ----------------------------------------------
log "Stage: decisions log (archive frozen + entries well-formed)"
if [ ! -f "$ARCHIVE" ]; then
err "${ARCHIVE} is missing. It is the frozen pre-2026-09-29 history and must stay in the tree."
summary "- decisions-log: **FAILED** (\`${ARCHIVE}\` missing)"
rc=1
elif [ ! -f "$PIN" ]; then
err "${PIN} is missing, so the archive's integrity cannot be established."
err " If you are re-pinning deliberately: sha256sum ${ARCHIVE} > ${PIN}"
summary "- decisions-log: **FAILED** (\`${PIN}\` missing — archive unverifiable)"
rc=1
else
want="$(awk 'NR==1{print $1}' "$PIN")"
got="$(sha256sum "$ARCHIVE" | awk '{print $1}')"
if [ "$want" != "$got" ]; then
err "${ARCHIVE} HAS CHANGED. It is a FROZEN ARCHIVE — do not append to it."
err " pinned: ${want}"
err " on disk: ${got}"
err ""
err " Every branch that appends to this file's tail conflicts with every other branch that"
err " does, on the same hunk. That is what made four of five open PRs unmergeable on"
err " 2026-09-29, and your commit would re-create it for branches that are not yours."
err ""
err " Write your handoff as its OWN FILE instead:"
err " decisions/\$(date +%F)-<your-branch-slug>.md (see decisions/README.md)"
err " and restore the archive: git checkout -- ${ARCHIVE}"
err ""
err " If the archive genuinely must change (redacting a leaked credential, say), that is a"
err " reviewed act: re-pin it with 'sha256sum ${ARCHIVE} > ${PIN}' and say why in your entry."
summary "- decisions-log: **FAILED — the frozen archive \`${ARCHIVE}\` was modified** (append to \`decisions/\` instead)"
rc=1
else
ok "${ARCHIVE}: unchanged (sha256 matches ${PIN})"
fi
fi
# ---- (2)(3)(4) every entry is well-formed ------------------------------------
shopt -s nullglob
entries=( decisions/[0-9]*.md )
shopt -u nullglob
# Zero entries is legitimate only until the first handoff lands; after that the glob always
# matches. It is never an error here — a branch that changes no behaviour owes no handoff, and
# CLAUDE.md, not this stage, is what asks for one.
if [ "${#entries[@]}" -eq 0 ]; then
log "no decisions/ entries yet"
fi
REQUIRED_SECTIONS=("What changed" "What I verified" "What I did NOT verify")
for f in "${entries[@]}"; do
base="${f#decisions/}"
# (2) filename
if [[ ! "$base" =~ ^([0-9]{4}-[0-9]{2}-[0-9]{2})-([a-z0-9][a-z0-9-]*)\.md$ ]]; then
err "${f}: name must be decisions/YYYY-MM-DD-<slug>.md with a lowercase [a-z0-9-] slug."
err " Use your branch's last segment as the slug — it is already unique per unit of work."
summary "- decisions-log: **FAILED** (\`${f}\` misnamed)"
rc=1
continue
fi
fname_date="${BASH_REMATCH[1]}"
if ! date -d "$fname_date" +%F >/dev/null 2>&1 || [ "$(date -d "$fname_date" +%F)" != "$fname_date" ]; then
err "${f}: '${fname_date}' is not a real calendar date."
summary "- decisions-log: **FAILED** (\`${f}\` — impossible date)"
rc=1
continue
fi
# (3) frontmatter
if [ "$(head -1 "$f")" != "---" ]; then
err "${f}: must open with a '---' YAML frontmatter block (date, branch, title)."
err " See decisions/README.md for the template."
summary "- decisions-log: **FAILED** (\`${f}\` — no frontmatter)"
rc=1
continue
fi
fm="$(awk 'NR==1&&$0=="---"{inb=1;next} inb&&$0=="---"{exit} inb{print}' "$f")"
missing_keys=()
for k in date branch title; do
grep -qE "^${k}:[[:space:]]*\S" <<<"$fm" || missing_keys+=("$k")
done
if [ "${#missing_keys[@]}" -gt 0 ]; then
err "${f}: frontmatter is missing a non-empty ${missing_keys[*]}."
summary "- decisions-log: **FAILED** (\`${f}\` — frontmatter missing \`${missing_keys[*]}\`)"
rc=1
continue
fi
fm_date="$(grep -m1 -E '^date:' <<<"$fm" | sed -E 's/^date:[[:space:]]*//; s/[[:space:]"'\'']//g')"
if [ "$fm_date" != "$fname_date" ]; then
err "${f}: frontmatter date '${fm_date}' disagrees with the filename's '${fname_date}'."
err " One of them was copied from a template and not edited; the log's ordering depends on both."
summary "- decisions-log: **FAILED** (\`${f}\` — date disagrees with filename)"
rc=1
continue
fi
# (4) the three required sections, each with a real body
body_start="$(awk 'NR>1 && $0=="---" {print NR; exit}' "$f")"
[ -n "$body_start" ] || body_start=1
for want_sec in "${REQUIRED_SECTIONS[@]}"; do
hdr_line="$(awk -v s="$body_start" -v w="## ${want_sec}" 'NR>s && $0==w {print NR; exit}' "$f")"
if [ -z "$hdr_line" ]; then
err "${f}: missing the required section '## ${want_sec}'."
err " CLAUDE.md §7 asks for what you changed, what you verified, and what you did NOT."
err " All three are structure here, because the third is the one that gets dropped."
summary "- decisions-log: **FAILED** (\`${f}\` — missing \`## ${want_sec}\`)"
rc=1
continue
fi
# A real body: at least 20 non-blank, non-heading characters before the next '## '.
nchars="$(awk -v s="$hdr_line" 'NR>s { if ($0 ~ /^## /) exit; print }' "$f" \
| tr -d '[:space:]' | wc -c)"
if [ "$nchars" -lt 20 ]; then
err "${f}: section '## ${want_sec}' is empty or a stub (${nchars} chars of content)."
err " If nothing was verified, WRITE THAT — 'nothing was verified; the gate was not run'"
err " is a valid and useful entry. A blank section is not."
summary "- decisions-log: **FAILED** (\`${f}\` — \`## ${want_sec}\` is a stub)"
rc=1
fi
done
done
if [ "$rc" -eq 0 ]; then
ok "decisions log: archive frozen, ${#entries[@]} entr$([ "${#entries[@]}" -eq 1 ] && echo y || echo ies) well-formed"
summary "- decisions-log: **passed** (archive frozen; ${#entries[@]} entries well-formed)"
else
err "decisions log stage FAILED"
fi
exit "$rc"
+43
View File
@@ -108,10 +108,12 @@ summary "## Merge gate (R-TEST-03)"
summary ""
run_stage spec-immutability "${CI_DIR}/spec-immutability.sh"
run_stage decisions-log "${CI_DIR}/decisions-log.sh"
run_stage lint "${CI_DIR}/lint.sh"
run_stage l0 "${CI_DIR}/l0.sh"
run_stage render-golden "${CI_DIR}/render-golden.sh"
run_stage l1 "${CI_DIR}/l1.sh"
run_stage gitea-settings "${CI_DIR}/gitea-settings.sh"
run_stage conformance "${CI_DIR}/conformance.sh"
echo >&2
@@ -188,6 +190,47 @@ case "${OUTCOME[spec-immutability]}" in
pass) ok "spec-immutability: green" ;;
esac
# gitea-settings: the ONLY check that can see Gitea's own database (branch protection +
# runner registration). A FAIL is real drift and always blocks -- that is the whole point.
# A SKIP means no token or unreachable Gitea, which is an UNWIRED CREDENTIAL, not a defect
# in the change under test. It is deliberately NOT in the strict-skips loop below: making an
# absent secret block every PR is precisely the lock-yourself-out failure this repo keeps
# walking into (a required check with nothing able to satisfy it). So a skip denies a PASS
# (exit 3) and says loudly what is unverified. Set REQUIRE_GITEA_SETTINGS=true once the
# token is wired, and the skip becomes a block as it should.
case "${OUTCOME[gitea-settings]}" in
fail) err "gitea-settings: Gitea DRIFTED from infra/gitea/expected-branch-protection.json -> blocks merge"; BLOCK=1 ;;
skip)
if truthy "${REQUIRE_GITEA_SETTINGS:-false}"; then
err "gitea-settings: skipped and REQUIRE_GITEA_SETTINGS=true -> blocks merge"
summary "- gitea-settings: **SKIPPED** (no token) — blocks merge (REQUIRE_GITEA_SETTINGS=true)"
BLOCK=1
else
warn "gitea-settings: skipped (no GITEA_SETTINGS_TOKEN) — branch protection and runner"
warn " registration are UNVERIFIED. Both were silently wrong for a month. Wire the token"
warn " (infra/gitea/set-ci-token.sh), then set REQUIRE_GITEA_SETTINGS=true."
summary "- gitea-settings: **SKIPPED** (no token — protection/runners unverified; not a pass)"
NOT_ESTABLISHED+=("gitea-settings")
fi ;;
pass) ok "gitea-settings: green" ;;
esac
# decisions-log: the handoff log's storage format. A FAIL means either the frozen archive
# DECISIONS-LOG.md was modified — which re-creates, for every OTHER open branch, the tail-hunk
# conflict that made four of five PRs unmergeable on 2026-09-29 — or a decisions/ entry is
# malformed. Always blocks, and has no skip path by construction: the stage reads only files in
# the tree under test and needs no tool or cluster, so there is no environment in which it
# "could not run". A guard against a silent, delayed, misattributed regression must not itself
# have a way to not-run.
case "${OUTCOME[decisions-log]}" in
fail) err "decisions-log: FAILED -> blocks merge (see decisions/README.md)"; BLOCK=1 ;;
skip) err "decisions-log: reported SKIPPED, which this stage has no path to produce -> treat"
err " as a defect in the stage itself and block."
summary "- decisions-log: **SKIPPED** (impossible by construction) — blocks merge"
BLOCK=1 ;;
pass) ok "decisions-log: green" ;;
esac
# L0/L1 stages: any failure blocks. A SKIP here means a pinned tool is missing or a seam is
# unwired — NEVER green. In CI (strict) that is a defect and blocks; on a workstation it
# degrades to UNDETERMINED, which is still non-zero and still not a pass. render-golden is a
+133
View File
@@ -0,0 +1,133 @@
#!/usr/bin/env bash
# ci/gitea-settings.sh — Stage: the Gitea platform settings that gate this repo still
# match what git declares.
#
# WHY THIS STAGE EXISTS AT ALL
# ----------------------------
# Every other check in this repo reads the repository, the rendered manifests, or the
# cluster. NONE of them can see Gitea's own database — and that is where branch
# protection and runner registration live. On 2026-09-29 that blind spot was measured:
#
# * `main` had NO branch protection, and
# * ZERO Actions runners were registered,
#
# for roughly a month, while DECISIONS-LOG.md recorded both as working and every agent
# and human in the loop repeated that claim because the repository said so. PR #1 — the
# first pull request this repo ever had — sat `pending` for 30.6 minutes with nothing to
# execute it. Later the same day `enable_status_check` and `required_approvals` were turned
# on and STILL did not bite, because `block_admin_merge_override` was false; PRs #5-#9
# merged with zero approvals recorded against `required_approvals: 1`.
#
# Three separate failures, one shape: a setting that READ as enforced while not being, with
# no mechanism anywhere that could notice. Scripts that ASSERT the desired state
# (infra/gitea/*.sh) do not detect drift from it. This stage is the detector.
#
# WHAT IT ASSERTS
# 1. every key in infra/gitea/expected-branch-protection.json `.protection` matches the
# LIVE rule at /repos/{repo}/branch_protections/{branch}, exactly;
# 2. at least `.runners_online_min` Actions runners report status "online".
#
# (2) is not decoration. A required status check with no runner is the deadlock this repo
# nearly walked into: every PR pends forever, including the one that would fix it.
#
# READ THE RULE OBJECT, NOT THE BRANCH VIEW. /repos/{repo}/branches/{branch} reports
# `protected`, `enable_status_check` and `required_approvals` but NOT
# `block_admin_merge_override` — it showed all-green while the rule was overridable by the
# only account that merges. This stage queries /branch_protections/ deliberately.
#
# EXIT CODES (the gate contract)
# 0 live Gitea matches the declaration
# 1 DIVERGENCE — something drifted, or was never applied. Blocks the merge.
# 3 SKIP — no token, or Gitea unreachable. Never a pass; see gate.sh, which treats this
# skip like spec-immutability's (UNDETERMINED, exit 3) rather than a hard block, so
# that an unwired credential cannot lock the repository.
#
# TO WIRE IT: create a Gitea repo secret SETTINGS_API_TOKEN (a token that can read repo
# settings) — infra/gitea/set-ci-token.sh does it. Until then this stage skips LOUDLY and
# the drift it exists to catch is still undetected.
#
# NOT `GITEA_SETTINGS_TOKEN`. This comment named that until 2026-09-29 and it instructed the
# reader to do something IMPOSSIBLE: Gitea reserves the `GITEA_` prefix for secret names and
# rejects the PUT with HTTP 400 "invalid secret name". That is not a style rule — it is why the
# detector sat inert for hours while a token existed and no secret did. The env var below still
# accepts GITEA_SETTINGS_TOKEN, but only for a LOCAL run: it can never be the CI wiring, because
# the secret that would feed it cannot be created at all.
set -euo pipefail
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
# shellcheck disable=SC1091
source "${CI_DIR}/lib.sh"
log "Stage: Gitea platform settings match the declaration"
DECL="${REPO_ROOT}/infra/gitea/expected-branch-protection.json"
GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
# SETTINGS_API_TOKEN, not GITEA_SETTINGS_TOKEN. Gitea RESERVES the `GITEA_` prefix for
# secret names and rejects a PUT of one with HTTP 400 "invalid secret name" — the same way
# GitHub reserves `GITHUB_`. The original name could therefore never be stored, and because
# infra/gitea/set-ci-token.sh sent that PUT to /dev/null and printed "stored" unconditionally,
# it looked like it had worked. A token was minted, no secret existed, and this stage would
# have skipped forever. GITEA_SETTINGS_TOKEN is still honoured for a local run.
TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
[ -f "$DECL" ] || die "declaration missing: ${DECL}"
require_tool jq
require_tool curl
if [ -z "$TOKEN" ]; then
skip "SETTINGS_API_TOKEN is not set — cannot read the live Gitea rule."
skip " Branch protection and runner registration are therefore UNVERIFIED. They were"
skip " wrong, unnoticed, for a month (2026-09-29). Wire it: infra/gitea/set-ci-token.sh"
summary "- gitea-settings: **SKIPPED — no token** (protection + runners UNVERIFIED; not a pass)"
exit 3
fi
api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
BRANCH="$(jq -r '.branch' "$DECL")"
LIVE="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/branch_protections/${BRANCH}" || true)"
if [ -z "$LIVE" ] || ! jq -e 'type == "object" and has("branch_name")' >/dev/null 2>&1 <<<"$LIVE"; then
skip "could not read the branch-protection rule from ${GITEA_URL} (unreachable, or the token"
skip " lacks permission). Not treating this as a pass."
summary "- gitea-settings: **SKIPPED — Gitea unreadable** (not a pass)"
exit 3
fi
RC=0
# --- 1. every declared protection key must match the live rule, exactly ---------------
while IFS= read -r key; do
want="$(jq -Sc --arg k "$key" '.protection[$k]' "$DECL")"
got="$(jq -Sc --arg k "$key" 'if has($k) then .[$k] else "<<ABSENT>>" end' <<<"$LIVE")"
if [ "$want" = "$got" ]; then
ok " ${key} = ${got}"
else
err " ${key}: DECLARED ${want} but LIVE is ${got}"
RC=1
fi
done < <(jq -r '.protection | keys[]' "$DECL")
# --- 2. a runner must actually be online ----------------------------------------------
MIN="$(jq -r '.runners_online_min // 1' "$DECL")"
RUNNERS="$(api "${GITEA_URL}/api/v1/admin/actions/runners" || true)"
N="$(jq -r '[(.runners//[])[]|select(.status=="online")]|length' <<<"${RUNNERS:-{\}}" 2>/dev/null || echo 0)"
if [ "${N:-0}" -ge "$MIN" ]; then
ok " runners online: ${N} (>= ${MIN})"
else
err " runners online: ${N}, DECLARED minimum ${MIN}."
err " A required status check with no runner leaves every PR pending forever —"
err " including the PR that would fix it. This is the 2026-09-29 outage condition."
RC=1
fi
if [ "$RC" -eq 0 ]; then
ok "Gitea settings match the declaration."
summary "- gitea-settings: **green** (branch protection + runners match \`infra/gitea/expected-branch-protection.json\`)"
else
err "Gitea has DRIFTED from infra/gitea/expected-branch-protection.json."
err " Re-assert it (infra/gitea/require-gate.sh, block-admin-override.sh), or if the live"
err " state is what you now want, change the DECLARATION in the same commit — deliberately."
summary "- gitea-settings: **DRIFT** — live Gitea does not match the declaration; blocks merge"
fi
exit "$RC"
+28 -3
View File
@@ -79,12 +79,37 @@ else
cp -a "${KYVERNO_POLICY_DIR}/." "$l0_render/"
find "$l0_render" -type f -name '*.yaml' -print0 \
| xargs -0 sed -i "s/\${KYVERNO_VALIDATION_ACTION}/${KYVERNO_VALIDATION_ACTION}/g"
if kyverno test "$l0_render"; then
ok "Kyverno L0 policy tests passed"
summary "- kyverno-l0: **passed**"
# DO NOT TRUST `kyverno test`'s EXIT CODE ALONE. A policy that fails to LOAD (schema
# error, bad match block, malformed JMESPath) does not fail the run: every result row
# becomes `Skip / Invalid Policy` and the CLI still prints "N tests passed" and exits 0.
#
# MEASURED 2026-09-29 while authoring conformance-trigger-job-shape: the policy had its
# `subjects` in a separate `match.all` entry from its `resources` — a schema error. All 9
# rows came back `Skip / Invalid Policy`, and the summary read:
# Test Summary: 9 tests passed and 0 tests failed
# `kyverno apply` on the same file reported `error: 1` and named the path. So the gate's
# own policy stage would have certified a SECURITY POLICY THAT DOES NOT LOAD as green.
#
# This is not specific to that policy: a typo in restrict-host-namespaces or
# secret-provenance would silently disable it and the gate would stay green. Scan the
# output for the load failure the exit code hides.
l0_kout="$(mktemp)"
if kyverno test "$l0_render" 2>&1 | tee "$l0_kout"; then
if grep -q 'Invalid Policy' "$l0_kout"; then
err "Kyverno reported 'Invalid Policy' — one or more policies FAILED TO LOAD and were"
err " skipped, so their rules were never evaluated. The CLI exits 0 for this; the gate"
err " must not. Run 'kyverno apply <policy> --resource <resources>' for the exact path."
grep -n 'Invalid Policy' "$l0_kout" | head -5 >&2
summary "- kyverno-l0: **FAILED — policy did not load** (Invalid Policy; rules never evaluated)"
rc=1
else
ok "Kyverno L0 policy tests passed"
summary "- kyverno-l0: **passed**"
fi
else
err "Kyverno L0 policy tests failed"; summary "- kyverno-l0: **FAILED**"; rc=1
fi
rm -f "$l0_kout"
rm -rf "$l0_render"
fi
+130
View File
@@ -0,0 +1,130 @@
#!/usr/bin/env bash
# ci/tests/decisions-log.test.sh — proves ci/decisions-log.sh actually fails closed.
#
# The stage exists because appending to DECISIONS-LOG.md silently re-creates a merge conflict
# for every other open branch. A guard against a silent regression is itself worth nothing
# unless somebody has watched it reject the thing it claims to reject, so each case below
# CONSTRUCTS the violation in a throwaway tree and asserts a non-zero exit — the same reasoning
# as ci/tests/gate-decision.test.sh, one layer down.
set -uo pipefail
REAL_CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
PASS=0 FAIL=0
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
bad() { printf ' FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT
# A minimal well-formed tree: the frozen archive, its pin, and one valid entry.
mk_tree() {
local d="$1"; rm -rf "$d"; mkdir -p "$d/ci" "$d/decisions"
cp "${REAL_CI_DIR}/decisions-log.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env" "$d/ci/"
printf 'frozen history\n' > "$d/DECISIONS-LOG.md"
( cd "$d" && sha256sum DECISIONS-LOG.md > decisions/ARCHIVE.sha256 )
mk_entry "$d/decisions/2026-09-29-a-good-entry.md" 2026-09-29
# REPO_ROOT in ci/lib.sh is derived from git; give the tree a repo so it resolves here.
( cd "$d" && git init -q . && git add -A && git -c user.email=t@t -c user.name=t commit -qm init )
}
mk_entry() {
cat > "$1" <<ENTRY
---
date: $2
branch: chore/example
title: An example entry
---
## What changed
Enough prose here to clear the twenty-character body floor comfortably.
## What I verified
Enough prose here to clear the twenty-character body floor comfortably.
## What I did NOT verify
Enough prose here to clear the twenty-character body floor comfortably.
ENTRY
}
# run <dir> -> sets RC and OUT
run() { OUT="$( cd "$1" && bash ci/decisions-log.sh 2>&1 )"; RC=$?; }
D="${TMP}/t"
echo "== 1. a well-formed tree passes"
mk_tree "$D"; run "$D"
[ "$RC" -eq 0 ] && ok "clean tree -> exit 0" || bad "clean tree -> exit ${RC}: ${OUT}"
echo
echo "== 2. THE REGRESSION THIS STAGE EXISTS FOR: appending to the frozen archive"
mk_tree "$D"; printf '\n## my new handoff\n' >> "$D/DECISIONS-LOG.md"; run "$D"
[ "$RC" -ne 0 ] && ok "append to DECISIONS-LOG.md -> non-zero (${RC})" \
|| bad "append to DECISIONS-LOG.md was ACCEPTED — the guard is inert"
grep -qi 'frozen archive' <<<"$OUT" && ok "the message names the archive as frozen" \
|| bad "the message does not say the archive is frozen"
grep -q 'decisions/\$(date +%F)' <<<"$OUT" && ok "the message names the file to write instead" \
|| bad "the message does not name the remedy"
echo
echo "== 3. a re-pin is a deliberate, accepted act"
mk_tree "$D"; printf '\nredacted\n' >> "$D/DECISIONS-LOG.md"
( cd "$D" && sha256sum DECISIONS-LOG.md > decisions/ARCHIVE.sha256 ); run "$D"
[ "$RC" -eq 0 ] && ok "archive changed AND re-pinned -> exit 0" || bad "re-pin rejected: ${OUT}"
echo
echo "== 4. a missing pin is unverifiable, not green"
mk_tree "$D"; rm "$D/decisions/ARCHIVE.sha256"; run "$D"
[ "$RC" -ne 0 ] && ok "no ARCHIVE.sha256 -> non-zero (${RC})" || bad "missing pin was accepted"
echo
echo "== 5. entry naming"
for badname in 2026-9-29-x.md 26-09-29-x.md 2026-09-29-Upper.md 2026-09-29-.md 2026-02-30-x.md; do
mk_tree "$D"; mk_entry "$D/decisions/${badname}" 2026-09-29; run "$D"
[ "$RC" -ne 0 ] && ok "rejects decisions/${badname}" || bad "ACCEPTED decisions/${badname}"
done
mk_tree "$D"; mk_entry "$D/decisions/2026-10-01-fix-thing-2.md" 2026-10-01; run "$D"
[ "$RC" -eq 0 ] && ok "accepts a digits-and-dashes slug" || bad "rejected a valid slug: ${OUT}"
echo
echo "== 6. frontmatter"
mk_tree "$D"; sed -i '1d' "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
[ "$RC" -ne 0 ] && ok "rejects an entry with no frontmatter" || bad "accepted a bare entry"
for key in date branch title; do
mk_tree "$D"; sed -i "/^${key}:/d" "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
[ "$RC" -ne 0 ] && ok "rejects frontmatter missing '${key}'" || bad "accepted missing '${key}'"
done
# The copied-template failure: the body was rewritten, the date never was.
mk_tree "$D"; mk_entry "$D/decisions/2026-09-30-copied.md" 2026-09-29; run "$D"
[ "$RC" -ne 0 ] && ok "rejects a frontmatter date that disagrees with the filename" \
|| bad "accepted a date/filename disagreement"
echo
echo "== 7. the three required sections, and their bodies"
for sec in 'What changed' 'What I verified' 'What I did NOT verify'; do
mk_tree "$D"; sed -i "/^## ${sec}\$/d" "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
[ "$RC" -ne 0 ] && ok "rejects an entry missing '## ${sec}'" || bad "accepted missing '## ${sec}'"
done
# A heading with nothing under it is the way this floor gets defeated in practice.
mk_tree "$D"
python3 - "$D/decisions/2026-09-29-a-good-entry.md" <<'PY'
import sys,re
p=sys.argv[1]; s=open(p).read()
s=re.sub(r'## What I did NOT verify\n\n.*', '## What I did NOT verify\n\nn/a\n', s, flags=re.S)
open(p,'w').write(s)
PY
run "$D"
[ "$RC" -ne 0 ] && ok "rejects a stub body under a required section" || bad "accepted an 'n/a' section"
echo
echo "== 8. README.md in decisions/ is not an entry and is not policed as one"
mk_tree "$D"; printf 'not an entry\n' > "$D/decisions/README.md"; run "$D"
[ "$RC" -eq 0 ] && ok "decisions/README.md ignored by the entry rules" || bad "README policed: ${OUT}"
echo
if [ "$FAIL" -eq 0 ]; then echo "decisions-log tests: PASS ($PASS assertions)"; exit 0
else echo "decisions-log tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi
+22 -1
View File
@@ -36,7 +36,13 @@ REPO="${TMP}/repo"; mkdir -p "$REPO"
cp "${REAL_CI_DIR}/gate.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env" "$CI/"
# One stub per stage. It exits with STAGE_RC_<stage-with-dashes-as-underscores>, default 0.
for stage in spec-immutability lint l0 render-golden l1 conformance; do
# NOTE: this list must name EVERY stage gate.sh runs. It did not: `gitea-settings` was added to
# gate.sh and never added here, so its script was absent from the stub CI dir, `run_stage` got
# exit 127 from the missing file, and the gate recorded a FAILED stage in every single case —
# 15 of 47 assertions failed on `main` for that reason alone, including every one that expects a
# PASS/UNDETERMINED/NOT-APPLICABLE headline. The suite was measuring the missing stub, not the
# decision logic. `decisions-log` is added here in the same change that adds it to gate.sh.
for stage in spec-immutability decisions-log lint l0 render-golden l1 gitea-settings conformance; do
cat > "${CI}/${stage}.sh" <<'STUB'
#!/usr/bin/env bash
n="$(basename "$0" .sh)"; v="STAGE_RC_${n//-/_}"
@@ -184,6 +190,21 @@ assert_verdict BLOCKED 1 "l0 FAILED + l1 SKIPPED"
run_gate STAGE_RC_spec_immutability=3 STAGE_RC_conformance=3
assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED + conformance declared-absent"
echo
echo "== 10. decisions-log: the handoff log's storage format has no non-blocking outcome"
# A FAIL here means the frozen archive was appended to (re-creating, for every OTHER open
# branch, the tail-hunk conflict that made 4 of 5 PRs unmergeable on 2026-09-29) or an entry is
# malformed. A SKIP is impossible by construction — the stage needs no tool and no cluster — so
# if one is ever reported it is a defect in the stage, and the gate must not soften it into
# UNDETERMINED. Both outcomes block, and neither is upgradeable by an env knob: there is
# deliberately no REQUIRE_DECISIONS_LOG, because a guard with an off switch is not a guard.
run_gate STAGE_RC_decisions_log=1
assert_verdict BLOCKED 1 "decisions-log FAILED (archive modified / entry malformed)"
run_gate STAGE_RC_decisions_log=3
assert_verdict BLOCKED 1 "decisions-log SKIPPED (impossible by construction -> defect, blocks)"
run_gate STAGE_RC_decisions_log=1 STAGE_RC_l1=3
assert_verdict BLOCKED 1 "decisions-log FAILED outranks an l1 non-run"
echo
if [ "$FAIL" -eq 0 ]; then echo "gate-decision tests: PASS ($PASS assertions)"; exit 0
else echo "gate-decision tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi
+14
View File
@@ -1,5 +1,19 @@
# clusters/
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
> This document speaks as though no cluster exists and the platform is scaffolding. **A
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
> coverage hole, not a phase.
>
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
> documentation rather than measurement, and `main` was unprotected with zero runners
> registered the whole time. **Query the live API before asserting platform state.**
Flux entrypoints and the environment-overlay model (**R-ENV-02**, **R-GITOPS-01/02/03**).
Governing spec: [`docs/05-gitops-and-delivery.md`](../../home-network-k3s/docs/05-gitops-and-delivery.md),
+31
View File
@@ -0,0 +1,31 @@
---
# Conformance identities (ns + conformance-runner + conformance-trigger RBAC).
#
# NON-GATING, exactly like apps-vms: `wait: false` and its own Kustomization, so a conformance
# identity can never gate the platform or app tiers. A test harness that stalls the estate it is
# meant to verify is worse than no harness.
#
# dependsOn platform-config only — it needs nothing from the app waves, and nothing in the app
# waves needs it.
#
# NOTE: this reconciles RBAC that is INERT until
# platform/kyverno/admission/conformance-trigger-job-shape.yaml is also enforcing. That policy
# ships in the `platform` tier, which reconciles before this. Do not reorder them so that the
# trigger identity can exist while the policy bounding it does not.
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: conformance
namespace: flux-system
spec:
interval: 10m
retryInterval: 2m
timeout: 5m
prune: true
wait: false
dependsOn:
- name: platform-config
sourceRef:
kind: GitRepository
name: flux-system
path: ./conformance/incluster
+1
View File
@@ -33,6 +33,7 @@ resources:
- apps-wave-2.yaml # Flux Kustomization -> ./apps/wave-2 (dependsOn apps-wave-1)
# VM tier (KubeVirt) — non-gating (wait:false), separate lifecycle from the app waves.
- apps-vms.yaml # Flux Kustomization -> ./apps/vms (dependsOn platform-config)
- conformance.yaml # Flux Kustomization -> ./conformance/incluster (non-gating; R-SEC-02 trigger identity)
# oauth2-proxy forward-auth — POST-platform tier (dependsOn platform-config, wait:false). A leaf
# service that needs Keycloak + trusted TLS + split-horizon DNS must NOT sit in the gating
# `platform` tier, where it self-gated its own dependencies into a cold-start deadlock (twice).
+14
View File
@@ -1,5 +1,19 @@
# conformance/ — the home-k3s conformance harness (WS-TEST)
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
> This document speaks as though no cluster exists and the platform is scaffolding. **A
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
> coverage hole, not a phase.
>
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
> documentation rather than measurement, and `main` was unprotected with zero runners
> registered the whole time. **Query the live API before asserting platform state.**
This subtree is the **definition of "done"** for the whole platform. Nothing is accepted
on a demo, a screenshot, or "it reconciled": every requirement is proven by an
**automated, reproducible, behavior-based test** tagged with the requirement ID(s) it
+14
View File
@@ -1,5 +1,19 @@
# In-cluster conformance Job (L2/L3)
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
> This document speaks as though no cluster exists and the platform is scaffolding. **A
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
> coverage hole, not a phase.
>
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
> documentation rather than measurement, and `main` was unprotected with zero runners
> registered the whole time. **Query the live API before asserting platform state.**
Per **ADR-0011** and **docs/14 "Where each layer runs"**: L0/L1 run in CI without a
cluster, but **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job**
on the dev/test cluster that **CI triggers and waits for**. CI holds **no** cluster
+14
View File
@@ -0,0 +1,14 @@
# Reconciled by clusters/base/conformance.yaml (non-gating, wait:false).
#
# RBAC ONLY. chainsaw-job.yaml is deliberately NOT listed: that manifest is a TEMPLATE the CI
# trigger renders and submits per run (with the commit SHA and the touched --requirements). Flux
# reconciling it would create a permanent Job object and fight the trigger for ownership.
#
# The identities here are inert without
# platform/kyverno/admission/conformance-trigger-job-shape.yaml, which is what actually bounds
# the trigger to the committed run shape. RBAC is the necessary half, that policy the sufficient
# half — see the header of rbac.yaml.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- rbac.yaml
+65
View File
@@ -174,3 +174,68 @@ subjects:
- kind: ServiceAccount
name: conformance-runner
namespace: conformance
---
# =============================================================================
# THE TRIGGER IDENTITY — deliberately NOT conformance-runner.
#
# THE ESCALATION THIS EXISTS TO CLOSE (docs/plans/l2-l3-conformance-ci-wiring.md §4):
# Kubernetes places NO `use` check on a Pod/Job's `serviceAccountName`. So any principal
# that can `create jobs` in this namespace can create a Job with
# `serviceAccountName: conformance-runner` and an ARBITRARY command — i.e. run arbitrary
# code with the runner's privileges (nodes/patch, pods/exec, namespaces/patch, jobs).
# A CI-held token with plain `create jobs` therefore IS the privileged identity, no matter
# how narrow its own RBAC looks. That would break R-SEC-02 ("CI holds no broad cluster
# access") while appearing to satisfy it.
#
# RBAC ALONE CANNOT FIX THIS. The real boundary is the admission policy
# platform/kyverno/admission/conformance-trigger-job-shape.yaml, which admits Jobs from
# THIS ServiceAccount only when they match the committed conformance shape. This Role is
# the necessary half; that policy is the sufficient half. Neither works alone — do not
# grant this Role without that policy reconciled and ENFORCING.
#
# Namespaced Role, bound only in `conformance`. No ClusterRole, deliberately.
# =============================================================================
apiVersion: v1
kind: ServiceAccount
metadata:
name: conformance-trigger
namespace: conformance
labels:
app.kubernetes.io/part-of: conformance
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: conformance-trigger
namespace: conformance
labels:
app.kubernetes.io/part-of: conformance
rules:
# Create the run, and watch it to completion.
- apiGroups: [batch]
resources: [jobs]
verbs: [create, get, list, watch]
# Read the Job's pod and its log — this is how results come back. Deliberately NOT
# pods/exec and NOT `kubectl cp`: the run tars its artifacts to stdout and CI decodes
# them from the log, so no execution channel into the runner pod is ever granted.
- apiGroups: [""]
resources: [pods, pods/log]
verbs: [get, list, watch]
# NOTHING ELSE. In particular NO delete on jobs (a trigger that can delete can hide a
# failed run), no secrets, no configmaps, no exec, no create on pods.
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: conformance-trigger
namespace: conformance
labels:
app.kubernetes.io/part-of: conformance
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: conformance-trigger
subjects:
- kind: ServiceAccount
name: conformance-trigger
namespace: conformance
@@ -0,0 +1,42 @@
# =============================================================================
# L0 STATIC — every authored admission policy is actually APPLIED (R-SEC-01, R-SEC-02)
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# WHY THIS EXISTS. `kyverno test` proves a policy is CORRECT. Nothing proved it was DEPLOYED.
# platform/kyverno/ holds three lists and only platform-config/kustomization.yaml applies
# anything; the other two are a reference copy and a namespace+HelmRelease render root. On
# 2026-09-29 the R-SEC-02 boundary policy was added to the reference copy: it parsed, its
# kyverno test passed, the gate passed, Flux reported the tier Ready at the new revision — and
# the cluster ran the conformance-trigger ServiceAccount with nothing bounding it.
#
# WHOLE-REPO SCOPE, DELIBERATELY. The failure mode is a policy that is present but wired
# nowhere, which by definition touches no requirement the diff attributes. A path-scoped run
# would have skipped this check on the very commit that introduced the defect.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: admission-policies-actually-applied
labels:
conformance.home.k3s/R-SEC-01: "true"
conformance.home.k3s/R-SEC-02: "true"
annotations:
conformance.home.k3s/requirements: "R-SEC-01,R-SEC-02"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Every ClusterPolicy authored under platform/kyverno/ is either referenced by
platform-config/kustomization.yaml (so Flux applies it) or is a documented CI-only
exemption that is genuinely not applied — so a policy cannot be authored, tested,
green, and absent from the cluster.
spec:
steps:
- name: every-authored-policy-is-applied-or-documented-ci-only
description: >-
Enumerate the ClusterPolicy files under platform/kyverno/ and assert each is wired
into the Kustomization that actually applies Kyverno CRs, or exempted with a reason.
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,120 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — admission-policies-actually-applied
# Serves: R-SEC-01, R-SEC-02 (admission enforcement is only real if it is APPLIED)
#
# WHY THIS EXISTS — a policy that ships green and never reaches the cluster.
#
# platform/kyverno/ holds THREE lists, and only one of them applies anything:
# 1. platform/kyverno/admission/kustomization.yaml — reference only. Its own sibling says
# "kept as reference; THIS file is the buildable render root".
# 2. platform/kyverno/kustomization.yaml — that render root: namespace +
# HelmRelease ONLY. It explicitly notes the ClusterPolicies are applied elsewhere.
# 3. platform-config/kustomization.yaml — THE ONE. Kyverno CRs go here because
# they need the CRDs from the HelmRelease in (2), so they reconcile a tier later.
#
# On 2026-09-29 a ClusterPolicy (conformance-trigger-job-shape, the R-SEC-02 boundary bounding
# what the conformance CI trigger may create) was added to (1). Everything went green: the file
# parsed, `kyverno test` exercised it BY PATH and passed, the gate passed, Flux reported
# `platform` Ready at the new revision — and the policy was never in any applied set. The
# cluster ended up running the conformance-trigger ServiceAccount with NOTHING bounding it,
# which is precisely the state that identity's own RBAC header says must never exist.
#
# Nothing could have caught it. `kyverno test` proves a policy is CORRECT, not that it is
# DEPLOYED. This check asserts the second thing.
#
# GIVEN the repository as git tracks it,
# WHEN every file under platform/kyverno/ that declares a ClusterPolicy is enumerated,
# THEN each one is either referenced by platform-config/kustomization.yaml (so Flux applies
# it), or named in CI_ONLY below with a reason — and CI_ONLY itself may not name a
# policy that IS applied, so the exemption list cannot rot into a lie either way.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
exec python3 - "$ROOT" <<'PY'
import sys, subprocess
from pathlib import Path
ROOT = Path(sys.argv[1])
FAILURES, CHECKS = [], 0
def check(cond, desc):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(desc)
# Policies authored for the CI L0 gate that are deliberately NOT applied in-cluster. Each needs
# a reason, and each is asserted BELOW to really be absent from the applied set — an exemption
# that silently starts being applied is as much a lie as one that silently stops.
CI_ONLY = {
"disallow-plaintext-secrets":
"CI-only by design: it requires a top-level `sops` block, which is IMPOSSIBLE at "
"admission because Flux decrypts before the Secret reaches the API server. Its "
"in-cluster counterpart is admission/secret-provenance.yaml.",
}
tracked = subprocess.run(["git", "ls-files", "-z", "--", "platform/kyverno/**/*.yaml",
"platform/kyverno/*.yaml"],
cwd=ROOT, capture_output=True, text=True, check=True)
files = [ROOT / p for p in tracked.stdout.split("\0") if p]
# A ClusterPolicy file, identified without a YAML parse of every doc: cheap and sufficient.
policies = {}
for f in files:
try:
txt = f.read_text()
except Exception:
continue
if "kind: ClusterPolicy" not in txt:
continue
for line in txt.splitlines():
st = line.strip()
if st.startswith("name:") and " name:" in line:
policies[f.relative_to(ROOT).as_posix()] = st.split("name:", 1)[1].strip()
break
check(len(policies) >= 6,
f"(0) the scan found {len(policies)} ClusterPolicy file(s) under platform/kyverno/ — "
f"too few means the scan broke, and every assertion below would be vacuous")
applied_txt = (ROOT / "platform-config" / "kustomization.yaml").read_text()
unapplied, wrongly_exempt = [], []
for rel, name in sorted(policies.items()):
# platform-config references these as ../platform/kyverno/...
referenced = rel in applied_txt or rel.replace("platform/", "../platform/", 1) in applied_txt
if name in CI_ONLY:
if referenced:
wrongly_exempt.append(f"{name} ({rel}) is listed CI_ONLY but IS applied by platform-config")
elif not referenced:
unapplied.append(f"{name} ({rel})")
check(not unapplied,
f"(1) every in-cluster ClusterPolicy is referenced by platform-config/kustomization.yaml "
f"— an unreferenced policy is authored, tested, green, and NEVER APPLIED {unapplied if unapplied else ''}")
check(not wrongly_exempt,
f"(2) no CI_ONLY exemption names a policy that IS applied — a stale exemption hides a "
f"real deployment {wrongly_exempt if wrongly_exempt else ''}")
for name, reason in CI_ONLY.items():
check(bool(reason.strip()),
f"(3) CI_ONLY['{name}'] carries a reason — an unexplained exemption is how this "
f"list stops being reviewable")
check(CHECKS >= 4,
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op")
if FAILURES:
print(f"CHECK FAIL (admission-policies-actually-applied): {len(FAILURES)} of {CHECKS} assertions failed",
file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
print(f"check OK: admission-policies-actually-applied — {CHECKS} assertions; "
f"{len(policies)} ClusterPolicy file(s), {len(policies) - len(CI_ONLY)} applied by "
f"platform-config, {len(CI_ONLY)} CI-only with reasons. Proves policies are DEPLOYED, "
f"not that they are correct — `kyverno test` is what proves correctness.")
PY
@@ -216,6 +216,20 @@ def is_literal_base(host: str) -> bool:
DELIVERY_ROOTS = ("apps", "platform", "platform-config", "platform-config-cdi",
"runbooks", "clusters", "bootstrap", "catalog")
# NARROW EXEMPTION to (0), for reconciled trees that deliberately serve NOTHING.
# `conformance/incluster` (the L2/L3 runner + CI-trigger identities, R-SEC-02) is reconciled
# by clusters/base/conformance.yaml but contains only a Namespace, ServiceAccounts, RBAC and
# an admission-bounded Job template. Adding "conformance" to DELIVERY_ROOTS instead would be
# wrong for the reason stated above: it would sweep in this harness's synthetic fixture
# routes and make (8) demand the operator's dashboard link them.
#
# THE EXEMPTION POLICES ITSELF. An unchecked allowlist is how (0) would rot into a fail-open —
# add a tree here, later put an HTTPRoute in it, and the forward direction goes quietly green
# on a hidden service. So each exempted tree is ASSERTED below to declare no route and no
# hostname-publishing Service. If that ever stops being true the check fails and the tree must
# become a real delivery root.
NON_SERVING_RECONCILED = ("conformance/incluster",)
def tracked_yaml() -> list[Path]:
out = subprocess.run(["git", "ls-files", "-z", "--", "*.yaml", "*.yml"],
@@ -262,12 +276,31 @@ flux_ks = [(p, d) for p, d in docs_all()
# a Kustomization declared elsewhere); they name no source and reconcile nothing, so requiring
# a path of them would fail on a file that is correct.
reconcilers = [(p, d) for p, d in flux_ks if (d.get("spec", {}) or {}).get("sourceRef")]
def _reconciled_path(d):
return (d.get("spec", {}) or {}).get("path", "").lstrip("./").rstrip("/")
unscanned = sorted({
f"{p.relative_to(ROOT)}:{(d.get('spec', {}) or {}).get('path', '') or '(no path)'}"
for p, d in reconcilers
if (d.get("spec", {}) or {}).get("path", "").lstrip("./").split("/")[0]
not in DELIVERY_ROOTS
if _reconciled_path(d).split("/")[0] not in DELIVERY_ROOTS
and _reconciled_path(d) not in NON_SERVING_RECONCILED
})
# The exemption's own guard: an exempted tree must actually serve nothing.
exempt_serving = sorted({
f"{p.relative_to(ROOT)} ({d.get('kind')})"
for p, d in docs_all()
if any(str(p.relative_to(ROOT)).startswith(t + "/") for t in NON_SERVING_RECONCILED)
and (d.get("kind") == "HTTPRoute"
or (d.get("kind") == "Service"
and any("hostname" in str(k).lower()
for k in ((d.get("metadata", {}) or {}).get("annotations", {}) or {}))))
})
check(not exempt_serving,
f"(0b) every NON_SERVING_RECONCILED tree really serves nothing — an exempted tree that "
f"gains a route would hide a service from this check entirely "
f"{exempt_serving if exempt_serving else ''}")
check(len(reconcilers) >= 10 and not unscanned,
f"(0) all {len(reconcilers)} Flux-reconciled path(s) lie inside a scanned root — a "
f"reconciled tree this check does not read is a service it cannot require a link for "
@@ -0,0 +1,82 @@
---
date: 2026-09-29
branch: chore/decisions-log-conflict-free
title: The handoff log was serialising every parallel branch — it is a directory now
---
## What changed
`DECISIONS-LOG.md` is frozen and the handoff log is now a directory, `decisions/`, with one
file per entry (`YYYY-MM-DD-<slug>.md`). Concurrent branches write disjoint paths, so there
is nothing for git to merge.
The problem was not any particular change. CLAUDE.md §7 told every agent to append a handoff,
and appending always landed on the *same last hunk* of one 5,228-line file. Two branches that
both did it could not auto-merge; N branches cost N-1 rebases, and each rebase invalidated the
gate run before it. The agents work in isolated worktrees precisely so they don't block each
other, and this one file undid that.
- `decisions/README.md` — the convention, the template, and how to read the log. No index file
and no sequence number, deliberately: an index is a shared mutable file (the thing that just
failed) and a counter has two agents racing for the same number. The directory listing is the
index.
- `DECISIONS-LOG.md` — a "FROZEN ARCHIVE" header prepended; content otherwise untouched. Its
SHA-256 is pinned in `decisions/ARCHIVE.sha256`.
- `ci/decisions-log.sh` — a new gate stage. Asserts the archive matches its pin, and that every
entry is named `YYYY-MM-DD-<slug>.md` with a real date, carries `date`/`branch`/`title`
frontmatter agreeing with its filename, and has all three of §7's sections with a real body
under each. Exit 0 or 1 only; it has no skip path by construction, because it reads only files
in the tree under test and needs no tool and no cluster.
- `ci/gate.sh` — runs the stage; both `fail` and (impossible) `skip` set `BLOCK`. There is no
`REQUIRE_DECISIONS_LOG` knob: a guard with an off switch is not a guard.
- `ci/tests/decisions-log.test.sh` — 22 assertions, new.
- `ci/tests/gate-decision.test.sh` — 3 new assertions for the stage's wiring, **plus a
pre-existing bug fixed**: its stub list never included `gitea-settings`, so that stage's
script was absent from the stub CI dir, `run_stage` recorded exit 127 as a FAILED stage in
every case, and 15 of 47 assertions failed on `main`. The suite was measuring its own missing
stub rather than the decision logic.
- `CLAUDE.md` §7 rewritten; §1's "stale runbook wording" caveat extended to the log.
- `ci/Makefile`, `ci/README.md`, `.gitea/workflows/README.md` — the stage named where stages are
named.
A guard rather than a paragraph, because the old habit is written into the runbooks and into
every agent that has read the log, and getting it wrong does not fail: it produces a clean-looking
commit that re-creates the conflict on *other people's* branches, later, misattributed.
## What I verified
Measured against the live Gitea (`GET /api/v1/repos/Home-Network/home-network-k3s-impl/pulls`,
2026-09-29), not inferred from the tree:
#22 fix/crafty-oauth2-proxy-signin mergeable=False
#20 feat/conformance-job-template mergeable=False
#19 feat/conformance-trigger-kubeconfig mergeable=False
#18 feat/conformance-runner-image mergeable=False
#17 feat/ci-is-the-approver mergeable=True
`git merge-tree --write-tree origin/main origin/<branch>` for each named
`CONFLICT (content): DECISIONS-LOG.md` and **no other conflicting path** in all four. #17 is
clean and touched no log entry. So the log's storage format was the whole of the blockage.
Tests: `ci/tests/decisions-log.test.sh` **PASS (22 assertions)**, including the one that
matters — an append to `DECISIONS-LOG.md` exits non-zero and the message names both the frozen
archive and the file to write instead. `ci/tests/gate-decision.test.sh` **PASS (50 assertions)**,
up from **FAIL (15 failed, 32 passed)** on `main` before the stub fix.
Gate: see the run quoted in the PR description.
## What I did NOT verify
- **Nothing was applied and nothing was merged.** No `kubectl`, no `flux reconcile`. This change
touches only CI and documentation; it has no cluster-visible effect at all.
- **L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), as always. Irrelevant to this
change's content but stated so the gate result is not over-read.
- **I have not watched a Gitea PR check run this stage.** The stage is proven by its own unit
test and by the gate locally; that the `incluster-dind` runner executes it is unobserved here.
A skipped Gitea job still reports `success`, so read the job log, not the tick.
- **The four hung PRs are not yet fixed by this commit.** They still carry their
`DECISIONS-LOG.md` appends and will still conflict until each is rebased onto this change with
its append converted to a `decisions/` entry. That is follow-up work, in flight separately.
- **`runbooks/` and older entries still say "append to DECISIONS-LOG.md"** in places. I did not
sweep them. The gate now rejects the action, so the stale wording misleads but cannot cause
the damage; it joins the existing tracked work on stale `git push origin main` wording.
+1
View File
@@ -0,0 +1 @@
b16e8cb7ea12720bc7c3a77a496211e3204534c5e6600ba748fb9ab14fa21c26 DECISIONS-LOG.md
+101
View File
@@ -0,0 +1,101 @@
# `decisions/` — the handoff log, one file per entry
This directory **is** the decision log. `../DECISIONS-LOG.md` is the frozen archive of
everything written before 2026-09-29 and is never edited again (see "The archive is
frozen" below).
## Why this exists
`DECISIONS-LOG.md` was append-only in a single file, and every agent appended to its
**tail**. Two branches that both append hit the same last hunk, so git cannot
auto-merge them: the second one to be reviewed conflicts, and so does the third, and
the fourth. Measured on 2026-09-29: **four of the five open PRs were unmergeable, and
`DECISIONS-LOG.md` was the only conflicting file in all four**:
#22 fix/crafty-oauth2-proxy-signin CONFLICT (content): DECISIONS-LOG.md
#20 feat/conformance-job-template CONFLICT (content): DECISIONS-LOG.md
#19 feat/conformance-trigger-kubeconfig CONFLICT (content): DECISIONS-LOG.md
#18 feat/conformance-runner-image CONFLICT (content): DECISIONS-LOG.md
#17 feat/ci-is-the-approver clean (it touched no log entry)
Nothing was wrong with any of those changes. The log's *storage format* serialised work
that was otherwise parallel, and it got worse with every concurrent branch: N branches
produce N-1 rebases, each of which invalidates any gate run that preceded it.
The fix is the obvious one for concurrent appenders: **stop sharing a file**. Each
handoff is its own file. Two agents writing handoffs touch disjoint paths, so there is
nothing for git to merge and nothing to conflict. This scales to any number of parallel
branches with zero coordination — which is the point, since the agents are deliberately
isolated from each other in separate worktrees and cannot coordinate.
There is deliberately **no index file** and no counter. An index would be a shared
mutable file — exactly the thing that just failed — and a sequence number would have
two agents racing for `0043`. The directory listing is the index.
## Writing an entry
One file per handoff, named:
decisions/YYYY-MM-DD-<slug>.md
`<slug>` is lowercase `[a-z0-9-]`. **Use your branch name's last segment** — it is
already unique per unit of work, which is precisely the property the filename needs.
A branch `fix/crafty-oauth2-proxy-signin` handed off on 2026-09-29 writes:
decisions/2026-09-29-crafty-oauth2-proxy-signin.md
The file is YAML frontmatter followed by three required sections:
```markdown
---
date: 2026-09-29
branch: fix/crafty-oauth2-proxy-signin
title: The Keycloak gate works — Crafty's own login was the wall
---
## What changed
Prose. What you actually did, and why.
## What I verified
The measurements, with their numbers. Quote the field you read, not the prose you
believed. Name the gate result and its exit code.
## What I did NOT verify
Everything else. An honest "UNVERIFIED" is worth more than an optimistic "green" —
this repo has been burned by the difference more than once.
```
All three headings are **required and enforced** (`ci/decisions-log.sh`). They are
CLAUDE.md §7's contract; making them a checked structure is what stops an entry
degrading into "done ✅".
## Reading the log
The whole log, oldest first:
ls decisions/[0-9]*.md | sort
The current PM handoff — the newest entry. Ties within a day are broken by what landed
on `main` last, which is the honest answer, so ask git rather than the filename:
git log --diff-filter=A --format='%H' --name-only -- 'decisions/[0-9]*.md' \
| grep '^decisions/' | head -1
For a quick skim of who said what:
grep -H '^title:' decisions/[0-9]*.md | sort
## The archive is frozen
`DECISIONS-LOG.md` is **read-only history**. Its SHA-256 is pinned in
`decisions/ARCHIVE.sha256` and `ci/decisions-log.sh` fails the merge gate if the file
changes. That is not ceremony: without it, the next agent to read `runbooks/` or an old
CLAUDE.md habit appends to the tail again and re-creates the conflict for everybody
else, silently, one PR at a time.
If the archive genuinely must change (a leaked credential to redact, say), that is a
deliberate, reviewed act: change it, run `sha256sum DECISIONS-LOG.md > decisions/ARCHIVE.sha256`,
and say in your handoff entry why. Re-pinning is visible in the diff. Appending is not.
+15 -2
View File
@@ -38,8 +38,21 @@ agents. So the credential model must be decided by the operator before anything
| Runner deployment | `apps/wave-1/gitea-actions.yaml` | act_runner + DinD (Docker executor). **`automountServiceAccountToken: false`** — the runner and its job containers hold NO SA token. Broad egress allowed (`- {}`), so it can *reach* the kube-API VIP `192.168.1.49:6443`, but has no credential. |
| Secrets encryption | `.sops.yaml` (repo root) | age recipient `age175l8m…mfzg`; encrypts `data`/`stringData` of `*.sops.yaml` under `platform|apps|catalog/**` and `secrets/**`. Private key is the in-cluster `sops-age` Secret Flux uses. |
**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the dev cluster
is LIVE (Phase 6). So `CLUSTER_AVAILABLE` is the only remaining gate, exactly as the blueprint says.
**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the cluster is
LIVE (and it is now PROD; dev was decommissioned 2026-09-12).
> **CORRECTION (2026-09-29): `CLUSTER_AVAILABLE` is NOT "the only remaining gate", and setting it
> today is actively DANGEROUS.** Measured: with `CLUSTER_AVAILABLE=true` and no reachable cluster,
> `conformance/run.sh` exits 0 in scaffold mode having marked all 54 tests `<skipped>`, and
> `ci/conformance.sh` then printed `[ ok ] L2/L3 conformance passed for touched subset` — **rc 0, a
> fake green**, on a merge gate that is now an enforcing required status check. The CI runner has
> no cluster credential at all (`automountServiceAccountToken: false`, by design), so the "local
> run" this stage attempts can never succeed; it can only skip and, before the guard added on
> 2026-09-29, lie about it.
>
> `ci/conformance.sh` now BLOCKS (exit 1) when a cluster is declared but every test skipped. **Do
> not set `CLUSTER_AVAILABLE=true` until §3's trigger path actually exists** — the variable does
> not enable L2/L3, it only changes which way this stage fails.
---
+113 -4
View File
@@ -43,7 +43,93 @@ list. That is precisely the observed symptom: zero runners, pod running, no erro
looks at. Clearing the PVC is what forces a real re-registration; the script prints the
commands.
## Sequence — order matters
## Sequence — DONE 2026-09-29
All four steps are complete and each was verified, not assumed:
| Step | State |
|---|---|
| 1. Runner registered | ✅ `incluster-dind` online. Root cause of its absence was a stale `/data/.runner` identity; clearing the PVC forced re-registration |
| 2. Gate observed green on a real PR | ✅ `gate / gate (pull_request)` ~1m35s |
| 3. `enable_status_check: true` | ✅ context `gate / gate (pull_request)` — that one ONLY |
| 4. Collaborator + approvals | ✅ `jakman85` is owner; `required_approvals: 1` |
Direct push to `main` was **provoked and rejected** (`pre-receive hook declined`) as
`gitea-admin`, a site admin — so push protection is not merely declared, it holds for the
account agents authenticate as.
### 5. Block the admin override — the step the first pass MISSED
`enable_status_check` and `required_approvals` were set on 2026-09-29 and **still did not
bite**, because `block_admin_merge_override` was `false`. Gitea lets repo admins override
merge requirements without it, and `jakman85` holds `owner` — so the two settings that looked
like the gate were overridable by precisely the account that merges.
**The evidence was sitting in plain sight:** PRs #5 through #9 all merged with
`required_approvals: 1` in force and **zero approvals recorded on any of them**. Checking the
*branch* view (`/branches/main`) hid this — it does not expose the field. Read the rule object
(`/branch_protections/main`) and enumerate its keys.
> **Do not probe for field names you assume exist.** This Gitea (1.24.3) has no
> `apply_to_admins`; asking `jq` for it returns `null`, which reads like "set to false" but
> means "key absent". Enumerate with `to_entries` and act on what is really there.
Now `block_admin_merge_override: true`. **That is still only a setting, not evidence** — it is
proven when a green PR is REFUSED a merge until approved, and not before.
### `require-gate.sh` / `verify-then-enforce.sh`
`verify-then-enforce.sh` is the re-runnable form of the whole safe order. It merges nothing —
merging is a human's reviewed act — but it forces the Flux reconcile and then **aborts** unless
the runner pod is healthy, a runner is online, and a real workflow goes green on it, before
enabling enforcement and probing the push. `require-gate.sh` is the enforcement step alone.
Re-run `verify-then-enforce.sh` after any future change to the act-runner Deployment.
### `expected-branch-protection.json` + `ci/gitea-settings.sh` — the detector
`expected-branch-protection.json` **declares** the desired rule. `ci/gitea-settings.sh` runs as
a merge-gate stage, reads the LIVE rule, and **fails the gate on any divergence** — plus asserts
at least one runner is online. It is the only thing in this repository that can see Gitea's own
database.
It reads `/branch_protections/main`, not `/branches/main`: the branch view does **not** expose
`block_admin_merge_override`, and it showed all-green for hours while the rule was overridable.
Behaviour, proven by test rather than assumed:
| Situation | Exit | Effect |
|---|---|---|
| live matches the declaration | 0 | green |
| any key drifted, or no runner online | 1 | **blocks the merge** |
| no `GITEA_SETTINGS_TOKEN`, or Gitea unreachable | 3 | skips **loudly**; never a pass, never a block |
The skip is deliberately *not* a block: an unwired credential blocking every PR is the
lock-yourself-out failure this repo keeps hitting. Wire the token with `set-ci-token.sh`, confirm
the stage goes **green on a real PR**, and only then set `REQUIRE_GITEA_SETTINGS=true` in
`gate.yml` so a future missing token blocks instead of skipping.
**Changing the declaration is how you change policy.** If the live state is what you now want,
edit this JSON in the same commit — deliberately, in review — rather than letting the two drift.
### Why these scripts exist at all, and what is still missing
Gitea's repo settings live in **its own database**, not in anything Flux reconciles or the
conformance gate can read. Nothing detects drift from what this directory declares. That is
the root cause of the entire 2026-09-29 session: branch protection was off and the runner was
gone for a month while the log recorded both as working, and **no check existed that could
have caught it.**
These scripts encode the desired state and can re-assert it; `ci/gitea-settings.sh` above is the
**detector** that turns divergence from a silent, invisible failure into a loud one. It closes the
root cause of the 2026-09-29 session.
It is only as good as its wiring: **until `GITEA_SETTINGS_TOKEN` exists it skips**, and a skipped
detector detects nothing. That is the remaining step.
## Historical sequence (kept for context)
### Original ordering notes
1. **Get an Actions runner registered** — `runner-token-rotate.sh`, then clear the PVC so
the runner actually re-registers, then confirm it shows `online` in the API. Do not
@@ -54,6 +140,29 @@ commands.
4. Add the human reviewer as a repo collaborator, then raise `required_approvals` to 1.
> **Do not do step 3 before step 1.** A required status check with no runner leaves every
> PR permanently pending — including the PR that would fix it. The same applies to
> `gate.yml`'s `conformance` job: it is `if: false` and never runs, yet still publishes a
> `gate / conformance (pull_request)` context. Never list that one as required.
> PR permanently pending — including the PR that would fix it. That half is measured: PR #1
> sat `pending` for 30.6 minutes across 122 polls with no runner registered, and never
> reported.
### Measured, once the runner came up (2026-09-29)
The first CI jobs this repo ever ran, on PR #1 and PR #2:
| Context | Result | Time |
|---|---|---|
| `gate / gate (pull_request)` | success | 1m58s (PR #1), 1m34s (PR #2) |
| `gate / gate (push)` | success | 1m32s (PR #2 only) |
| `gate / conformance (pull_request)` | **success — "Has been skipped"** | — |
**Require `gate / gate (pull_request)`. Do NOT require either `conformance` context —
but not for the reason an earlier draft of this file gave.** A skipped Gitea Actions job
does not hang a status check: Gitea reports it as **`success`** with the description
"Has been skipped". So requiring it would not deadlock anything — it would pass
*vacuously*, every time, forever, while nothing ran. Under HARD RULE 4 that is worse than
a deadlock: a deadlock is loud, a permanently-green check that verifies nothing is
exactly the fail-open shape this repo keeps finding.
The `(push)` contexts appear only for branches matching `gate.yml`'s push trigger
(`feature/**`, `fix/**`, `docs/**`, `chore/**`). `feat/**` is **not** in that list, which
is why PR #1's `feat/` branch produced only `pull_request` contexts. Require only the
`pull_request` context, or PRs from non-matching prefixes will never satisfy the rule.
+36
View File
@@ -0,0 +1,36 @@
#!/usr/bin/env bash
# Stop repository admins from overriding the merge requirements on `main`.
#
# WHY. Gitea lets repo admins override branch-protection merge requirements unless
# `block_admin_merge_override` is set. It was FALSE, and jakman85 holds `owner` -- so
# required_approvals and the required status check were both overridable by the one account
# that actually merges. Evidence: PRs #5-#9 all merged with ZERO approvals recorded against
# `required_approvals: 1`.
#
# NOTE ON FIELD NAMES: this Gitea (1.24.3) has NO `apply_to_admins` field. Querying it with jq
# returns null, which reads like "set to false" but actually means "key absent". Enumerate the
# real keys (`to_entries`) rather than probing names you assume exist.
#
# Recoverable: an owner can PATCH this back to false if it gets in the way.
set -euo pipefail
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
B="https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main"
echo "==> before: block_admin_merge_override = $(api "$B" | jq -r .block_admin_merge_override)"
api -X PATCH -H 'Content-Type: application/json' \
-d '{"block_admin_merge_override": true}' "$B" >/dev/null
AFTER=$(api "$B" | jq -r .block_admin_merge_override)
echo "==> after: block_admin_merge_override = $AFTER"
[ "$AFTER" = true ] || { echo "PATCH did not take -- do not record this as enforced." >&2; exit 1; }
api "$B" | jq -r '{enable_push,enable_status_check,status_check_contexts,required_approvals,block_admin_merge_override}'
cat <<'NOTE'
NOW PROVE IT -- the setting is not the evidence. On the next PR:
* let the gate go green, then try to merge WITHOUT approving it.
* Gitea must REFUSE. If it merges, this still is not doing its job.
Do not record approvals as enforced until you have seen a merge refused.
NOTE
@@ -0,0 +1,20 @@
{
"_comment": "DECLARED desired state of Gitea branch protection on main. ci/gitea-settings.sh fails the merge gate when the LIVE Gitea diverges from this. Gitea settings live in Gitea database - invisible to Flux, unreadable by every other check - so this file plus that check are the only drift detection that exists for them.",
"_generated": "2026-09-29 from the live rule, after the session that found protection OFF and zero runners for a month",
"branch": "main",
"runners_online_min": 1,
"protection": {
"enable_push": false,
"enable_push_whitelist": false,
"enable_merge_whitelist": false,
"enable_status_check": true,
"status_check_contexts": [
"gate / gate (pull_request)"
],
"required_approvals": 1,
"enable_approvals_whitelist": false,
"block_admin_merge_override": true,
"block_on_rejected_reviews": true,
"dismiss_stale_approvals": true
}
}
+60
View File
@@ -0,0 +1,60 @@
#!/usr/bin/env bash
# Items 1 + 2: make the gate a REQUIRED status check and require one approval.
#
# RUN THIS LAST. Prerequisites, in order:
# a) PR #7 (dind native sidecar) is MERGED, and
# b) you have SEEN a real workflow run go green on the converted pod.
# The runner Deployment is the merge gate for the whole repo. Turning on a required check
# while the runner is broken means no PR can merge -- including the revert. (Recoverable:
# an owner can set enable_status_check back to false. But do not rely on that.)
#
# Requires exactly ONE context: `gate / gate (pull_request)`.
# * NOT `gate / conformance (*)` -- that job is `if: false`, and a skipped Gitea job reports
# SUCCESS ("Has been skipped"). Requiring it would pass VACUOUSLY forever while nothing
# ran, which under HARD RULE 4 is worse than a deadlock: a deadlock is loud.
# block_admin_merge_override: true is LOAD-BEARING and was missing on the first pass. Gitea
# lets repo admins override merge requirements without it, so required_approvals and the
# required status check are both bypassable by exactly the account that does the merging.
# Measured: PRs #5-#9 merged with ZERO approvals recorded against required_approvals: 1.
#
# FIELD NAMES: this Gitea (1.24.3) has NO `apply_to_admins`. Asking jq for it returns null,
# which reads like "false" but means "key absent" -- enumerate real keys with `to_entries`
# instead of probing names you assume exist. That mistake cost a wasted PATCH.
#
# * NOT the `(push)` variant -- after PR #6 there are no per-branch push runs at all, so a
# PR would never produce that context.
set -euo pipefail
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
echo "==> updating branch protection on main"
jq -n '{
enable_push: false,
enable_status_check: true,
status_check_contexts: ["gate / gate (pull_request)"],
required_approvals: 1,
block_on_rejected_reviews: true,
dismiss_stale_approvals: true,
block_on_outdated_branch: false,
block_admin_merge_override: true
}' | api -X PATCH -H 'Content-Type: application/json' --data @- \
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main" >/dev/null
echo "==> verifying"
api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \
| jq -r '{protected,user_can_push,enable_status_check,status_check_contexts,required_approvals}'
cat <<'NOTE'
NOTE ON required_approvals: 1 -- Gitea will not let an author approve their own PR.
* Agent-opened PRs (author gitea-admin) are approved by jakman85. Fine.
* PRs YOU open as jakman85 need gitea-admin to approve. If that is awkward, either open
agent work as gitea-admin and review as jakman85, or set required_approvals back to 0
and rely on the status check alone.
Then prove it, rather than trusting the API:
git commit --allow-empty -m probe && git push origin main # EXPECT: rejected
git reset --hard HEAD~1
NOTE
+56
View File
@@ -0,0 +1,56 @@
#!/usr/bin/env bash
# Mint a Gitea API token and store it as the repo Actions secret GITEA_SETTINGS_TOKEN,
# so ci/gitea-settings.sh can read the live branch-protection rule during the gate.
#
# The token value is NEVER printed. It goes from the token API into the secrets API and
# nowhere else -- not to stdout, not to a file, not into an agent transcript.
#
# Scope: `read:repository` + `read:admin` are what the check needs (the rule object, and the
# runner list). Gitea token scopes are coarse; grant the least that works on your version and
# confirm the check goes green rather than assuming the scope name.
set -euo pipefail
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
# NOT GITEA_* — Gitea reserves that prefix for secret names and returns HTTP 400
# "invalid secret name". The first version of this script used GITEA_SETTINGS_TOKEN, sent the
# PUT to /dev/null, and printed "stored" regardless; the secret never existed and nobody knew.
SECRET=SETTINGS_API_TOKEN
NAME="ci-gitea-settings-$(date +%Y%m%d%H%M%S)"
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
basic() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
echo "==> minting token '${NAME}' as ${U}"
TOKEN="$(basic -X POST -H 'Content-Type: application/json' \
-d "{\"name\":\"${NAME}\",\"scopes\":[\"read:repository\",\"read:admin\"]}" \
"https://${HOST}/api/v1/users/${U}/tokens" | jq -r '.sha1 // empty')"
[ -n "$TOKEN" ] || { echo "token creation failed (check scopes for your Gitea version)" >&2; exit 1; }
echo "==> got a ${#TOKEN}-character token (value not shown)"
echo "==> storing it as repo Actions secret ${SECRET}"
code="$(basic -o /tmp/set-ci-token.out -w '%{http_code}' -X PUT -H 'Content-Type: application/json' \
-d "$(jq -n --arg d "$TOKEN" '{data:$d}')" \
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets/${SECRET}")"
unset TOKEN
if [ "$code" != 201 ] && [ "$code" != 204 ]; then
echo " PUT failed: HTTP ${code} -- $(head -c 200 /tmp/set-ci-token.out)" >&2
rm -f /tmp/set-ci-token.out; exit 1
fi
rm -f /tmp/set-ci-token.out
# VERIFY BY READING IT BACK. curl -sS exits 0 on an HTTP error, so without this the script
# reports success for a store that never happened -- which is exactly what it did.
names="$(basic "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets" \
| jq -rc '[.. | objects | select(has("name")) | .name] | unique')"
grep -q "$SECRET" <<<"$names" \
|| { echo " stored HTTP ${code} but ${SECRET} is NOT in ${names} -- refusing to claim success" >&2; exit 1; }
echo "==> stored and VERIFIED present: ${names}"
cat <<'NOTE'
Next:
1. .gitea/workflows/gate.yml already passes this secret to the gate step.
2. Open any PR and confirm the gate prints "gitea-settings: green" -- it SKIPS silently
without the token, so a green run is the only proof the wiring works.
3. THEN set REQUIRE_GITEA_SETTINGS=true in gate.yml, so a future missing token BLOCKS
instead of skipping. Do that only after you have seen it green, or you lock the repo.
NOTE
+83
View File
@@ -0,0 +1,83 @@
#!/usr/bin/env bash
# verify-then-enforce.sh — run this AFTER you have merged PRs 8, 5, 6, 7 in that order.
#
# It does NOT merge anything. Merging is your reviewed act (CLAUDE.md: a human approves and
# merges). This script does the mechanical half: prove the converted runner actually works,
# and only then turn on enforcement.
#
# WHY THIS ORDER. The act-runner Deployment IS the merge gate for this repository. Enabling a
# required status check while the runner is broken means NO PR can merge -- including the
# revert. So PR #7 lands while the gate is still advisory, is PROVEN green on a real run, and
# only then does enforcement go on.
#
# Aborts at the first checkpoint that does not pass. Safe to re-run.
set -euo pipefail
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
REPO_ROOT=/var/home/keith/source/home-network-k3s-impl
D="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
cd "$REPO_ROOT"
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
A="https://${HOST}/api/v1/repos/${OWNER}/${REPO}"
say() { printf '\n\033[1m==> %s\033[0m\n' "$1"; }
die() { printf '\n\033[31mABORT: %s\033[0m\n' "$1" >&2; exit 1; }
say "PRECHECK — all four PRs must be merged first"
for n in 8 5 6 7; do
m=$(api "$A/pulls/$n" | jq -r .merged)
printf ' PR #%s merged=%s\n' "$n" "$m"
[ "$m" = true ] || die "PR #$n is not merged yet. Merge 8, 5, 6, 7 in that order, then re-run."
done
say "STEP 1 — force Flux to apply the sidecar now, not on the 10m interval"
./infra/devcontainer/dev.sh run bash -c \
'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-1 --with-source'
say "STEP 2 — CHECKPOINT: the converted pod must come up healthy"
./infra/devcontainer/dev.sh run bash -c '
kubectl --context k3s-prod -n gitea-actions rollout status deploy/act-runner --timeout=180s
echo "--- sidecar (initContainers) ---"
kubectl --context k3s-prod -n gitea-actions get deploy act-runner \
-o jsonpath="{range .spec.template.spec.initContainers[*]}{.name} restartPolicy={.restartPolicy}{\"\n\"}{end}"
kubectl --context k3s-prod -n gitea-actions get pod -l app.kubernetes.io/name=act-runner' \
|| die "converted act-runner pod is not healthy -- REVERT PR #7 before anything else"
say "STEP 3 — CHECKPOINT: a runner must be ONLINE again"
n=0
for i in $(seq 1 40); do
n=$(api "https://${HOST}/api/v1/admin/actions/runners" | jq -r '[(.runners//[])[]|select(.status=="online")]|length')
printf '\r online runners: %s (%ss) ' "$n" "$((i*5))"
[ "${n:-0}" -ge 1 ] && { echo; break; }
sleep 5
done
[ "${n:-0}" -ge 1 ] || die "no runner online after the conversion -- REVERT PR #7"
say "STEP 4 — CHECKPOINT: a REAL workflow must go green ON THE CONVERTED POD"
MAIN_SHA=$(api "$A/branches/main" | jq -r .commit.id)
echo " watching the post-merge push run on main (${MAIN_SHA:0:8})..."
st=""
for i in $(seq 1 60); do
st=$(api "$A/commits/$MAIN_SHA/status" | jq -r .state)
printf '\r main gate: %-8s (%ss) ' "$st" "$((i*10))"
case "$st" in success) echo; break;; failure|error) die "gate FAILED on the converted pod -- REVERT PR #7";; esac
sleep 10
done
[ "$st" = success ] || die "gate never completed on the converted pod -- REVERT PR #7; do NOT enable enforcement"
echo " the converted sidecar ran a real job green. Enforcement is now safe."
say "STEP 5 — require the gate + one approval"
"$D/require-gate.sh"
say "STEP 6 — prove a direct push is rejected (do not trust the API's word)"
git fetch -q origin && git checkout -q main && git reset -q --hard origin/main
git commit -q --allow-empty -m 'protection probe'
if git push origin main 2>&1 | grep -q 'protected branch'; then
echo " CONFIRMED: push to main rejected by the pre-receive hook."
else
echo " WARNING: the push was NOT rejected -- protection is not doing what it claims."
fi
git reset -q --hard origin/main
say "FINAL STATE"
api "$A/branches/main" | jq -r '{protected,user_can_push,enable_status_check,status_check_contexts,required_approvals}'
+94
View File
@@ -0,0 +1,94 @@
#!/usr/bin/env bash
# plex-claim-ship.sh — run this IMMEDIATELY AFTER editing the SOPS secret with a fresh
# plex.tv claim token. It drives the whole CI/CD loop end to end and prints a running
# stopwatch so you can see the budget being spent.
#
# branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verify
#
# It never sees, prints, or handles the token: the file is already encrypted by sops.
set -euo pipefail
HOST=gitea.home.keithjackson.dev
OWNER=Home-Network
REPO=home-network-k3s-impl
SECRET=apps/wave-0/plex-claim-secret.sops.yaml
T0=$(date +%s)
el() { printf '\n[T+%02d:%02d] %s\n' $(( ($(date +%s)-T0)/60 )) $(( ($(date +%s)-T0)%60 )) "$1"; }
cd "$(git rev-parse --show-toplevel)"
# --- preflight: refuse to ship a file that is not what we think it is -------------
git diff --quiet -- "$SECRET" && { echo "ERROR: $SECRET is unchanged. Edit it first:"; \
echo " infra/devcontainer/dev.sh run sops $SECRET"; exit 1; }
grep -q 'PLEX_CLAIM: ENC\[' "$SECRET" || { echo "ERROR: $SECRET is not encrypted. REFUSING."; exit 1; }
[ -z "$(git status --porcelain | grep -v " $SECRET\$")" ] || echo "NOTE: other files are dirty; only $SECRET will be committed."
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
# Branch prefix is now cosmetic. It used to matter: gate.yml's push trigger covered
# feature/** fix/** docs/** chore/** but not feat/**, and with act_runner at capacity 1 a
# matching prefix fired a redundant run that serialised ahead of the PR run. Those
# per-branch push triggers were deleted, so no prefix produces a duplicate run any more.
BRANCH="feat/plex-claim-token-$(date +%H%M%S)"
el "committing on $BRANCH"
git checkout -q -b "$BRANCH"
git add "$SECRET"
git commit -q -m 'chore(plex): fresh plex.tv claim token
Injected inside the plex.tv claim window per runbooks/plex-claim.md.'
SHA=$(git rev-parse HEAD)
el "pushing"
git push -q -u origin "$BRANCH"
el "opening PR"
PR=$(jq -n --arg h "$BRANCH" --arg b main --arg t 'chore(plex): fresh plex.tv claim token' \
'{head:$h,base:$b,title:$t,body:"Token injection inside the claim window. Wiring already reviewed in #1."}' \
| api -X POST -H 'Content-Type: application/json' --data @- \
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls" | jq -r '.number')
[ -n "$PR" ] && [ "$PR" != "null" ] || { echo "PR creation failed"; exit 1; }
echo " PR #$PR"
el "waiting for the gate (expect ~1m40s)"
for i in $(seq 1 60); do
st=$(api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/commits/${SHA}/status" | jq -r '.state')
printf '\r gate=%s [T+%ss] ' "$st" "$(( $(date +%s)-T0 ))"
case "$st" in success) echo; break;; failure|error) echo; echo "GATE FAILED — stopping."; exit 1;; esac
sleep 5
done
[ "$st" = "success" ] || { echo; echo "gate did not finish in time"; exit 1; }
el "merging PR #$PR"
api -X POST -H 'Content-Type: application/json' -d '{"Do":"merge"}' \
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls/${PR}/merge" >/dev/null
git checkout -q main && git pull -q --ff-only origin main
el "forcing the Flux reconcile (--with-source, or it applies the OLD revision)"
./infra/devcontainer/dev.sh run bash -c \
'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-0 --with-source'
# MANDATORY here. Only the Secret ciphertext changed, so the Deployment pod template is
# byte-identical and NOTHING restarts on its own. PLEX_CLAIM is read once, at container
# start. Skipping this is failure mode #2 in runbooks/plex-claim.md.
el "rolling the pod (mandatory — a changed Secret restarts nothing)"
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex rollout restart deployment/plex && \
kubectl --context k3s-prod -n plex rollout status deployment/plex --timeout=120s'
el "verdict"
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex logs deploy/plex --since=3m | grep -i -e claim -e token || true'
cat <<'NOTE'
"Attempting to obtain server token..." + "Token obtained successfully" -> CLAIMED
"Attempting..." with no success line -> token expired; retry, nothing is broken
neither line -> already claimed, or env did not reach the container
Confirm it persisted (prints the attribute NAME only, never its value):
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c \
"grep -o PlexOnlineToken= \"/config/Library/Application Support/Plex Media Server/Preferences.xml\" | head -1"'
NOTE
+8
View File
@@ -34,6 +34,14 @@ resources:
# policies, the next import is blocked in the gap.
- ../platform/kyverno/policies/label-cdi-worker-pods.yaml
- ../platform/kyverno/admission/secret-provenance.yaml
# R-SEC-02 — bounds what the conformance CI trigger may create. MUST be listed HERE: this
# Kustomization is what actually applies ClusterPolicies. Adding a policy only to
# platform/kyverno/admission/kustomization.yaml is a SILENT NO-OP — that file is reference
# only ("kept as reference; THIS file is the buildable render root" in
# platform/kyverno/kustomization.yaml, which itself ships only the namespace + HelmRelease).
# That is exactly how this policy shipped green and never reached the cluster on 2026-09-29,
# leaving the conformance-trigger identity live with nothing bounding it.
- ../platform/kyverno/admission/conformance-trigger-job-shape.yaml
# cilium: L2 LB IP pool + announcement policy (cilium.io CRDs already registered by Cilium)
- cilium/l2-ip-pool.yaml
- cilium/l2-announcement-policy.yaml
+317 -10
View File
@@ -21,7 +21,7 @@ Cilium L2 LoadBalancer (game protocols authenticate players themselves).
| PVC subPaths | `servers`→`/crafty/servers`, `backups`→`/crafty/backups`, `logs`→`/crafty/logs`, `import`→`/crafty/import`, `config`→`/crafty/app/config` |
| Identity | `runAsUser:1000 runAsGroup:0 fsGroup:0 runAsNonRoot:true` (see below) |
| Web host | `crafty.home.keithjackson.dev` → shared Gateway → `crafty` Service :80 → sidecar :4180 → Crafty :8443 |
| Game VIP | **192.168.1.55** (Cilium L2 LoadBalancer; `.53`=Gateway, `.54`=unifi, `.55`=crafty) |
| Game VIP | `${CRAFTY_L4_VIP}` (Cilium L2 LoadBalancer) — **prod `192.168.2.22`** (live `EXTERNAL-IP`, measured 2026-09-29); base/dev `192.168.1.55` |
| Game ports | starter set: **TCP 25565-25567** (Java), **UDP 19132-19134** (Bedrock) |
### fsGroup / user decision (the seerr lesson, applied)
@@ -76,15 +76,16 @@ can't span namespaces). Both are age-encrypted to the repo `sops-age` recipient
## Game-port L4 exposure
The Gateway is L7-HTTPS-only; raw TCP/UDP game ports can't ride it. `service-l4.yaml` exposes them as
two `type=LoadBalancer` Services (TCP + UDP) sharing the pinned VIP **192.168.1.55**
two `type=LoadBalancer` Services (TCP + UDP) sharing the pinned VIP `${CRAFTY_L4_VIP}` (**prod
`192.168.2.22`**)
(`lbipam.cilium.io/sharing-key: crafty`), ARP-announced by the existing `default-l2` policy. This path
is **NOT auth-gated** — Minecraft Java (Mojang/Microsoft) and Bedrock (Xbox Live) authenticate players
themselves. `external-dns` publishes `crafty-mc.home.keithjackson.dev → 192.168.1.55`.
themselves. `external-dns` publishes `crafty-mc.home.keithjackson.dev → ${CRAFTY_L4_VIP}`.
### Adding more game ports (when the operator creates servers)
The operator has **not created any servers yet**; the starter set covers 3 Java + 3 Bedrock slots. To
add capacity:
As of 2026-09-29 **two** servers exist (see *Runtime state* below): Java `homelab` on 25565 and
Bedrock `K3s-Bedrock` on 19132. The starter set covers 3 Java + 3 Bedrock slots. To add capacity:
1. In the Crafty UI, create the server and bind it to a NEW port (Java TCP or Bedrock UDP).
2. Add that port to the matching Service in `service-l4.yaml` (one
@@ -93,6 +94,274 @@ add capacity:
4. (Optional) add a `containerPort` in `deployment.yaml` for documentation.
5. Commit → Flux reconciles.
### Bedrock on UDP 19132 — the transport trap, and the ERROR banner that is NOT a refusal (MEASURED on prod 2026-09-29)
Two traps in sequence. The first hides the UDP socket; the second frightens you into undoing the fix.
As of 2026-09-29 21:34 this server is **reachable on `19132/udp` and answering**, measured from
off-cluster. Read both traps before changing anything.
#### Trap 1 — `transport=nethernet` (the shipped default) opens no RakNet UDP listener
Measured 20:54–21:32 against `K3s-Bedrock` (`c5837301-12b1-404e-b50e-70eafb37226b`, DB
`type=minecraft-bedrock`, `server_port=19132`, process `./bedrock_server` alive):
- `/proc/net/udp` and `/proc/net/udp6` held **no socket on 19132** (`0x4ABC`) at all. Port 19132 was
bound as **TCP** (`/proc/net/tcp6`, `…:4ABC`, state `0A`/LISTEN), not UDP.
- That is nethernet working as designed, and the **shipped `bedrock_server_how_to.html`** — the
vendor's own doc, sitting on the PVC beside the binary — says exactly why: nethernet *"listens on a
single dual-stack **TCP** socket on `server-port` for an HTTP-based signaling handshake, then
negotiates a UDP connection with each client that carries the gameplay traffic."*
- **So nethernet could never have worked here anyway, on two independent counts:** `service-l4.yaml`
exposes 19132 **only in the UDP Service** (`crafty-l4-tcp` carries 25565-25567 and nothing else), and
`crafty-allow-ingress` allows `19132 UDP` and **not** `19132 TCP`. The signaling port was unreachable
at both the Service and the policy layer. The server's own `packet-statistics.txt` for that run
agrees: `Total Received: 0, 0B` over 2293 s.
- Crafty does not manage this property (`transport` appears nowhere under `/crafty/app/classes/`) and
its *only* Bedrock probe is `raknet_ping.py`, so the panel logged `WARNING
app.classes.remote_stats.raknet_ping - Got timeout while issuing bedrock ping to 127.0.0.1:19132`
every ~5 s — **155 such lines** — against a server it had started successfully.
#### Trap 2 — under `transport=raknet` this build prints a scary ERROR banner and then works anyway
The operator set `transport=raknet` at 21:33:31 (`/crafty/logs/audit.log`: `Edited file
/crafty/servers/c5837301-…/server.properties`) and started the server at 21:33:58. BDS **1.26.52**
printed, at ERROR level:
```
[ERROR] ================ TRANSPORT TYPE ERROR ===================
[ERROR] Your current connection type is not set to NetherNet. In this release, NetherNet is the only
supported transport type.
[ERROR] Players will not be able to connect to your game without NetherNet.
[ERROR] To switch, set 'transport=nethernet' in server.properties.
```
**That banner is advisory, not fatal. Do not revert `transport` on seeing it.** Every claim it makes
about connectivity was false on this cluster within five seconds of it being printed:
- The process **kept running** (`./bedrock_server`, pid 47006) and **bound the RakNet sockets**:
`/proc/net/udp` `00000000:4ABC` (19132, IPv4) and `/proc/net/udp6` `…:4ABD` (19133, IPv6), both owned
by `bedrock_server` — `/proc/47006/fd/25` and `fd/26` → those socket inodes. The earlier **TCP** bind
on `4ABC` is **gone**.
- Crafty's warning **changed kind** at that instant: last `Got timeout` at `21:34:04,306`, first
`Bedrock ping returned a string with too few fields` at `21:34:09,854`. That distinction is
load-bearing — read `raknet_ping.py`: `Got timeout` is emitted *only* when `sock.recv` raises
`TimeoutError` (nothing came back at all), whereas `too few fields` is reachable *only after* a
`0x1C` Unconnected Pong has arrived **and its magic has validated**. The server is answering pings.
- **End-to-end, from off-cluster:** a RakNet Unconnected Ping sent from the LAN to the prod VIP
`192.168.2.22:19132` returns a 139-byte `0x1c` Unconnected Pong in ~12 ms, magic valid:
MCPE;Dedicated Server;2193;1.26.52;0;10;11676629557768398479;Bedrock level;Survival;1;19132;19133;0;0;0;
Edition `MCPE`, protocol `2193`, version **`1.26.52`**, `0/10` players. **Players can reach this
server now**, through the VIP, the Service, and the CiliumNetworkPolicy, with no manifest change.
- The vendor doc on disk contradicts the banner and matches the observed behaviour. `transport` is
documented as *"Allowed values: raknet, nethernet"*, and: *"**raknet** — The traditional Bedrock UDP
transport. The server listens directly on `server-port` (IPv4) and `server-portv6` (IPv6), and
clients connect to that address and port."* RakNet is also still fully compiled into the binary —
`strings bedrock_server` yields `RakNetConnector`, `createUniqueRakPeer`, `RakPeerInterface` and the
vendored build path `…/src-deps/raknet/raknet/RakNetSocket2.cpp`.
So *"NetherNet is the only supported transport type"* is a **support statement, not a capability
statement**: Microsoft declines to support RakNet and ships it working. Treat the banner as a
deprecation notice with an **unknown expiry** — a future BDS may genuinely remove RakNet, and that is
the day this exposure needs GeyserMC (see *If RakNet is ever really removed* below).
> **The shipped-comment trap, stated plainly.** `server.properties` lists `raknet` as allowed and the
> binary logs an ERROR saying it is not supported. Both are the vendor, and they disagree. The tiebreak
> is neither prose: it is `/proc/net/udp` and a pong on the wire. Whenever this file says a Bedrock
> server is or is not reachable, it is because someone pinged it — not because a comment said so.
#### The steady-state `too few fields` warning is benign and is now the expected signature
Crafty will log `Bedrock ping returned a string with too few fields - missing values [None]` every
~5 s forever. **That means the ping SUCCEEDED.** BDS 1.26.52's pong carries 16 `;`-separated fields;
Crafty 4.10.8's `pingpong_string_fields` table knows 14 and expects the last to be an empty
terminator. It instead finds `'0'`, so `last_enumeration` stops at index 12 and the parser reports the
one `None` terminator entry as missing. Every field that matters (edition, MOTD, protocol, version
name, player count, max players, uuid, game mode, both ports) parsed correctly. **Do not chase this
warning.** The failure signature is the *other* message, `Got timeout`.
### "You are not invited to play on this server" — the join gate is the allowlist, not the network (MEASURED 2026-09-29 21:56; **RESOLVED and confirmed 22:03**)
A Bedrock client reported **"Unable to connect / You are not invited to play on this server"**. That is
an **authorization** refusal, and it is the expected, guaranteed behaviour of this server's current
configuration. Measured:
| Property / file | Value | Source |
|---|---|---|
| `allow-list` | **`true`** (line 37) | `server.properties` |
| `allowlist.json` | **`[]`** — 3 bytes, mtime 20:51, never edited since server create | `cat allowlist.json` |
| `online-mode` | `true` (line 31) | `server.properties` |
| `transport` | **`raknet`** (line 71), file mtime **still 21:33:31** | `server.properties` |
| TCP 19132 listener | **none** (`NO-TCP-4ABC`) | `grep 4ABC /proc/net/tcp /proc/net/tcp6` |
| UDP 19132 listener | present, `00000000:4ABC`, pid 47006 | `grep 4ABC /proc/net/udp` |
**`allow-list=true` with an empty `allowlist.json` refuses every player, unconditionally.** The shipped
`bedrock_server_how_to.html` states it: *"If the `allow-list` property is enabled in server.properties
then the server will only allow selected users to connect. To allow a user to connect you need to know
their Xbox Live **Gamertag**."* An empty list selects nobody. This is not a network problem, not a
transport problem, and not a platform problem — and no amount of Service or policy work will change it.
#### CONFIRMED: one gamertag added, and a player joined
**This is no longer a hypothesis. The fix was applied and it worked.** *(The join itself is
**operator-reported** — 2026-09-29, the operator added the player's gamertag and the Bedrock client
connected and joined. I did not watch it.)* Two independent things corroborate it **directly on disk**:
- `allowlist.json` went from `[]` to **one entry**, and that entry's **`xuid` is populated**
(`keys=['ignoresPlayerLimit','name','xuid']`, file 75 bytes, mtime **22:03:17**). Per the vendor doc,
`xuid` is *"Optional. If it's not set then it will be populated when someone with a matching name
**connects**."* The operator wrote only the name — `audit.log` records `Edited file
…/allowlist.json` at **22:02:13** — so **the XUID was written by the server, on connect, at 22:03:17**.
That is server-authored evidence of an Xbox-Live-authenticated join. *(Observed: the file state and
both timestamps. Reasoned from the vendor doc: that a populated XUID implies a connect.)*
- The RakNet path still answers after the restart this involved: a pong from `192.168.2.22:19132` at
22:06, `MCPE;…;1.26.52;0;10;…` (`players=0` — the player had left by then), new `server_guid` matching
the new pid.
**So the causal chain is closed: `allow-list=true` + `allowlist.json == []` was the wall, and nothing
else was.** The earlier mapping of the client's *"You are not invited"* string to the empty allowlist was
**reasoned** from the vendor doc (there is no BDS log on disk to read it from); adding exactly one
gamertag and getting exactly one successful join **confirms that mapping**. No transport change, no
manifest change, and no network change was involved in the fix.
> **Note the route the operator actually used:** they **edited `allowlist.json` directly** and then
> **restarted** the server (`audit.log`: edit 22:02:13 → `stop_server` 22:02:37 → `start_server`
> 22:02:41; pid 47006 → **47875**). That is route 2 below, and a restart is a valid substitute for
> `allowlist reload`. Route 1 (`allowlist add` in the console) avoids the restart and is still preferred.
**This is also positive evidence that the RakNet path works end to end.** A refusal authored by *our*
server required the client to reach our server and get an answer back. The only listener on 19132 is
the **RakNet UDP** socket — there is **no TCP 19132 listener at all**, and NetherNet requires a TCP
signalling socket on `server-port`. So the client cannot have used NetherNet against this server; it
arrived over `19132/udp`, through `crafty-l4-udp` and the VIP. **The 19132-19134/UDP exposure is not
inert — it is the path that carried both the pong and this refusal.** *(Observed: the listeners, the
absent TCP socket, `transport=raknet`. Reasoned from those: that the refusal traversed that path. Not
captured: the actual player flow — see the caveat below.)*
#### Caveat: BDS console output is NOT on disk, so a join attempt cannot be correlated from the filesystem
`K3s-Bedrock`'s `log_path` in `crafty.sqlite` is the **empty string**, and `Dedicated_Server.txt` is
**0 bytes**. Crafty does not persist a Bedrock server's stdout anywhere on the PVC — the console exists
only in the panel UI. `commander.log` between 21:39 and 21:56 contains nothing but `homelab`
crash-detection lines and the `raknet_ping` chatter: **no join, no gamertag, no XUID, no allowlist
denial.** That absence is **not** evidence about where the refusal came from; it is evidence that
Crafty logs no BDS stdout at all. **To see a rejected join you must watch the panel console live.**
UNVERIFIED, and only observable in the UI.
#### How a Bedrock player actually gets authorized here
Identity is an **Xbox Live Gamertag** (`online-mode=true` forces XBL auth, which is what gives BDS a
gamertag/XUID to check). `allowlist.json` is a JSON array; per the vendor doc the keys are `name` (the
gamertag), optional `xuid` (*"populated when someone with a matching name connects"*), and
`ignoresPlayerLimit`. So the operator adds **one entry per player, keyed on gamertag** — there is no
group, role, or Keycloak integration, and **none of this touches the oauth2-proxy/Keycloak gate**,
which protects only the web panel.
Three ways to do it, all **operator actions**. **Routes 1 and 2 are both proven** — route 2 is the one
that produced the confirmed join above:
1. **Preferred — the panel console.** In Crafty, open `K3s-Bedrock` → Console and send:
`allowlist add <Gamertag>` (quote it if it contains a space: `allowlist add "Example Name"`).
This writes `allowlist.json` **and** takes effect immediately, with no restart.
2. **Edit the file, then reload.** Write `/crafty/servers/c5837301-…/allowlist.json`, then send
`allowlist reload` in the console — *"After you've modified the file you need to run the command
`allowlist reload` to make sure that the server knows about your new change."* Shape:
```json
[
{ "ignoresPlayerLimit": false, "name": "FirstPlayerGamertag" },
{ "ignoresPlayerLimit": false, "name": "SecondPlayerGamertag" }
]
```
3. **Turn the allowlist off entirely** — `allow-list=false` in `server.properties` (restart), or
`allowlist off` in the console. Note the vendor's warning: `allowlist on|off` *"does not change the
value in the server.properties file"*, so a console `off` silently reverts on the next start.
**Not recommended:** the game port is deliberately un-gated at the network layer (see *Game-port L4
exposure*), so the allowlist is the **only** thing deciding who may join. Keep it `true` and enumerate
players.
> **`allowlist.json` is application state on the PVC**, exactly like `transport=raknet`. Neither is in
> git and **Flux can restore neither**. Be precise about what breaks them, because the two cases differ:
>
> - **A pod or server *restart* preserves both** — observed: the 22:02:48 restart (pid 47006 → 47875)
> kept `transport=raknet` (`server.properties` mtime unchanged at 21:33:31) and kept the allowlist
> entry. What a **pod** restart *does* break is that `K3s-Bedrock` has `auto_start=0`, so the Bedrock
> server simply **does not come back** until someone starts it by hand.
> - **A server *recreate* or reinstall silently resets both** — a fresh `server.properties` ships
> `transport=nethernet` (nothing bound on 19132/udp) and a fresh `allowlist.json` is `[]` (everyone
> refused). Neither change announces itself, and the server will look healthy and started in the panel
> while being unreachable *and* unjoinable.
>
> **Treat both files as backup-relevant, and re-run the 5-step connect procedure above after any Bedrock
> recreate or BDS update.**
#### Why this does not change the transport recommendation
`transport=raknet` is correct, and with the allowlist entry in place the combination is **proven
end-to-end by a player who has joined** — not merely the best-reasoned option. The allowlist refusal was
orthogonal to transport all along: it would have refused players identically under NetherNet. Switching
to NetherNet would *add* work (TCP 19132 in `crafty-l4-tcp` **and** in `crafty-allow-ingress`, plus
`server-udp-ports` pinned and exposed) and would admit **not one extra player**. The ranking below stands,
and option 1 is now settled rather than provisional.
#### Decision (PROVEN, not provisional): `transport=raknet` stays; the 19132-19134/UDP exposure is correct and live
No manifest change is needed or wanted. `service-l4.yaml`, `networkpolicy.yaml` and the
`containerPort`s were already shaped for exactly RakNet, and that shape is now **proven by a player who
has connected and joined over `192.168.2.22:19132`** — not assumed, and not merely reasoned. The options
that would have required manifest work are all **not recommended**:
| Option | Manifest consequence | Verdict |
|---|---|---|
| **Keep `transport=raknet`** + allowlist entries (**this is what works**) | **none** — the existing Service, policy and ports were already correct | **Done; proven.** A player has joined over the VIP. Zero manifest change, vanilla parity, real Bedrock world. |
| **GeyserMC + Floodgate** on the `homelab` Paper server | none either (Geyser also binds `19132/udp`), but it would **collide** with the live BDS on 19132 — one of them must move | Hold in reserve. Only worth it if RakNet is really removed, or if you *want* one shared world. Cost: Bedrock players join the **Java** world, not a Bedrock world; Java-only mechanics and some content differ; it is another moving part to patch. |
| **Pin/downgrade BDS** to a pre-nethernet release | none | Unnecessary — the *current* build already speaks RakNet. Also fragile: Microsoft removes old BDS downloads, and Crafty's Bedrock installer tracks latest. UNVERIFIED whether Crafty can pin a version at all. |
| **A third-party server** (PocketMine-MP, Nukkit) | none | No. Loses vanilla parity and the world format; a large change to solve a problem that is not occurring. |
| **Accept NetherNet** | 19132-19134/UDP becomes **inert**; nethernet would instead need **TCP** 19132 added to `crafty-l4-tcp` *and* to `crafty-allow-ingress`, plus `server-udp-ports` pinned and exposed for the negotiated media path | No — and note it is **not currently in use**: `transport=raknet` and there is no TCP 19132 listener. Strictly more manifest work than the option that already works, it admits no extra player (the allowlist is the gate), and WebRTC media behind a shared VIP and SNAT is unproven here. |
#### If RakNet is ever really removed
Then BDS opens no UDP listener and this exposure has nothing to bind to — an architecture problem, not
a config one. The migration is **GeyserMC + Floodgate as a plugin on the existing `homelab` Paper
server**, because Geyser speaks RakNet on `19132/udp` itself: the VIP, the Service, the policy and the
`containerPort`s all stay **byte-for-byte as they are**, and Bedrock clients keep connecting to
`crafty-mc.home.keithjackson.dev:19132`. What changes is the world: Bedrock players land in the Java
world via Floodgate (no Xbox-authenticated Bedrock account needed on the Java side), the Bedrock server
is retired, and `K3s-Bedrock` stops the day Geyser starts — **they cannot share 19132**. Installing it
is a PVC write plus a Paper restart, i.e. an **operator action**: drop `Geyser-Spigot.jar` and
`floodgate-spigot.jar` into `/crafty/servers/13c7b85d-…/plugins/` and restart `homelab`, then set
Geyser's `bedrock.port: 19132` and `clone-remote-port: false` in its generated `config.yml`.
Geyser's own Bedrock-version support is **external and unmeasured here** — check that it supports the
Bedrock client version in use before committing.
### Bedrock realities that are NOT platform defects but present identically
- **LAN / Friends-tab discovery can never work here.** Bedrock discovery is a UDP broadcast on 19132
inside one L2 segment; this server sits behind a LoadBalancer VIP (and SNAT), not on the LAN, so it
will **never** appear in a client's Friends/LAN list. Players must use **Add Server** and type the
address and port by hand: `crafty-mc.home.keithjackson.dev` (or the VIP) with port **19132**.
`external-dns` publishes that name from the **-tcp** Service, so it resolves to the shared VIP and
is correct for Bedrock too — only the port differs from Java's 25565.
- **Consoles cannot reach this at all.** Switch/Xbox/PlayStation Bedrock clients have no
arbitrary-server-address field; only Windows/mobile/Fire do. Reaching it from a console needs a DNS
redirect of a featured-server hostname, which is out of scope for `service-l4.yaml`.
- **`enable-lan-visibility=true` (the shipped default) collides with Bedrock slots 1 and 2 — now
OBSERVED, not merely documented.** Under `transport=raknet` it makes a server bind the **default**
ports 19132 *and* 19133 even when `server-port`/`server-portv6` say otherwise; the shipped
`bedrock_server_how_to.html` says so, and on 2026-09-29 a RakNet ping to
`192.168.2.22:19133` returned **the same pong, from the same `server_guid`, as `:19132`** — one
server occupying two of the three Bedrock slots. `:19134` correctly timed out. So a second Bedrock
server put on 19133 will fight the first. Set `enable-lan-visibility=false` on every Bedrock server
after the first; LAN discovery cannot work here anyway (above).
- **`server-portv6` is the IPv6 listener and is not exposed; `19133` is nonetheless in use.**
`service-l4.yaml` is IPv4 single-stack, so the *IPv6* socket `…:4ABD` that BDS opens for
`server-portv6=19133` is unreachable from the LAN. But because `enable-lan-visibility=true` (above)
the server *also* answers on **IPv4 19133**, which the UDP Service does expose — measured. So
`19133` in `server.properties` genuinely does mean "slot 1 is taken", contrary to what this file
previously said.
## Policies
`namespace.yaml` sets only `gateway-access=shared` (platform namespace, like headlamp/homepage), so
@@ -114,14 +383,48 @@ Gateway VIP). The sidecar→Crafty hop is pod-local (`127.0.0.1`) and not subjec
local admin. A **non-platform-admins** user → **403** (authenticated but not authorized).
2. **Pod health / fsGroup:** `kubectl -n crafty get pod` Ready 2/2; Crafty writes its DB under
`/crafty/app/config` and `/crafty/servers` without EACCES.
3. **Server-create + game reach:** in the Crafty UI create a Java server bound to `25565`; start it;
connect a Minecraft client to `crafty-mc.home.keithjackson.dev:25565` (or `192.168.1.55:25565`)
from the LAN and confirm the world loads. (Bedrock: a server on `19132/udp` reached at the VIP.)
3. **Server-create + game reach (Java):** in the Crafty UI create a Java server bound to `25565`;
start it; connect a Minecraft client to `crafty-mc.home.keithjackson.dev:25565` (or
`192.168.2.22:25565`) from the LAN and confirm the world loads.
4. **Bedrock: the connect procedure (VERIFIED end-to-end 2026-09-29 — a player has joined).**
Creating and starting the server is **not sufficient**. Do these in order; each step has a distinct
failure signature:
1. **`transport=raknet`** in `server.properties`. The vendor default is `nethernet`, which opens
**no UDP listener at all**.
2. **A live UDP listener:** `grep 4ABC /proc/net/udp /proc/net/udp6` → a wildcard bind. Under
`nethernet` you instead see `4ABC` on **TCP**, which nothing here exposes.
3. **A pong from off-cluster:** a RakNet Unconnected Ping to `192.168.2.22:19132` returns a `0x1c`
packet whose MOTD string starts `MCPE;`. This proves the VIP, Service and policy path.
4. **An allowlist entry for the player's Xbox Gamertag.** `allow-list=true` with
`allowlist.json == []` refuses **every** player with *"You are not invited to play on this
server"* while the server is otherwise perfectly reachable. In the panel console:
`allowlist add <Gamertag>`.
5. **Add Server** by hand in a Windows/mobile client: `crafty-mc.home.keithjackson.dev`, port
`19132`. It will never appear in the LAN/Friends tab, and a console cannot reach it at all.
**Two log messages appear on a fully working server and must be ignored:** BDS's
`================ TRANSPORT TYPE ERROR ===================` banner (advisory — see *Trap 2*), and
Crafty's perpetual `Bedrock ping returned a string with too few fields`. Neither indicates a fault;
chasing them is what cost two sessions. The **real** failure signatures are `Got timeout while
issuing bedrock ping` (no listener) and *"You are not invited"* (allowlist).
## Runtime state: the servers themselves are NOT in git
As of 2026-09-14 one Java server exists — **"homelab"**, Paper 26.2, TCP 25565, 2 GiB heap, autostart
on, UUID `13c7b85d-00bb-4a58-8c86-2433ca47891c`. It was created through Crafty's v2 API, and **Mojang's
As of 2026-09-29 **two** servers exist (both measured live, `select * from servers`):
| Name | Type | Port | UUID | `auto_start` |
|---|---|---|---|---|
| `homelab` | `minecraft-java` (Paper 26.2, 2 GiB heap) | TCP 25565 | `13c7b85d-00bb-4a58-8c86-2433ca47891c` | **1** |
| `K3s-Bedrock` | `minecraft-bedrock` | UDP 19132 | `c5837301-12b1-404e-b50e-70eafb37226b` | **0** ⚠ |
`K3s-Bedrock` was created 2026-09-29 20:51 and **still** has `auto_start=0` **and**
`crash_detection=0` — re-measured 2026-09-29 21:41 straight out of `crafty.sqlite` (read-only), so the
create-API trap documented at the bottom of this file is still live and still unfixed. **It will not
come back after a pod restart**, and merging any change to `deployment.yaml` restarts the pod
(`strategy: Recreate`). PATCH both to true and read them back.
`homelab` was created through Crafty's v2 API, and **Mojang's
EULA (<https://aka.ms/MinecraftEULA>) was accepted on the operator's behalf** to let it start; see the
DECISIONS-LOG entry of that date.
@@ -132,6 +435,10 @@ application state on the `crafty-data` Longhorn PVC, in two places, and **Flux c
heap sizes, `auto_start` / `crash_detection`.
- `/crafty/servers/<uuid>/` (`servers` subPath) — `paper.jar`, `eula.txt`, `server.properties`, and
**the generated world**. The jar is re-downloadable; the world is not.
**`transport=raknet` lives here too.** It is the one setting that makes Bedrock reachable at all (see
*the transport trap*), it is not in git, Flux cannot restore it, and a freshly created or reinstalled
Bedrock server gets the vendor default `transport=nethernet` back — i.e. it silently regresses to
"nothing bound on 19132/udp". Re-apply and re-ping after any Bedrock server recreate or BDS update.
So a rebuild from git alone yields a healthy but **empty** Crafty with nothing listening on 25565.
Recreating a server by hand means: create it in the UI or API, accept the EULA again, and PATCH
@@ -0,0 +1,144 @@
# Serves: R-SEC-02 (CI holds no broad cluster access)
#
# THIS POLICY IS THE SECURITY BOUNDARY. The RBAC in conformance/incluster/rbac.yaml is not.
#
# Kubernetes places NO `use` check on a Pod/Job's `serviceAccountName`. Any principal that can
# `create jobs` in namespace `conformance` can therefore create a Job that runs as
# `conformance-runner` — a privileged SA (nodes/patch, pods/exec, namespaces/patch) — with an
# arbitrary command. So the CI trigger credential, however narrow its own Role looks, IS the
# privileged identity unless something constrains the SHAPE of what it may create.
#
# That is what this does: for Jobs created by `conformance-trigger` ONLY, admit exactly the
# committed conformance run and deny everything else. The operator chose this model
# (docs/plans/l2-l3-conformance-ci-wiring.md §4, primary model) over the zero-credential
# GitOps-native alternative.
#
# ENFORCE AND FAIL-CLOSED, NOT SUBSTITUTED. Note this does NOT use
# ${KYVERNO_VALIDATION_ACTION} like the other admission policies: a security boundary that an
# environment overlay can downgrade to Audit is not a boundary. `failurePolicy: Fail` means a
# Kyverno outage blocks conformance Jobs rather than silently admitting unconstrained ones.
# Blocking CI is the correct failure direction here; admitting arbitrary privileged code is not.
#
# SCOPE: it constrains ONLY `conformance-trigger`. Jobs created by humans, by Flux, or by any
# other SA are untouched — so a broken policy cannot wedge the rest of the cluster.
#
# IMAGE PINNING — HONEST LIMIT, READ BEFORE RELYING ON THIS. The plan specifies pinning the
# exact `CONFORMANCE_RUNNER_IMAGE` digest. That image does not exist yet, so this enforces the
# enforceable half: the image must live under the conformance repository path AND be
# digest-pinned (`@sha256:`), never a mutable tag. Residual surface: anyone who can PUSH to that
# registry path could publish a different digest. TIGHTEN THIS when the image is built by
# replacing the `image` pattern below with the literal digest — a one-line change, reviewed in a
# PR, which is exactly where a change to the security boundary belongs.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: conformance-trigger-job-shape
annotations:
policies.kyverno.io/title: Constrain what the conformance CI trigger may create
policies.kyverno.io/category: Supply chain
policies.kyverno.io/severity: critical
homelab.keithjackson.dev/requirements: R-SEC-02
homelab.keithjackson.dev/scope: admission-only
spec:
validationFailureAction: Enforce
failurePolicy: Fail
background: false
rules:
- name: trigger-may-create-only-the-committed-conformance-run
# resources AND subjects in the SAME filter entry — that is how Kyverno ANDs them.
# As two separate `all` entries it is a schema error ("match resources not specified")
# and the whole policy loads as INVALID, which the CLI reports as a SKIP.
match:
all:
- resources:
kinds:
- Job
namespaces:
- conformance
subjects:
- kind: ServiceAccount
name: conformance-trigger
namespace: conformance
validate:
message: >-
conformance-trigger may create ONLY the committed conformance run: a single container,
digest-pinned conformance-runner image, serviceAccountName conformance-runner,
command ["./run.sh"], no host namespaces, non-root with privilege escalation disabled,
and only the `workspace` and `results` volumes. See
platform/kyverno/admission/conformance-trigger-job-shape.yaml.
pattern:
spec:
template:
spec:
serviceAccountName: conformance-runner
=(hostNetwork): "false"
=(hostPID): "false"
=(hostIPC): "false"
containers:
- image: "gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*"
command:
- "./run.sh"
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
=(privileged): "false"
- name: trigger-volumes-must-be-workspace-or-results
# resources AND subjects in the SAME filter entry — that is how Kyverno ANDs them.
# As two separate `all` entries it is a schema error ("match resources not specified")
# and the whole policy loads as INVALID, which the CLI reports as a SKIP.
match:
all:
- resources:
kinds:
- Job
namespaces:
- conformance
subjects:
- kind: ServiceAccount
name: conformance-trigger
namespace: conformance
validate:
message: >-
conformance-trigger Jobs may mount only the `workspace` and `results` volumes. An extra
volume is how a constrained pod reaches a host path, a secret, or another SA's token.
# foreach, NOT a `pattern` list: a Kyverno array pattern asserts "at least one element
# matches", which would happily admit a Job carrying `workspace` PLUS a hostPath. This
# asserts EVERY volume is one of the two allowed names.
foreach:
- list: "request.object.spec.template.spec.volumes"
deny:
conditions:
all:
- key: "{{ element.name }}"
operator: AnyNotIn
value: ["workspace", "results"]
- name: trigger-args-must-be-the-committed-flags
# resources AND subjects in the SAME filter entry — that is how Kyverno ANDs them.
# As two separate `all` entries it is a schema error ("match resources not specified")
# and the whole policy loads as INVALID, which the CLI reports as a SKIP.
match:
all:
- resources:
kinds:
- Job
namespaces:
- conformance
subjects:
- kind: ServiceAccount
name: conformance-trigger
namespace: conformance
validate:
message: >-
conformance-trigger Job args may contain only the conformance run's own flags
(--layer, --requirements, --include-scope, --strict-coverage) and their plain values.
Anything else — a shell, a path, a redirect — is denied.
foreach:
- list: "request.object.spec.template.spec.containers[0].args"
deny:
conditions:
all:
- key: "{{ regex_match('^(--layer|--requirements|--include-scope|--strict-coverage|[A-Za-z0-9][A-Za-z0-9,._-]*)$', element) }}"
operator: Equals
value: false
@@ -20,3 +20,7 @@ resources:
# --- secrets: the admission variant REPLACES the CI-only sops-block policy ---
# (../policies/disallow-plaintext-secrets.yaml is CI-only; see secret-provenance.yaml)
- secret-provenance.yaml
# --- R-SEC-02: constrain what the conformance CI trigger may create ---
# THE security boundary for CI cluster access. Enforce + fail-closed, deliberately NOT
# subject to ${KYVERNO_VALIDATION_ACTION}. See the file header.
- conformance-trigger-job-shape.yaml
@@ -0,0 +1,57 @@
# R-SEC-02 — the conformance CI trigger cannot escalate to arbitrary privileged execution.
#
# Each DENY case below is a real escalation path, not a style violation. The one that matters
# most is `evil-command`: correct SA, correct image, arbitrary command. That is the attack the
# whole model exists to stop, and RBAC alone cannot — Kubernetes places no `use` check on
# serviceAccountName, so `create jobs` in this namespace IS the privileged identity without
# this policy.
#
# user-info.yaml supplies the requesting identity. WITHOUT it the policy's `subjects` match
# never fires and every case trivially passes — a test that proves nothing.
apiVersion: cli.kyverno.io/v1alpha1
kind: Test
metadata:
name: conformance-trigger-job-shape
policies:
- ../../admission/conformance-trigger-job-shape.yaml
resources:
- resources.yaml
userinfo: user-info.yaml
results:
# ---- the committed run is ADMITTED ----
- policy: conformance-trigger-job-shape
rule: trigger-may-create-only-the-committed-conformance-run
kind: Job
resources: [good-run]
result: pass
- policy: conformance-trigger-job-shape
rule: trigger-volumes-must-be-workspace-or-results
kind: Job
resources: [good-run]
result: pass
- policy: conformance-trigger-job-shape
rule: trigger-args-must-be-the-committed-flags
kind: Job
resources: [good-run]
result: pass
# ---- THE escalation: runner SA + arbitrary command ----
- policy: conformance-trigger-job-shape
rule: trigger-may-create-only-the-committed-conformance-run
kind: Job
resources: [evil-command, evil-image, tag-not-digest, host-namespaces]
result: fail
# ---- an extra volume is the way out of a constrained pod ----
- policy: conformance-trigger-job-shape
rule: trigger-volumes-must-be-workspace-or-results
kind: Job
resources: [extra-volume]
result: fail
# ---- args beyond the committed flags ----
- policy: conformance-trigger-job-shape
rule: trigger-args-must-be-the-committed-flags
kind: Job
resources: [injected-args]
result: fail
@@ -0,0 +1,154 @@
# The committed shape — must be ADMITTED.
apiVersion: batch/v1
kind: Job
metadata:
name: good-run
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
args: ["--layer", "l2,l3", "--requirements", "R-NET-01"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
- name: results
emptyDir: {}
---
# THE ESCALATION: runner SA, but an arbitrary command. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: evil-command
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["/bin/sh", "-c", "kubectl get secrets -A"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
---
# An arbitrary image running as the privileged SA. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: evil-image
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: docker.io/library/alpine:latest
command: ["./run.sh"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
---
# A mutable TAG instead of a digest — republishable under the same reference. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: tag-not-digest
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner:latest
command: ["./run.sh"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
---
# Host namespaces + privilege escalation. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: host-namespaces
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
hostPID: true
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
securityContext:
runAsNonRoot: false
allowPrivilegeEscalation: true
volumes:
- name: workspace
emptyDir: {}
---
# An extra volume — the way out of a constrained pod. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: extra-volume
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
- name: hostroot
hostPath:
path: /
---
# Injected args beyond the committed flags. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: injected-args
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
args: ["--layer", "l2", ";", "curl", "http://evil/"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
@@ -0,0 +1,12 @@
# The requesting identity the policy keys on. Without this the `subjects` match never fires and
# every case below would trivially "pass" — proving nothing.
apiVersion: cli.kyverno.io/v1alpha1
kind: UserInfo
metadata:
name: conformance-trigger
userInfo:
username: system:serviceaccount:conformance:conformance-trigger
groups:
- system:serviceaccounts
- system:serviceaccounts:conformance
- system:authenticated
+10 -2
View File
@@ -6,7 +6,7 @@ Every alert rule in this platform carries an annotation of the form
runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"
46 of them, over the 35 names below. Until this directory existed, that path returned **HTTP 404** on the
46 of them, over 35 names. Until this directory existed, that path returned **HTTP 404** on the
live cluster: every alert that reached a human handed them a dead link at the exact moment they needed
help. These pages are what makes the link resolve.
@@ -33,7 +33,8 @@ negative is how a CLI quirk becomes real data loss. Write `backups.velero.io`,
`backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`,
`volumes.longhorn.io`, `nodes.longhorn.io`.
`git push origin main` **is** the deploy; there is no other apply step. Durable changes go through git.
Merging a PR into `main` **is** the deploy; there is no other apply step. `main` is protected, so a
direct push is rejected — durable changes go through a branch, a PR, and a green gate.
## Alerts you should not trust the silence of (measured 2026-09-13)
@@ -108,6 +109,13 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
- **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
every green signal missed.
### Operator procedures (no alert links here)
- **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only
applies after a `/config` wipe. The claim token is short-lived — read the countdown on
plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
Includes why a changed Secret does not restart anything, and why an expired token is harmless.
### Virtualisation
- **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.
- **`kubevirt-launch-failed`** — a VMI reached the Failed phase. The three k3s-specific launch traps.
+4 -2
View File
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -78,8 +78,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -55,8 +55,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -63,8 +63,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -87,8 +87,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -69,8 +69,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -71,8 +71,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -68,8 +68,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+5 -3
View File
@@ -7,7 +7,7 @@
## What this means here
Something in the GitOps chain is stuck at Ready=False. On this cluster that is more consequential than
it sounds, because **`git push origin main` IS the deploy** — there is no other apply step. A stuck
it sounds, because **merging a PR into `main` IS the deploy** — there is no other apply step. A stuck
Kustomization means git and the cluster have diverged and nothing will close the gap.
The chain has **14 tiers**: `flux-system` → `cilium` → `platform` → `platform-config` →
@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -58,8 +58,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -87,8 +87,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -67,8 +67,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -59,8 +59,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -57,8 +57,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+5
View File
@@ -81,6 +81,11 @@ configMapGenerator:
- node-io=node-io.md
- node-memory=node-memory.md
- node-memory-swap=node-memory-swap.md
# Operator procedure, not an alert page (no alert links to it — the
# alert-runbooks-resolvable check treats that as advisory). It is served
# because the plex.tv claim window is short and is not a thing to go looking
# for in git: /runbooks/plex-claim has to resolve the moment it is needed.
- plex-claim=plex-claim.md
- pvc-capacity=pvc-capacity.md
- targets-down=targets-down.md
- update-rollback=update-rollback.md
+4 -2
View File
@@ -83,8 +83,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -74,8 +74,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -75,8 +75,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -52,8 +52,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -49,8 +49,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -56,8 +56,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -56,8 +56,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -227,8 +227,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -52,8 +52,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+189
View File
@@ -0,0 +1,189 @@
# plex-claim — claiming the Plex Media Server inside the token window
**Alert:** none. This is an **operator procedure**, not an incident page — it is indexed here because
it is the page you need at the moment you need it, with a stopwatch running.
**Files:** `catalog/plex.yaml` (env `PLEX_CLAIM`), `apps/wave-0/plex-claim-secret.sops.yaml`
(the SOPS Secret `plex-claim`, key `PLEX_CLAIM`), `apps/wave-0/plex.yaml` (the rendered golden).
## What this means here
Plex will not let you **create libraries** over the mounted bulk media share while the server is
**unclaimed** — the 8.33 T NFS tree at `/media` is there, readable, and unusable. Claiming binds the
server to a plex.tv account. It is a **one-shot** operation: the token is exchanged once for a durable
`PlexOnlineToken` written into `Preferences.xml` on the `plex-config` Longhorn PVC, and from then on
the `PLEX_CLAIM` env is **ignored on every start** (upstream gate:
`[ ! -z "${PLEX_CLAIM}" ] && [ -z "${token}" ]` in `root/etc/cont-init.d/40-plex-first-run`).
**A `https://plex.tv/claim` token is short-lived — minutes, not hours.** Read the countdown the
claim page shows you; do not budget against a number written here. That is shorter than PR → gate →
merge → Flux's 10-minute reconcile interval → pod roll if you let those run on their own timers,
which is why this page exists: the loop is scripted and the reconcile is **forced** rather than
waited for. Measured end to end on 2026-09-29: **~2m45s**.
**An expired or wrong token does not break anything.** Verified against the image's own first-run
script (and the copy inside `plexinc/pms-docker:latest`): a failed exchange gets a 403 with an empty
body, `curl`'s status is never checked, the token `sed` yields empty, the `if` does not fire, there is
no `else` and no `exit 1`, and the script (no `set -e`) exits 0. The pod comes up **unclaimed and
serving**. So a burnt token costs you one more attempt, not an outage — and the repo deliberately
ships a **placeholder** in that Secret for exactly this reason (the env `secretKeyRef` has no
`optional`, so the Secret must always exist even when its value is useless).
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that
hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and that default has already changed once (k3s-dev until dev
was decommissioned on 2026-09-13, k3s-prod now). Do not run two `dev.sh` invocations at once.
Mutating `kubectl` and `flux reconcile` are blocked for agents here. **This whole page is operator
work** — the reconcile and the pod roll below are deliberate operator actions, and the durable change
is still a git change.
## First commands
**Before spending a token, find out whether the server is already claimed.** This prints only the
attribute NAME, never its value — do not widen it to print the token:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c \
"grep -o PlexOnlineToken= \"/config/Library/Application Support/Plex Media Server/Preferences.xml\" | head -1"'
```
`PlexOnlineToken=` printed ⇒ **already claimed**; stop, there is nothing to do and no token to burn.
Empty output ⇒ unclaimed, continue.
Is the workload healthy enough to claim at all, and is the wave reconciling?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex get deploy,pod,pvc'
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0'
```
## The procedure
> **STATUS: this server was CLAIMED on 2026-09-29** and `PlexOnlineToken` is present in
> `/config` on the `plex-config` PVC. The upstream exchange is gated on that pref being
> *absent*, so `PLEX_CLAIM` is now a **no-op on every start** whatever it holds. You only need
> this page again after a `/config` wipe or a rebuild onto an empty PVC.
**On the clock.** The token is short-lived — **read the countdown that plex.tv/claim shows you**
rather than trusting a number written here. (Earlier revisions of this page asserted a "~4-minute
TTL" in four places with no citation; upstream `pms-docker` documents no expiry at all. The page
is the only authority, and it is on screen at T+0.)
**A burnt token costs nothing.** An expired token gets a 403 with an empty body, `curl`'s status is
never checked, the `if` does not fire, there is no `else` and no `exit 1`, and the script (no
`set -e`) exits 0. The pod comes up unclaimed and serving. So retry freely.
### One command
1. **T+0 — get the token.** Sign in to plex.tv as the account that should OWN this server, open
<https://plex.tv/claim>, copy the `claim-…` string.
2. **Put it in the Secret** — in place, encrypted, never touching a plaintext file:
```
infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
```
3. **Ship it.** This drives the whole delivery loop and prints a running stopwatch:
```
./infra/plex/claim-ship.sh
```
branch → commit → push → PR → gate → merge → `flux reconcile --with-source` → `rollout restart`
→ verdict. It refuses to run if the file is unchanged or not encrypted.
**Measured end to end, 2026-09-29: ~2m45s.** Gate ~1m40s, merge + reconcile ~25s, pod roll and
container start ~30s.
### Why it is a script and not six pasted commands
`main` is a **protected branch** — the direct push this page used to instruct is now rejected — so
the deploy is a PR merged after a green gate. Three of those steps are easy to get wrong under time
pressure, and two are silent when skipped:
- **`--with-source` is load-bearing.** Without it the Kustomization applies the previously-fetched
revision, not your commit.
- **The pod roll is mandatory.** Only the Secret's ciphertext changes, so the Deployment's pod
template is byte-identical and *nothing restarts on its own*. `PLEX_CLAIM` is read once, at
container start. Skip this and the new Secret sits in the cluster while the running pod keeps the
old value forever.
- **The branch is named `feat/…` on purpose.** `act_runner` capacity is 1, and `gate.yml`'s push
trigger covers `feature/** fix/** docs/** chore/**` but *not* `feat/**`. A `chore/` branch fires a
second redundant run that serialises ahead of the PR run and roughly doubles the wall clock.
### Reading the verdict
* `Attempting to obtain server token from claim token` **and** `Token obtained successfully`
⇒ **claimed.** Re-run the First-commands `PlexOnlineToken=` check to confirm it persisted, then
create the libraries over `/media/movies`, `/media/tv`, `/media/photos`.
* `Attempting…` with **no** success line ⇒ the token was expired or invalid. Nothing is broken; go
back to step 1 with a new one.
* **Neither line** ⇒ the exchange was skipped: `/config` already had a `PlexOnlineToken` (already
claimed — the expected case now), or the env did not reach the container. Check by NAME only:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
-o jsonpath="{.items[0].spec.containers[0].env[*].name}"'
```
## Likely causes, most likely first
When the claim does not take, in order of how often it is the answer here:
1. **The token expired** — you exceeded the countdown, usually by reviewing the PR instead of
merging it the moment the gate went green. Symptom: the "Attempting…" line with no success line.
Remedy: a new token, and let `claim-ship.sh` drive the loop rather than pasting steps.
2. **The pod was never restarted** (step 5 skipped). Symptom: no claim log lines at all from a pod
whose `AGE` predates your push. The Secret is correct in git and in the cluster and the running
container still holds the placeholder.
3. **Flux had not applied the commit yet** when you rolled the pod — step 5 run before step 4
finished, so the new pod read the OLD Secret. Check
`kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0 -o jsonpath='{.status.lastAppliedRevision}'`
against your commit, then roll again.
4. **Already claimed** — the exchange is gated on `PlexOnlineToken` being absent from `/config`, so a
server claimed earlier ignores every later token silently. This is the benign case.
5. **`plex.tv` unreachable.** The upstream `curl` has no `--max-time`/`--connect-timeout`, so a stalled
connection can leave the container sitting in cont-init with **nothing listening on 32400** — the
one genuine hang risk in this path. Symptom: pod Running but never Ready, log stops after
"Attempting…". Check egress (`plex` is `egress: internet`) rather than re-rolling repeatedly.
6. **Wrong plex.tv account.** The claim succeeds and binds the server to whoever was signed in at
step 1. Unclaiming is a plex.tv-side operation on that account — see "escalate or stop".
## When to escalate or stop
- **Stop after two failed token attempts** and fix the timing, not the token. The third rushed attempt
is how people start editing live Secrets by hand, which leaves the cluster holding a value that is
in no git commit — precisely the drift `qbittorrent-bootstrap` had to be recovered from.
- **Never `kubectl edit`/`patch` the `plex-claim` Secret in the cluster, and never
`kubectl create secret` it.** Everything durable goes through the SOPS file; a live-only Secret is
reverted (or adopted with the wrong value) on the next reconcile.
- **Never `sops -d` this file into a terminal, a log, or a shared screen**, and never widen the
`PlexOnlineToken=` check into one that prints the value. The token is a plex.tv account credential.
- **If the server got claimed to the wrong account, stop and get the operator.** Reversing it means
removing the server from that account on plex.tv and clearing `PlexOnlineToken` from
`Preferences.xml` on the PVC — a destructive edit of app state on the replicated tier, not a git
change.
- **Do not delete the `plex-config` PVC to "start clean".** It holds the library database, metadata
and watch state (`policy: daily-offcluster`), and it is expensive to rebuild. The claim is a
two-line pref, not a reason to lose the DB.
- After a successful claim, **leave the expired token in the SOPS file**. It is inert (the exchange is
skipped once claimed) and the Secret must keep existing, because the env `secretKeyRef` has no
`optional` — deleting it would put the pod into `CreateContainerConfigError`.
---
Source of truth: `runbooks/plex-claim.md`. Index: `/runbooks/`
Upstream behaviour cited here is from `plexinc/pms-docker` branch `master`,
`root/etc/cont-init.d/40-plex-first-run` (s6-overlay v2), cross-checked against the same file inside
the running `plexinc/pms-docker:latest` image.
+4 -2
View File
@@ -58,8 +58,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -60,8 +60,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -54,8 +54,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands
+4 -2
View File
@@ -66,8 +66,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
## First commands