Compare commits
16
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d29a73cad8 | ||
|
|
07f03536b3 | ||
|
|
20ea70d221 | ||
|
|
8c7a770de2 | ||
|
|
1c5974013e | ||
|
|
4791d4a43a | ||
|
|
dfd608ac32 | ||
|
|
1d2d9d6549 | ||
|
|
4b31f97486 | ||
|
|
50a48e6f07 | ||
|
|
911ead5fbd | ||
|
|
8eb5baffa4 | ||
|
|
4634b44cc7 | ||
|
|
6586919132 | ||
|
|
e6f6e4febb | ||
|
|
e8e6b9c681 |
No files matched your search
@@ -110,9 +110,27 @@ jobs:
|
||||
# UNVERIFIED and NAMED in the PASS headline, never faked green. GATE_STRICT_SKIPS + the
|
||||
# tolerated exit 3 are explained in the file header; the tolerance covers exactly one
|
||||
# thing, the unverifiable spec repo, and nothing else can produce a 3 here.
|
||||
# GITEA_SETTINGS_TOKEN feeds ci/gitea-settings.sh, the only check that can see Gitea's
|
||||
# OWN database (branch protection + runner registration) — the blind spot that let `main`
|
||||
# sit unprotected with zero runners for a month while this repo recorded both as working.
|
||||
# Referenced BY NAME only; create it with infra/gitea/set-ci-token.sh.
|
||||
#
|
||||
# WIDENED TOLERANCE, DECLARED: `|| [ "$?" -eq 3 ]` used to cover exactly one skip
|
||||
# (spec-immutability). It now also covers gitea-settings skipping when that secret is
|
||||
# absent. That is a deliberate, temporary weakening: making an unwired credential BLOCK
|
||||
# every PR is the lock-yourself-out failure this repo keeps walking into. Once the secret
|
||||
# exists and the stage has been SEEN green, set REQUIRE_GITEA_SETTINGS=true below and the
|
||||
# tolerance narrows back to spec-immutability alone.
|
||||
- name: Merge gate — ci/gate.sh (what `make -C ci gate` runs)
|
||||
env:
|
||||
GATE_STRICT_SKIPS: "true"
|
||||
SETTINGS_API_TOKEN: ${{ secrets.SETTINGS_API_TOKEN }}
|
||||
# ENABLED 2026-09-29. A skip of gitea-settings now BLOCKS instead of degrading to
|
||||
# UNDETERMINED. Safe because THIS PR's own CI run is the proof: if the stage were
|
||||
# skipping, this line turns that into a failure and the PR goes red instead of
|
||||
# merging. Green here means the token resolved and the live Gitea matched the
|
||||
# declaration in infra/gitea/expected-branch-protection.json.
|
||||
REQUIRE_GITEA_SETTINGS: "true"
|
||||
run: ./ci/gate.sh || [ "$?" -eq 3 ]
|
||||
|
||||
# NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4
|
||||
|
||||
@@ -4964,3 +4964,502 @@ account rubber-stamping the human. That is ceremony, not review. Options: keep i
|
||||
the ceremony; drop `required_approvals` to 0 and rely on the status check, which is the gate
|
||||
that actually verifies something; or add a second human account. **This is a decision for the
|
||||
operator, not a default to be chosen silently.**
|
||||
|
||||
## 2026-09-29 (end of session) — **PM HANDOFF.** Six PRs open; read this before touching anything.
|
||||
|
||||
The session that began "set a SOPS secret" ended having rebuilt the delivery pipeline. Plex is
|
||||
claimed and scanning. `main` is genuinely protected and the gate genuinely enforces. What is
|
||||
open is listed below with the reason each is not merged.
|
||||
|
||||
### MERGE THESE — both gated green
|
||||
|
||||
| PR | What | Risk |
|
||||
|---|---|---|
|
||||
| #12 | `CLUSTER_AVAILABLE=true` would have FAKED A GREEN over 54 skipped tests — now blocks | low; makes a wrong state loud |
|
||||
| #13 | Admission boundary constraining the conformance CI trigger (R-SEC-02) + a fail-open fix in `ci/l0.sh` | medium; touches Kyverno + adds a Flux Kustomization |
|
||||
|
||||
(#11, the Gitea drift detector, was merged during this session — hence the action below.)
|
||||
|
||||
Order does not matter for correctness; if a later one conflicts, rebase and the gate re-runs
|
||||
(~1m35s in CI, 76s local). **Each was gated in the pinned toolbox, not on a host.**
|
||||
|
||||
### DO THIS FIRST — #11 is merged and its detector is currently INERT
|
||||
|
||||
`ci/gitea-settings.sh` SKIPS until its token exists, and **a skipped detector detects nothing**:
|
||||
|
||||
./infra/gitea/set-ci-token.sh # mints + stores GITEA_SETTINGS_TOKEN, never prints it
|
||||
|
||||
Then open any PR and confirm the gate prints `gitea-settings: green`. ONLY THEN uncomment
|
||||
`REQUIRE_GITEA_SETTINGS: "true"` in `.gitea/workflows/gate.yml`, so a future missing token blocks
|
||||
instead of skipping. Doing that before seeing it green locks the repo.
|
||||
|
||||
Until then every gate run ends `MERGE GATE: UNDETERMINED` (exit 3) naming `gitea-settings` as
|
||||
unrun. That is BY DESIGN and `gate.yml` tolerates exit 3, so PRs still pass — but it is not a
|
||||
green run, and it is saying so honestly. The token is what turns it green.
|
||||
|
||||
### A DIFFERENT COMPUTER NEEDS THREE THINGS THAT ARE NOT IN GIT
|
||||
|
||||
This is the part that will silently waste an hour otherwise. None of these are in the repo, by
|
||||
design, and without them the toolbox and the cluster are unreachable:
|
||||
|
||||
1. **`~/.config/sops/age/keys.txt`** — the age PRIVATE key. Without it NO `*.sops.yaml` can be
|
||||
read or edited, and `sops` will fail in a way that looks like a broken file. Copy it from this
|
||||
workstation; the public recipient in `.sops.yaml` is only half the pair.
|
||||
2. **`infra/ansible/.cache/prod.kubeconfig`** — untracked, so it exists only in the main checkout
|
||||
on this machine. Without it `dev.sh` refuses to start. In a git WORKTREE it is absent even
|
||||
here: use `DEV_KUBECONFIG=/dev/null` for anything that needs no cluster (see `CLAUDE.md` §3).
|
||||
3. **A Gitea credential** for `gitea.home.keithjackson.dev` (this session used the stored
|
||||
`~/.git-credentials`). `git push` and every `infra/gitea/*.sh` script needs it.
|
||||
|
||||
### STATE, MEASURED — do not re-derive it from this log, re-measure it
|
||||
|
||||
`CLAUDE.md` §2b exists because this log was confidently wrong for a month. Verify before relying:
|
||||
|
||||
protected:true user_can_push:false enable_status_check:true
|
||||
status_check_contexts:["gate / gate (pull_request)"] required_approvals:1
|
||||
block_admin_merge_override:true runner incluster-dind: online gate ~1m35s
|
||||
|
||||
### NEXT WORK, in priority order
|
||||
|
||||
1. **Finish the L2/L3 wiring** now that the credential model is decided and #13 lands the
|
||||
boundary. Still missing (`docs/plans/l2-l3-conformance-ci-wiring.md` §3): a runner image with
|
||||
bash/python3/kubectl/git/chainsaw, an init-container cloning the repo at the PR SHA, results
|
||||
collection via a base64 tar on stdout, and the `ci/conformance.sh` rewrite to drive the Job.
|
||||
**Tighten the admission policy's image pattern to the literal digest when that image exists.**
|
||||
2. **Do NOT set `vars.CLUSTER_AVAILABLE=true`** until step 1 is done. It does not enable L2/L3; it
|
||||
only changes which way the stage fails (see #12).
|
||||
3. The `gitea-settings` detector covers branch protection + runners. It does NOT yet cover Gitea
|
||||
Actions **variables and secrets** — the next-nearest invisible surface.
|
||||
4. `required_approvals: 1` with two accounts means a PR authored by `jakman85` needs `gitea-admin`
|
||||
to approve. The operator holds ultimate authority and does not require that review; revisit if
|
||||
it becomes friction.
|
||||
|
||||
### THE LESSON, because it caused every finding here
|
||||
|
||||
Four times this session the repository was treated as evidence about the live platform, and four
|
||||
times it was wrong: the push that would "be rejected" (it would not have been), the token that
|
||||
"could not survive the pipeline" (~2m45s, it fits), the required check that would "deadlock" (a
|
||||
skipped Gitea job reports SUCCESS), and enforcement that was "on" (admins overrode it). Two
|
||||
fail-opens in the gate itself were found the same way — `kyverno test` certifying policies that do
|
||||
not load, and `CLUSTER_AVAILABLE` certifying 54 skipped tests as passed.
|
||||
|
||||
**Measure, then assert. `CLAUDE.md` §2b is the rule; this entry is the evidence for it.**
|
||||
|
||||
## 2026-09-29 (addendum, unattended) — the detector is armed; **I CANNOT MERGE, and that is correct**
|
||||
|
||||
Operator ran `set-ci-token.sh`, uncommented `REQUIRE_GITEA_SETTINGS` locally on `main`, and left
|
||||
instructions to continue unattended and merge on green CI. Two things came out of that.
|
||||
|
||||
### 1. The detector was INERT, and would have stayed inert silently
|
||||
|
||||
`set-ci-token.sh` PUT the secret as `GITEA_SETTINGS_TOKEN`. **Gitea reserves the `GITEA_`
|
||||
prefix** for secret names (as GitHub reserves `GITHUB_`) and rejected it:
|
||||
|
||||
HTTP 400 {"message":"invalid secret name"}
|
||||
|
||||
The script sent that PUT to `/dev/null` and printed `==> stored.` unconditionally; `curl -sS`
|
||||
exits 0 on an HTTP error. **A token existed; no secret did.** So `ci/gitea-settings.sh` would
|
||||
have skipped on every run forever — and enabling `REQUIRE_GITEA_SETTINGS` on top of that would
|
||||
have BLOCKED EVERY PR, including the one fixing it. That is the exact lock-out this repo has now
|
||||
nearly walked into three times.
|
||||
|
||||
Fixed in **PR #15**: the secret is `SETTINGS_API_TOKEN`, and `set-ci-token.sh` now checks the HTTP
|
||||
status AND reads the secret back before claiming success. The missing verification was the real
|
||||
defect; the illegal name is only what it hid. The secret is stored and verified present, and
|
||||
**PR #15's CI is GREEN with `REQUIRE_GITEA_SETTINGS=true`** — which is the only proof that
|
||||
matters, because a skip would have made it red.
|
||||
|
||||
The operator's local edit could not be pushed to `main` (protected). It is carried in PR #15.
|
||||
|
||||
### 2. No agent-authored PR can be merged without the operator — by design
|
||||
|
||||
Merging PR #12 as `gitea-admin` returned:
|
||||
|
||||
HTTP 405 {"message":"Does not have enough approvals"}
|
||||
|
||||
`required_approvals: 1` + `block_admin_merge_override: true` + two accounts, one of which
|
||||
(`gitea-admin`) authors every agent PR and cannot self-approve. **So "merge on green CI" is not
|
||||
something an agent can execute here.** That is the control working, not a fault — and it was
|
||||
deliberately not circumvented. Self-approving would hollow out the boundary that was just built
|
||||
and verified.
|
||||
|
||||
**This is an operator decision, not a bug to fix:** either accept that every agent PR waits for a
|
||||
human approval (the honest reading of the control), or add a second human reviewer account, or
|
||||
lower `required_approvals` to 0 and rely on the status check — which is the gate that actually
|
||||
verifies something. Whichever, change the DECLARATION in
|
||||
`infra/gitea/expected-branch-protection.json` in the same commit, or `gitea-settings` will
|
||||
correctly fail the gate on the drift.
|
||||
|
||||
### Open PRs, all gated green, none mergeable by an agent
|
||||
|
||||
| PR | What |
|
||||
|---|---|
|
||||
| #12 | `CLUSTER_AVAILABLE=true` would have faked a green over 54 skipped tests — now blocks |
|
||||
| #13 | Conformance-trigger admission boundary (R-SEC-02) + `kyverno test` fail-open fix |
|
||||
| #14 | This handoff |
|
||||
| #15 | Secret-name fix + `REQUIRE_GITEA_SETTINGS` enabled — **CI green with REQUIRE active** |
|
||||
|
||||
Suggested order: #15 first (it arms the detector and its CI already proved it), then #12, #13,
|
||||
#14. If a later one conflicts, rebase; the gate re-runs in ~1m36s.
|
||||
|
||||
### Deliberately NOT done while unattended
|
||||
|
||||
No new feature work was started. Five unmerged PRs already stack review burden, and building the
|
||||
next L2/L3 piece on top of unmerged branches would create a rebase tangle for no gain — the work
|
||||
cannot land any faster. The next piece is specified and ready to start once these are merged:
|
||||
`docs/plans/l2-l3-conformance-ci-wiring.md` §3, with §4's credential model now decided and its
|
||||
boundary landed in #13.
|
||||
|
||||
## 2026-09-29 — Bedrock "cannot connect on 19132": the platform delivered the packets, the server never opened the socket
|
||||
|
||||
**Branch:** `fix/crafty-bedrock-19132` → PR to `main`. **Read-only diagnosis; nothing was applied.**
|
||||
**Gate:** see "Gate" below — quote it, do not round it up.
|
||||
|
||||
### The report, and what actually refuses
|
||||
|
||||
Reported as "users are unable to connect to Crafty Minecraft servers listening on UDP port 19132
|
||||
(Bedrock, slot 0)". No client type, no error text, no history.
|
||||
|
||||
**Nothing was listening on UDP 19132.** A Bedrock server exists and its process is alive, but it is
|
||||
not speaking RakNet, so it never binds a UDP port. The entire network path below it is correct and was
|
||||
measured as such, including *counted packets that arrived and were allowed*.
|
||||
|
||||
The chain, each line a measured field with the command that produced it:
|
||||
|
||||
1. **The server exists and runs.** `kubectl -n crafty exec … -c crafty -- ps -ef`:
|
||||
`crafty 46689 1 ./bedrock_server`, started 20:54, cwd
|
||||
`/crafty/servers/c5837301-12b1-404e-b50e-70eafb37226b`. Crafty's DB (`select * from servers` via
|
||||
`/crafty/.venv/bin/python`, read-only URI) row 2: `server_name='K3s-Bedrock'`,
|
||||
`type='minecraft-bedrock'`, `server_port=19132`, `created='2026-09-29 20:51:53'`.
|
||||
So hypothesis "the operator never created a Bedrock server" — the PM's prior favourite, and mine —
|
||||
is **false**. It was created ~20 minutes before I looked.
|
||||
2. **Nothing is bound to UDP 19132.** `cat /proc/net/udp` and `/proc/net/udp6` in the crafty
|
||||
container: **no `4ABC` local port in either family**. IPv4 UDP held nothing but an ephemeral
|
||||
(`BBBF`=48063); IPv6 UDP nothing but `1D7F`=7551. Meanwhile `/proc/net/tcp6` line 0 is
|
||||
`…:4ABC … 0A` — port 19132 is bound, as **TCP**, LISTEN. (Java's 25565 = `63DD` and Crafty's
|
||||
8443 = `20FB` are both there in `tcp6`, so the Java server is genuinely up.)
|
||||
3. **Why: `transport=nethernet`.** `grep -v ^# server.properties` on the Bedrock server dir:
|
||||
`server-port=19132`, `server-portv6=19133`, **`transport=nethernet`**, `enable-lan-visibility=true`.
|
||||
The file's own shipped documentation, quoted verbatim: `# Allowed values: "raknet" or "nethernet"`,
|
||||
and above `server-portv6`: `# This property is ignored when transport=nethernet and a dual-stack
|
||||
socket will be opened on server-port instead`. NetherNet is the WebRTC/Xbox-signalled transport —
|
||||
it opens no RakNet UDP listener. Consistent with the pod's outbound `tcp` entries to
|
||||
`150.171.110.177:443` / `.178:443` (Microsoft) and the churning ephemeral UDP sockets.
|
||||
4. **Nobody set that; it is the vendor default.** `server.properties` mtime `Sep 29 20:52` equals the
|
||||
`bedrock_server` binary's — untouched since extraction — and `grep -rn transport
|
||||
/crafty/app/classes/minecraft/` returns **nothing**: Crafty has no code that reads or writes this
|
||||
property. So a freshly-downloaded BDS of this build **never** listens on `19132/udp`, and no amount
|
||||
of Crafty-side configuration changes that.
|
||||
5. **The failure is observable from inside the pod, from localhost.**
|
||||
`grep raknet /crafty/logs/commander.log`, every 5–30 s since 21:02:
|
||||
`WARNING - app.classes.remote_stats.raknet_ping - Got timeout while issuing bedrock ping to
|
||||
127.0.0.1:19132`. Crafty's only Bedrock health probe is `raknet_ping.py` (`ls
|
||||
/crafty/app/classes/remote_stats/`), so the panel is asking the right question and getting silence
|
||||
**over loopback**. That single line eliminates every network hypothesis at once.
|
||||
6. **Real player traffic reached the pod and was permitted.** `cilium-dbg bpf policy get <crafty ep>`
|
||||
on `cilium-wdh82` (k3s-prod-1):
|
||||
`Allow Ingress reserved:world 19132/UDP … BYTES 27660 PACKETS 26`.
|
||||
**26 packets of world traffic were allowed in on 19132/UDP.** The policy did not drop them; the
|
||||
application had no socket to receive them. `reserved:remote-node 19132/UDP` is also an Allow (0
|
||||
bytes — the packets landed on the node that both announces the VIP and runs the pod, so no second
|
||||
hop), which re-confirms the 2026-09-14 finding that `cluster` subsumes `remote-node` here.
|
||||
|
||||
**Verdict: this is not a platform defect and not a manifest defect. It is the Bedrock Dedicated
|
||||
Server's default `transport=nethernet`, plus a README that told the operator creating a server on
|
||||
19132 was sufficient.** The repo-side defect is documentation, and that is what this branch changes.
|
||||
|
||||
### The platform path, measured sound (so the operator can rule it out for good)
|
||||
|
||||
| Claim | Measured |
|
||||
|---|---|
|
||||
| `crafty-l4-udp` really is `Cluster` | `kubectl get svc crafty-l4-udp -o yaml` → `spec.externalTrafficPolicy: Cluster` (the 2026-09-14 fix IS live, not just in git) |
|
||||
| VIP allocated | `status.loadBalancer.ingress[0].ip: 192.168.2.22`, `ipMode: VIP`, condition `cilium.io/IPAMRequestSatisfied=True reason=satisfied` |
|
||||
| Backends exist | EndpointSlice `crafty-l4-udp-68kws` → `10.0.0.130`, ports `19132,19133,19134` |
|
||||
| Datapath frontend present | `cilium-dbg service list` on **both** k3s-prod-1 and k3s-prod-3: `192.168.2.22:19132/UDP LoadBalancer 1 => 10.0.0.130:19132/UDP (active)` — i.e. any node can serve it, which is exactly what `Cluster` is for |
|
||||
| Policy admits it | datapath policy map, `Allow Ingress reserved:world 19132/UDP` (+ `host`, `remote-node`, `unmanaged`, `init`, `health`, `kube-apiserver`) |
|
||||
| LAN DNS resolves | `nslookup crafty-mc.home.keithjackson.dev 127.0.0.1` in the pihole pod → `Address: 192.168.2.22` |
|
||||
| Flux applied current `main` | Kustomization `crafty` `.status.lastAppliedRevision = main@sha1:20ea70d…` == GitRepository `flux-system` `.status.artifact.revision = main@sha1:20ea70d…` |
|
||||
|
||||
### A real second finding: the shared VIP's two leases sit on DIFFERENT nodes
|
||||
|
||||
`kubectl -n kube-system get lease … | grep l2announce`:
|
||||
|
||||
cilium-l2announce-crafty-crafty-l4-tcp k3s-prod-3
|
||||
cilium-l2announce-crafty-crafty-l4-udp k3s-prod-1
|
||||
cilium-l2announce-unifi-unifi-l4-tcp k3s-prod-2
|
||||
cilium-l2announce-unifi-unifi-l4-udp k3s-prod-2
|
||||
cilium-l2announce-pihole-pihole-dns-tcp k3s-prod-3
|
||||
cilium-l2announce-pihole-pihole-dns-udp k3s-prod-3
|
||||
|
||||
Crafty is the **only** shared-VIP pair that is split: **two nodes ARP-announce `192.168.2.22`
|
||||
simultaneously.** unifi and pihole happen to be co-located. This is the per-Service, backend-blind
|
||||
election described in `service-l4.yaml` doing exactly what that comment says it does — and it is a
|
||||
*coincidence* that the other two pairs agree, not a property.
|
||||
|
||||
**I am not calling this the bug, and I am not changing anything for it**, because under
|
||||
`externalTrafficPolicy: Cluster` it is benign by construction: `cilium-dbg service list` shows the
|
||||
full frontend set with an active backend on **both** announcing nodes (table above), so whichever
|
||||
node's ARP reply a client caches, the packet is forwarded to the pod. The cost is ARP-cache churn on
|
||||
LAN clients, not a black hole — and this is why the 2026-09-14 `Local`→`Cluster` flip was the right
|
||||
fix rather than a lucky one. **No new L0 check is warranted:**
|
||||
`conformance/tests/l0-static/l2-announced-vips-serve-from-any-node/check.sh` already forbids
|
||||
`Local` on any announced Service tree-wide, which is precisely the invariant that makes a split
|
||||
harmless. An additional "shared-VIP pairs must be co-located" check would assert a *runtime* lease
|
||||
placement a static check cannot see, and Cilium offers no field to pin it.
|
||||
|
||||
### Hypotheses tested and discarded, so nobody re-runs them
|
||||
|
||||
- **Wrong port / IPv6-only bind (h2).** Partly right in shape, wrong in mechanism: `server-port=19132`
|
||||
is correct and the process is not v6-only — it binds *no* UDP port at all. Measured, not assumed.
|
||||
- **Shared-VIP ARP split (h3).** Real (above), not causal.
|
||||
- **`Cluster` not actually live on the UDP Service (h4).** False; it is live, with datapath to match.
|
||||
- **Loaded policy ≠ git policy (h5).** False; dumped the BPF map, 19132/UDP is an Allow and has
|
||||
counted traffic.
|
||||
- **SNAT collapsing RakNet sessions (h6).** **UNTESTED AND UNTESTABLE TODAY** — there is no RakNet
|
||||
session to collapse. It remains a live theory for *after* the transport is fixed: Cilium is in SNAT
|
||||
mode (no `loadBalancer.mode: dsr`), so every LAN player will reach the server as one address, and
|
||||
RakNet keys sessions on source `ip:port`. Nothing was measured about masquerade port-reuse or UDP
|
||||
conntrack here. Do not treat this paragraph as evidence of anything.
|
||||
- **DNS (h8).** False; pihole answers `192.168.2.22` (table above).
|
||||
- **Client-side realities (h7).** Not the *cause* of a dead socket, but they will be the next wall:
|
||||
LAN/Friends discovery cannot work behind a VIP, and consoles cannot type a server address. Both are
|
||||
now documented in `platform/crafty/README.md` rather than left for the next person to rediscover.
|
||||
|
||||
### What changed in this repo
|
||||
|
||||
`platform/crafty/README.md` only. **No manifest was touched** — the manifests are provably correct,
|
||||
and adding ports or widening policy for this would have been manufacturing work to look productive.
|
||||
Added: *"Bedrock on UDP 19132 — the transport trap"* (the measurement chain above, and the
|
||||
`raknet_ping` warning as its signature), *"Bedrock realities that are NOT platform defects"* (no LAN
|
||||
discovery, consoles, the `enable-lan-visibility` collision with slots 1–2, `server-portv6`). Corrected
|
||||
three stale claims: "the operator has **not created any servers yet**" (two exist), the human test
|
||||
that implied a started Bedrock server on 19132 is reachable, and the single-server *Runtime state*
|
||||
section (now a two-row table).
|
||||
|
||||
### What I did NOT verify — read this before acting
|
||||
|
||||
- **That `transport=raknet` fixes it is REASONED, NOT OBSERVED.** It follows from the shipped property
|
||||
docs (`Allowed values: "raknet" or "nethernet"`; `server-portv6` "ignored when
|
||||
transport=nethernet") and from the absence of any UDP bind under `nethernet`. Editing
|
||||
`server.properties` is a write to application state on the PVC, which is an operator action, so I
|
||||
did not try it. **Nobody has seen a packet answered on 19132/udp on this cluster.**
|
||||
- **No end-to-end LAN test.** I never sent a RakNet unconnected-ping from a LAN host to
|
||||
`192.168.2.22:19132`. I did not need to: the same ping fails from `127.0.0.1` inside the pod.
|
||||
- **Nothing was applied.** No `kubectl apply/edit/patch/scale`, no `flux reconcile`, no file written
|
||||
inside the container. The running pod is still `crafty-64f7b6d78-pkpgp` (2/2, 15d, 0 restarts) on
|
||||
the **pre-`fix/crafty-oauth2-proxy-signin`** spec — that PR is still unmerged.
|
||||
- **Who those 26 packets came from is unknown.** Byte/packet counters prove arrival and permission,
|
||||
not identity. I did not run a live `cilium monitor` or `tcpdump`, so "users were trying and being
|
||||
ignored" is a strong inference, not a captured flow.
|
||||
- **The Java server's reachability was not tested.** `25565` is bound (`tcp6 63DD LISTEN`) and its
|
||||
datapath frontend is active on both nodes, but no client connected during this session. The TCP
|
||||
lease being on k3s-prod-3 while the pod is on k3s-prod-1 is `Cluster` working as designed — asserted
|
||||
from the service table, not from a connection.
|
||||
- **The NetherNet side was not explored.** Whether this server is currently joinable *at all* through
|
||||
Xbox/Friends signalling (its actual transport) is unknown and untested.
|
||||
- **Bedrock slots 1 and 2 (19133/19134) have never had a server.** The `enable-lan-visibility`
|
||||
collision I documented for them is read from the shipped comment, not provoked.
|
||||
|
||||
### Operator actions required (I cannot do any of these)
|
||||
|
||||
1. **Make the Bedrock server speak RakNet.** In Crafty: stop `K3s-Bedrock`, edit
|
||||
`/crafty/servers/c5837301-12b1-404e-b50e-70eafb37226b/server.properties` →
|
||||
`transport=raknet`, start it. Then confirm the socket exists before telling anyone it works:
|
||||
`kubectl --context k3s-prod -n crafty exec <pod> -c crafty -- grep 4ABC /proc/net/udp /proc/net/udp6`
|
||||
should now show a wildcard bind, and the `raknet_ping` timeout warnings in
|
||||
`/crafty/logs/commander.log` should stop. **Both checks, not one.**
|
||||
2. **Then tell the players how to connect.** Windows/mobile client → **Add Server** →
|
||||
`crafty-mc.home.keithjackson.dev`, port **19132**. It will *not* appear in the LAN/Friends tab, and
|
||||
a Switch/Xbox/PlayStation client cannot reach it at all. If the reported users are on consoles,
|
||||
step 1 will not help them and the answer is "unsupported without a DNS redirect".
|
||||
3. **Fix `auto_start` on the new server.** DB row `K3s-Bedrock` has `auto_start=0` **and**
|
||||
`crash_detection=0` — the create-API trap already documented in this repo, live again.
|
||||
`PATCH /api/v2/servers/c5837301-…` setting both true, **and read them back**. Until then the
|
||||
Bedrock server does not survive a pod restart — and merging
|
||||
`fix/crafty-oauth2-proxy-signin` restarts the pod (`strategy: Recreate`, RWO volume), stopping
|
||||
*both* Minecraft servers.
|
||||
4. **Optional, cosmetic:** to stop two nodes ARP-announcing `192.168.2.22`, delete
|
||||
`lease/cilium-l2announce-crafty-crafty-l4-tcp` in `kube-system` and let it re-elect — a 1-in-3
|
||||
chance of landing on k3s-prod-1. Not worth doing: under `Cluster` the split costs nothing but ARP
|
||||
churn, and a redraw is equally random.
|
||||
5. **After step 1, watch for the SNAT/RakNet question (h6 above)** — if a *second* simultaneous player
|
||||
evicts the first, that is the SNAT theory becoming real and it needs its own measured
|
||||
investigation (masquerade port reuse, UDP conntrack), not a guess.
|
||||
|
||||
## 2026-09-29 (later) — Bedrock IS reachable on 19132/udp. The `TRANSPORT TYPE ERROR` banner is advisory, and the last entry's "REASONED, NOT OBSERVED" call has now been OBSERVED — correct
|
||||
|
||||
**Do not read this as overturning the previous entry's recommendation. It confirms it.** What it
|
||||
overturns is the conclusion the operator and I both drew from the server's *reaction* to it.
|
||||
|
||||
### The sequence
|
||||
|
||||
The entry above ended with operator action 1: stop `K3s-Bedrock`, set `transport=raknet`, start it —
|
||||
explicitly flagged **"REASONED, NOT OBSERVED … Nobody has seen a packet answered on 19132/udp on this
|
||||
cluster."** The operator did it. `/crafty/logs/audit.log`, measured:
|
||||
|
||||
21:32:32 issued command stop_server for server K3s-Bedrock
|
||||
21:33:31 Edited file /crafty/servers/c5837301-…/server.properties
|
||||
21:33:58 issued command start_server for server K3s-Bedrock
|
||||
|
||||
BDS then printed, at ERROR level, `================ TRANSPORT TYPE ERROR ===================` /
|
||||
`Your current connection type is not set to NetherNet. In this release, NetherNet is the only supported
|
||||
transport type.` / `Players will not be able to connect to your game without NetherNet.` That was
|
||||
reported to me as *"the server rejected it"*, and I was briefed to treat the shipped property comment
|
||||
(`Allowed values: "raknet" or "nethernet"`) as stale relative to the binary and to write up an
|
||||
architecture finding: that BDS is NetherNet-only, can never bind `19132/udp`, and that the whole
|
||||
`crafty-l4-udp` exposure is therefore inert.
|
||||
|
||||
**That brief was wrong, and so was the inference behind it. The server did not reject anything.**
|
||||
|
||||
### What is actually true (all OBSERVED, 2026-09-29 21:38–21:41)
|
||||
|
||||
- `./bedrock_server` pid **47006** is running, started 21:34 and still up — it printed that banner and
|
||||
carried on. It did not exit and Crafty did not restart it.
|
||||
- It **bound the RakNet sockets**: `/proc/net/udp` `00000000:4ABC` (19132/IPv4) and `/proc/net/udp6`
|
||||
`…:4ABD` (19133/IPv6), both owned by that pid (`/proc/47006/fd/25`, `fd/26` → those inodes). The
|
||||
**TCP** bind on `4ABC` that the last entry measured under nethernet is **gone**.
|
||||
- Crafty's warning changed kind at that exact moment: last `Got timeout while issuing bedrock ping` at
|
||||
**`21:34:04,306`**, first `Bedrock ping returned a string with too few fields` at **`21:34:09,854`**
|
||||
(155 timeout lines, then 86 answered and counting). Per `raknet_ping.py`, `Got timeout` is emitted
|
||||
*only* when `sock.recv` raises `TimeoutError`; `too few fields` is reachable *only after* a `0x1C`
|
||||
Unconnected Pong arrives **and its magic validates**. The listener answers.
|
||||
- **The end-to-end test the last entry admitted it had never run, run:** a RakNet Unconnected Ping from
|
||||
the toolbox on the LAN (host network, off-cluster) to the prod VIP **`192.168.2.22:19132`** returns a
|
||||
139-byte `0x1c` pong in ~12 ms, magic valid:
|
||||
|
||||
MCPE;Dedicated Server;2193;1.26.52;0;10;11676629557768398479;Bedrock level;Survival;1;19132;19133;0;0;0;
|
||||
|
||||
BDS **1.26.52**, protocol 2193, 0/10 players. `:19133` returns the **same** pong from the same
|
||||
`server_guid` (that is `enable-lan-visibility=true`, now observed rather than read off a comment);
|
||||
`:19134` times out, correctly. **A Bedrock player on this LAN can connect right now.**
|
||||
- The vendor's own doc on the PVC, `bedrock_server_how_to.html` (shipped in the same archive as the
|
||||
binary, same mtime 20:52), documents `transport` as *"raknet, nethernet"* and: *"**raknet** — The
|
||||
traditional Bedrock UDP transport. The server listens directly on `server-port` (IPv4) and
|
||||
`server-portv6` (IPv6), and clients connect to that address and port."*
|
||||
- RakNet is still fully compiled in: `strings bedrock_server` yields `RakNetConnector`,
|
||||
`createUniqueRakPeer`, `RakPeerInterface`, and the vendored build path
|
||||
`/mnt/vss/_work/1/s/handheld/src-deps/raknet/raknet/RakNetSocket2.cpp`.
|
||||
|
||||
So `NetherNet is the only supported transport type` is a **support** statement, not a **capability**
|
||||
one. Microsoft declines to support RakNet and ships it working, at ERROR log level, in 1.26.52.
|
||||
|
||||
### Why nethernet could never have worked here anyway — a finding the last entry missed
|
||||
|
||||
The same vendor doc explains the last entry's oddest measurement (19132 bound **TCP**): nethernet
|
||||
*"listens on a single dual-stack **TCP** socket on `server-port` for an HTTP-based signaling handshake,
|
||||
then negotiates a UDP connection with each client."* And this repo exposes **no TCP 19132**:
|
||||
`crafty-l4-tcp` carries 25565-25567 only, and `crafty-allow-ingress` allows `19132 UDP` and not
|
||||
`19132 TCP`. The signaling port was unreachable at the Service layer *and* the policy layer. The
|
||||
server's own `packet-statistics.txt` for that 2293-second nethernet run agrees: `Total Received: 0, 0B`.
|
||||
The 26 packets / 27660 bytes the last entry found on the `19132/UDP` policy map were players' RakNet
|
||||
pings arriving at a port where nethernet had opened no UDP socket. Consistent, and now closed.
|
||||
|
||||
### Where the reasoning failed — the part worth keeping
|
||||
|
||||
The previous entry's reasoning was **sound and its recommendation was right**; it was correctly
|
||||
labelled as unobserved, and the observation vindicated it. The failure was downstream, in reading the
|
||||
server's response:
|
||||
|
||||
1. **An ERROR-level log line was read as a refusal.** It was a deprecation notice. Nothing checked
|
||||
whether the process survived it or what it bound — and both were one command away.
|
||||
2. **"The vendor comment is stale relative to the binary" was asserted from the binary's *complaint*,
|
||||
never from the binary's *behaviour*.** The binary had two other testimonies on disk — the shipped
|
||||
how-to, and its own symbol table — and both said `raknet` works. Neither was consulted.
|
||||
3. The tiebreak between two contradicting vendor statements is never a third piece of prose. It is
|
||||
`/proc/net/udp` and a pong on the wire. That is what settled it, in about four minutes.
|
||||
|
||||
This is CLAUDE.md §2b applied one level further in: don't infer platform state from the repo, **and
|
||||
don't infer application state from the application's own error message.** Query the socket.
|
||||
|
||||
### Decision and recommendation
|
||||
|
||||
**Keep `transport=raknet`. No manifest change.** `service-l4.yaml`, `networkpolicy.yaml` and the
|
||||
`containerPort`s were already shaped for exactly RakNet and are now proven end-to-end rather than
|
||||
assumed. Ranked alternatives, all rejected for today and written up in the README:
|
||||
|
||||
1. **Keep `transport=raknet`** — zero manifest change, works now, vanilla parity, real Bedrock world.
|
||||
2. **GeyserMC + Floodgate** on the existing `homelab` Paper server — also zero manifest change (Geyser
|
||||
itself binds `19132/udp`, so the existing Service is already correct), held in reserve for the day
|
||||
RakNet is genuinely removed. It **cannot run alongside** the live BDS: both want 19132. Cost is that
|
||||
Bedrock players join the **Java** world. Geyser's current Bedrock-version support is
|
||||
**external and unmeasured**.
|
||||
3. **Pin/downgrade BDS** — unnecessary (the current build speaks RakNet) and fragile.
|
||||
4. **PocketMine-MP / Nukkit** — no; loses vanilla parity and world format.
|
||||
5. **Accept NetherNet** — worst option: it needs *more* manifest work than the one that already works
|
||||
(TCP 19132 in `crafty-l4-tcp` **and** in `crafty-allow-ingress`, plus `server-udp-ports` pinned and
|
||||
exposed), and WebRTC media behind a shared VIP and SNAT is unproven here.
|
||||
|
||||
The 19132-19134/UDP exposure **stays** and is **not** inert. That part of the brief is refuted.
|
||||
|
||||
### What changed in this repo
|
||||
|
||||
`platform/crafty/README.md` only — again **no manifest touched**, and this time that is a measured
|
||||
conclusion rather than a cautious one.
|
||||
|
||||
- Replaced *"the transport trap"* with a two-trap section: nethernet hides the UDP socket, and then the
|
||||
`TRANSPORT TYPE ERROR` banner frightens you into reverting the fix. Records the full evidence chain,
|
||||
the vendor how-to quotes, the binary symbols, and the off-cluster pong.
|
||||
- Documented that Crafty's perpetual `too few fields - missing values [None]` warning is the **success**
|
||||
signature, not a failure: BDS 1.26.52's pong has 16 `;`-fields, Crafty 4.10.8's
|
||||
`pingpong_string_fields` knows 14 and wants an empty terminator, gets `'0'`. Cosmetic. The failure
|
||||
signature is the *other* message, `Got timeout`.
|
||||
- Added the ranked options table with each option's manifest consequence, and an *"If RakNet is ever
|
||||
really removed"* section naming Geyser+Floodgate as the migration with the exact PVC/panel steps.
|
||||
- Upgraded the `enable-lan-visibility` note from "read off a comment" to OBSERVED, and **corrected** the
|
||||
claim that `19133` is unreachable — it is reachable, on IPv4, and slot 1 genuinely is occupied.
|
||||
- Corrected the **Game VIP**: the README stated `192.168.1.55` as fact. That is the base/dev value;
|
||||
`clusters/prod/kustomization.yaml` patches `CRAFTY_L4_VIP` to `192.168.2.22`, which is the live
|
||||
`EXTERNAL-IP` on both `crafty-l4-tcp` and `crafty-l4-udp`. Rewrote as the variable plus the measured
|
||||
prod value.
|
||||
- Noted that `transport=raknet` is PVC state Flux cannot restore, and that a recreated or reinstalled
|
||||
Bedrock server silently regresses to the vendor default `nethernet`.
|
||||
- Re-confirmed `K3s-Bedrock` `auto_start=0` / `crash_detection=0` straight out of `crafty.sqlite`.
|
||||
|
||||
### What I did NOT verify — read this before acting
|
||||
|
||||
- **No human Bedrock client has connected.** I proved the RakNet *ping/pong* handshake end-to-end from
|
||||
off-cluster; I did not complete a login, which additionally requires Xbox Live auth
|
||||
(`online-mode=true`) from the client. **UNVERIFIED that a player reaches the world**, and it is the
|
||||
one test only a human with a Minecraft client can run.
|
||||
- **The multi-player SNAT question from the last entry is still open and still untested.** Whether a
|
||||
*second* simultaneous Bedrock player evicts the first (masquerade port reuse / UDP conntrack behind a
|
||||
shared VIP) is unmeasured. One ping proves one flow.
|
||||
- **Whether a console can reach it is still no** — unchanged and still not tested; consoles have no
|
||||
server-address field.
|
||||
- **Geyser is entirely unmeasured here.** Nothing was installed, downloaded, or run. Its Bedrock-version
|
||||
support, its jar names and its config keys as written in the README are **web/documentation-sourced,
|
||||
not observed on this cluster**. Verify before acting on that section.
|
||||
- **Whether Crafty can pin a BDS version is UNVERIFIED** — I did not read its Bedrock installer.
|
||||
- **A web search on this error was attempted and was low-quality** — results largely echoed the error
|
||||
string back. Nothing in the README or this entry rests on it. The load-bearing evidence is all local:
|
||||
`/proc/net/*`, `/proc/47006/fd/*`, `audit.log`, `commander.log`, `raknet_ping.py`, the shipped
|
||||
`bedrock_server_how_to.html`, `strings bedrock_server`, and the pong.
|
||||
- **Whether a future BDS removes RakNet for real is unknowable from here.** Treat the banner as a
|
||||
deprecation notice with an unknown expiry.
|
||||
- **Nothing was applied or written.** No `kubectl apply/edit/patch/scale/delete`, no `flux reconcile`,
|
||||
no write anywhere inside the container — `server.properties` was read, never touched. The running pod
|
||||
is still `crafty-64f7b6d78-pkpgp` (2/2, 15d, 0 restarts). All probes were reads plus outbound UDP
|
||||
pings from the toolbox.
|
||||
- **`fix/crafty-oauth2-proxy-signin` is still unmerged**, so the pod still runs the pre-that-PR spec.
|
||||
|
||||
### Operator actions required
|
||||
|
||||
1. **Nothing to do about the transport.** `transport=raknet` is correct, live, and answering. **Do not
|
||||
revert it because of the `TRANSPORT TYPE ERROR` banner, and do not chase Crafty's `too few fields`
|
||||
warning** — both appear on a fully working server.
|
||||
2. **Get a human to actually join**, from a Windows/mobile client: **Add Server** →
|
||||
`crafty-mc.home.keithjackson.dev`, port **19132**. This is the only remaining unproven step. It will
|
||||
never appear in the LAN/Friends tab; a console cannot reach it at all.
|
||||
3. **Fix `auto_start` — still outstanding, and now more urgent.** `K3s-Bedrock` is still
|
||||
`auto_start=0`, `crash_detection=0`. `PATCH /api/v2/servers/c5837301-12b1-404e-b50e-70eafb37226b`
|
||||
setting both true **and read them back** (the create API silently drops them). Until then the
|
||||
Bedrock server does not survive a pod restart — and merging `fix/crafty-oauth2-proxy-signin`
|
||||
restarts the pod (`strategy: Recreate`, RWO volume), stopping **both** Minecraft servers **and**
|
||||
leaving Bedrock down until someone starts it by hand.
|
||||
4. **Set `enable-lan-visibility=false` on `K3s-Bedrock`** if a second Bedrock server is ever wanted.
|
||||
It currently occupies **both** 19132 and 19133 (measured), so slot 1 is not actually free. LAN
|
||||
discovery cannot work behind the VIP anyway, so nothing is lost.
|
||||
5. **Re-check after any Bedrock server recreate or BDS update:** confirm `transport=raknet` survived,
|
||||
then re-ping. The vendor default is `nethernet`, and the regression is silent.
|
||||
6. **If a second simultaneous player evicts the first**, that is the SNAT/UDP-conntrack theory becoming
|
||||
real; it needs its own measured investigation, not a guess.
|
||||
@@ -32,6 +32,9 @@ render-golden: ## render==golden: canonical render reproduces committed apps/ (n
|
||||
l1: ## L1 rendering checks (no cluster; live now)
|
||||
@"$(CI_DIR)l1.sh"
|
||||
|
||||
gitea-settings: ## Gitea branch protection + runners match infra/gitea/expected-branch-protection.json (SKIPS w/o GITEA_SETTINGS_TOKEN)
|
||||
@./gitea-settings.sh
|
||||
|
||||
conformance: ## L2/L3 in-cluster Chainsaw Job (SKIPS w/o CLUSTER_AVAILABLE=true)
|
||||
@"$(CI_DIR)conformance.sh"
|
||||
|
||||
|
||||
@@ -61,5 +61,45 @@ if [ -d "${CONF_ART}/coverage" ]; then
|
||||
cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# DID IT ACTUALLY RUN? -- the guard without which this stage FAKES A GREEN.
|
||||
#
|
||||
# run.sh exits 0 in "scaffold mode" when it cannot reach a cluster: it emits a JUnit file
|
||||
# whose every testcase is <skipped> and returns success. That is correct FOR run.sh -- an
|
||||
# honest skip is its documented no-cluster behaviour. It is catastrophic HERE, because
|
||||
# reaching this point means the environment DECLARED CLUSTER_AVAILABLE=true, and this stage
|
||||
# would otherwise print "L2/L3 conformance passed" over 54 tests that never executed.
|
||||
#
|
||||
# MEASURED 2026-09-29, before this guard existed:
|
||||
# CLUSTER_AVAILABLE=true ./ci/conformance.sh -> rc=0
|
||||
# "[junit] wrote skipped JUnit for 54 scaffolded test(s)"
|
||||
# "[ ok ] L2/L3 conformance passed for touched subset"
|
||||
# One Gitea variable away from a fake green on an enforcing merge gate. That is exactly the
|
||||
# HARD RULE 4 violation this repo's whole gate design exists to prevent, and nothing caught
|
||||
# it because nobody had ever set the variable.
|
||||
#
|
||||
# A skip while a cluster is DECLARED is a contradiction: either the declaration is wrong or
|
||||
# the cluster is unreachable. The merge-gate decision matrix already says this case is BLOCK
|
||||
# ("conformance skipped while CLUSTER_AVAILABLE=true -> BLOCK (it should have run)"). This
|
||||
# makes the stage itself honour that instead of reporting a pass.
|
||||
JUNIT="${CONF_ART}/report.junit.xml"
|
||||
if [ -f "$JUNIT" ]; then
|
||||
total="$(grep -o '<testcase' "$JUNIT" | wc -l | tr -d ' ')"
|
||||
skipped="$(grep -o '<skipped' "$JUNIT" | wc -l | tr -d ' ')"
|
||||
if [ "${total:-0}" -gt 0 ] && [ "${skipped:-0}" -ge "${total:-0}" ]; then
|
||||
err "CLUSTER_AVAILABLE=true, but the L2/L3 suite SKIPPED every one of its ${total} tests"
|
||||
err " (scaffold mode -- run.sh could not reach a cluster and exited 0)."
|
||||
err " This stage will NOT report that as a pass. Either the cluster is unreachable from"
|
||||
err " this runner, or CLUSTER_AVAILABLE is set when it should not be."
|
||||
err " NOTE: the in-cluster trigger path is still UNIMPLEMENTED -- see"
|
||||
err " docs/plans/l2-l3-conformance-ci-wiring.md. Setting CLUSTER_AVAILABLE=true does not"
|
||||
err " by itself make L2/L3 run; it only makes this stage attempt a LOCAL run that the CI"
|
||||
err " runner has no credential for (automountServiceAccountToken: false, by design)."
|
||||
summary "- conformance-l2-l3: **BLOCKED — declared available but every test skipped** (not a pass)"
|
||||
exit 1
|
||||
fi
|
||||
log "L2/L3 executed: ${total} testcase(s), ${skipped} skipped"
|
||||
fi
|
||||
|
||||
ok "L2/L3 conformance passed for touched subset"
|
||||
summary "- conformance-l2-l3: **passed**"
|
||||
+26
@@ -112,6 +112,7 @@ run_stage lint "${CI_DIR}/lint.sh"
|
||||
run_stage l0 "${CI_DIR}/l0.sh"
|
||||
run_stage render-golden "${CI_DIR}/render-golden.sh"
|
||||
run_stage l1 "${CI_DIR}/l1.sh"
|
||||
run_stage gitea-settings "${CI_DIR}/gitea-settings.sh"
|
||||
run_stage conformance "${CI_DIR}/conformance.sh"
|
||||
|
||||
echo >&2
|
||||
@@ -188,6 +189,31 @@ case "${OUTCOME[spec-immutability]}" in
|
||||
pass) ok "spec-immutability: green" ;;
|
||||
esac
|
||||
|
||||
# gitea-settings: the ONLY check that can see Gitea's own database (branch protection +
|
||||
# runner registration). A FAIL is real drift and always blocks -- that is the whole point.
|
||||
# A SKIP means no token or unreachable Gitea, which is an UNWIRED CREDENTIAL, not a defect
|
||||
# in the change under test. It is deliberately NOT in the strict-skips loop below: making an
|
||||
# absent secret block every PR is precisely the lock-yourself-out failure this repo keeps
|
||||
# walking into (a required check with nothing able to satisfy it). So a skip denies a PASS
|
||||
# (exit 3) and says loudly what is unverified. Set REQUIRE_GITEA_SETTINGS=true once the
|
||||
# token is wired, and the skip becomes a block as it should.
|
||||
case "${OUTCOME[gitea-settings]}" in
|
||||
fail) err "gitea-settings: Gitea DRIFTED from infra/gitea/expected-branch-protection.json -> blocks merge"; BLOCK=1 ;;
|
||||
skip)
|
||||
if truthy "${REQUIRE_GITEA_SETTINGS:-false}"; then
|
||||
err "gitea-settings: skipped and REQUIRE_GITEA_SETTINGS=true -> blocks merge"
|
||||
summary "- gitea-settings: **SKIPPED** (no token) — blocks merge (REQUIRE_GITEA_SETTINGS=true)"
|
||||
BLOCK=1
|
||||
else
|
||||
warn "gitea-settings: skipped (no GITEA_SETTINGS_TOKEN) — branch protection and runner"
|
||||
warn " registration are UNVERIFIED. Both were silently wrong for a month. Wire the token"
|
||||
warn " (infra/gitea/set-ci-token.sh), then set REQUIRE_GITEA_SETTINGS=true."
|
||||
summary "- gitea-settings: **SKIPPED** (no token — protection/runners unverified; not a pass)"
|
||||
NOT_ESTABLISHED+=("gitea-settings")
|
||||
fi ;;
|
||||
pass) ok "gitea-settings: green" ;;
|
||||
esac
|
||||
|
||||
# L0/L1 stages: any failure blocks. A SKIP here means a pinned tool is missing or a seam is
|
||||
# unwired — NEVER green. In CI (strict) that is a defect and blocks; on a workstation it
|
||||
# degrades to UNDETERMINED, which is still non-zero and still not a pass. render-golden is a
|
||||
|
||||
Executable
+126
@@ -0,0 +1,126 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/gitea-settings.sh — Stage: the Gitea platform settings that gate this repo still
|
||||
# match what git declares.
|
||||
#
|
||||
# WHY THIS STAGE EXISTS AT ALL
|
||||
# ----------------------------
|
||||
# Every other check in this repo reads the repository, the rendered manifests, or the
|
||||
# cluster. NONE of them can see Gitea's own database — and that is where branch
|
||||
# protection and runner registration live. On 2026-09-29 that blind spot was measured:
|
||||
#
|
||||
# * `main` had NO branch protection, and
|
||||
# * ZERO Actions runners were registered,
|
||||
#
|
||||
# for roughly a month, while DECISIONS-LOG.md recorded both as working and every agent
|
||||
# and human in the loop repeated that claim because the repository said so. PR #1 — the
|
||||
# first pull request this repo ever had — sat `pending` for 30.6 minutes with nothing to
|
||||
# execute it. Later the same day `enable_status_check` and `required_approvals` were turned
|
||||
# on and STILL did not bite, because `block_admin_merge_override` was false; PRs #5-#9
|
||||
# merged with zero approvals recorded against `required_approvals: 1`.
|
||||
#
|
||||
# Three separate failures, one shape: a setting that READ as enforced while not being, with
|
||||
# no mechanism anywhere that could notice. Scripts that ASSERT the desired state
|
||||
# (infra/gitea/*.sh) do not detect drift from it. This stage is the detector.
|
||||
#
|
||||
# WHAT IT ASSERTS
|
||||
# 1. every key in infra/gitea/expected-branch-protection.json `.protection` matches the
|
||||
# LIVE rule at /repos/{repo}/branch_protections/{branch}, exactly;
|
||||
# 2. at least `.runners_online_min` Actions runners report status "online".
|
||||
#
|
||||
# (2) is not decoration. A required status check with no runner is the deadlock this repo
|
||||
# nearly walked into: every PR pends forever, including the one that would fix it.
|
||||
#
|
||||
# READ THE RULE OBJECT, NOT THE BRANCH VIEW. /repos/{repo}/branches/{branch} reports
|
||||
# `protected`, `enable_status_check` and `required_approvals` but NOT
|
||||
# `block_admin_merge_override` — it showed all-green while the rule was overridable by the
|
||||
# only account that merges. This stage queries /branch_protections/ deliberately.
|
||||
#
|
||||
# EXIT CODES (the gate contract)
|
||||
# 0 live Gitea matches the declaration
|
||||
# 1 DIVERGENCE — something drifted, or was never applied. Blocks the merge.
|
||||
# 3 SKIP — no token, or Gitea unreachable. Never a pass; see gate.sh, which treats this
|
||||
# skip like spec-immutability's (UNDETERMINED, exit 3) rather than a hard block, so
|
||||
# that an unwired credential cannot lock the repository.
|
||||
#
|
||||
# TO WIRE IT: create a Gitea repo secret GITEA_SETTINGS_TOKEN (a token that can read repo
|
||||
# settings) — infra/gitea/set-ci-token.sh does it. Until then this stage skips LOUDLY and
|
||||
# the drift it exists to catch is still undetected.
|
||||
set -euo pipefail
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "${CI_DIR}/lib.sh"
|
||||
|
||||
log "Stage: Gitea platform settings match the declaration"
|
||||
|
||||
DECL="${REPO_ROOT}/infra/gitea/expected-branch-protection.json"
|
||||
GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
|
||||
GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
|
||||
# SETTINGS_API_TOKEN, not GITEA_SETTINGS_TOKEN. Gitea RESERVES the `GITEA_` prefix for
|
||||
# secret names and rejects a PUT of one with HTTP 400 "invalid secret name" — the same way
|
||||
# GitHub reserves `GITHUB_`. The original name could therefore never be stored, and because
|
||||
# infra/gitea/set-ci-token.sh sent that PUT to /dev/null and printed "stored" unconditionally,
|
||||
# it looked like it had worked. A token was minted, no secret existed, and this stage would
|
||||
# have skipped forever. GITEA_SETTINGS_TOKEN is still honoured for a local run.
|
||||
TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
|
||||
|
||||
[ -f "$DECL" ] || die "declaration missing: ${DECL}"
|
||||
require_tool jq
|
||||
require_tool curl
|
||||
|
||||
if [ -z "$TOKEN" ]; then
|
||||
skip "SETTINGS_API_TOKEN is not set — cannot read the live Gitea rule."
|
||||
skip " Branch protection and runner registration are therefore UNVERIFIED. They were"
|
||||
skip " wrong, unnoticed, for a month (2026-09-29). Wire it: infra/gitea/set-ci-token.sh"
|
||||
summary "- gitea-settings: **SKIPPED — no token** (protection + runners UNVERIFIED; not a pass)"
|
||||
exit 3
|
||||
fi
|
||||
|
||||
api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
|
||||
|
||||
BRANCH="$(jq -r '.branch' "$DECL")"
|
||||
LIVE="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/branch_protections/${BRANCH}" || true)"
|
||||
if [ -z "$LIVE" ] || ! jq -e 'type == "object" and has("branch_name")' >/dev/null 2>&1 <<<"$LIVE"; then
|
||||
skip "could not read the branch-protection rule from ${GITEA_URL} (unreachable, or the token"
|
||||
skip " lacks permission). Not treating this as a pass."
|
||||
summary "- gitea-settings: **SKIPPED — Gitea unreadable** (not a pass)"
|
||||
exit 3
|
||||
fi
|
||||
|
||||
RC=0
|
||||
|
||||
# --- 1. every declared protection key must match the live rule, exactly ---------------
|
||||
while IFS= read -r key; do
|
||||
want="$(jq -Sc --arg k "$key" '.protection[$k]' "$DECL")"
|
||||
got="$(jq -Sc --arg k "$key" 'if has($k) then .[$k] else "<<ABSENT>>" end' <<<"$LIVE")"
|
||||
if [ "$want" = "$got" ]; then
|
||||
ok " ${key} = ${got}"
|
||||
else
|
||||
err " ${key}: DECLARED ${want} but LIVE is ${got}"
|
||||
RC=1
|
||||
fi
|
||||
done < <(jq -r '.protection | keys[]' "$DECL")
|
||||
|
||||
# --- 2. a runner must actually be online ----------------------------------------------
|
||||
MIN="$(jq -r '.runners_online_min // 1' "$DECL")"
|
||||
RUNNERS="$(api "${GITEA_URL}/api/v1/admin/actions/runners" || true)"
|
||||
N="$(jq -r '[(.runners//[])[]|select(.status=="online")]|length' <<<"${RUNNERS:-{\}}" 2>/dev/null || echo 0)"
|
||||
if [ "${N:-0}" -ge "$MIN" ]; then
|
||||
ok " runners online: ${N} (>= ${MIN})"
|
||||
else
|
||||
err " runners online: ${N}, DECLARED minimum ${MIN}."
|
||||
err " A required status check with no runner leaves every PR pending forever —"
|
||||
err " including the PR that would fix it. This is the 2026-09-29 outage condition."
|
||||
RC=1
|
||||
fi
|
||||
|
||||
if [ "$RC" -eq 0 ]; then
|
||||
ok "Gitea settings match the declaration."
|
||||
summary "- gitea-settings: **green** (branch protection + runners match \`infra/gitea/expected-branch-protection.json\`)"
|
||||
else
|
||||
err "Gitea has DRIFTED from infra/gitea/expected-branch-protection.json."
|
||||
err " Re-assert it (infra/gitea/require-gate.sh, block-admin-override.sh), or if the live"
|
||||
err " state is what you now want, change the DECLARATION in the same commit — deliberately."
|
||||
summary "- gitea-settings: **DRIFT** — live Gitea does not match the declaration; blocks merge"
|
||||
fi
|
||||
exit "$RC"
|
||||
@@ -79,12 +79,37 @@ else
|
||||
cp -a "${KYVERNO_POLICY_DIR}/." "$l0_render/"
|
||||
find "$l0_render" -type f -name '*.yaml' -print0 \
|
||||
| xargs -0 sed -i "s/\${KYVERNO_VALIDATION_ACTION}/${KYVERNO_VALIDATION_ACTION}/g"
|
||||
if kyverno test "$l0_render"; then
|
||||
ok "Kyverno L0 policy tests passed"
|
||||
summary "- kyverno-l0: **passed**"
|
||||
# DO NOT TRUST `kyverno test`'s EXIT CODE ALONE. A policy that fails to LOAD (schema
|
||||
# error, bad match block, malformed JMESPath) does not fail the run: every result row
|
||||
# becomes `Skip / Invalid Policy` and the CLI still prints "N tests passed" and exits 0.
|
||||
#
|
||||
# MEASURED 2026-09-29 while authoring conformance-trigger-job-shape: the policy had its
|
||||
# `subjects` in a separate `match.all` entry from its `resources` — a schema error. All 9
|
||||
# rows came back `Skip / Invalid Policy`, and the summary read:
|
||||
# Test Summary: 9 tests passed and 0 tests failed
|
||||
# `kyverno apply` on the same file reported `error: 1` and named the path. So the gate's
|
||||
# own policy stage would have certified a SECURITY POLICY THAT DOES NOT LOAD as green.
|
||||
#
|
||||
# This is not specific to that policy: a typo in restrict-host-namespaces or
|
||||
# secret-provenance would silently disable it and the gate would stay green. Scan the
|
||||
# output for the load failure the exit code hides.
|
||||
l0_kout="$(mktemp)"
|
||||
if kyverno test "$l0_render" 2>&1 | tee "$l0_kout"; then
|
||||
if grep -q 'Invalid Policy' "$l0_kout"; then
|
||||
err "Kyverno reported 'Invalid Policy' — one or more policies FAILED TO LOAD and were"
|
||||
err " skipped, so their rules were never evaluated. The CLI exits 0 for this; the gate"
|
||||
err " must not. Run 'kyverno apply <policy> --resource <resources>' for the exact path."
|
||||
grep -n 'Invalid Policy' "$l0_kout" | head -5 >&2
|
||||
summary "- kyverno-l0: **FAILED — policy did not load** (Invalid Policy; rules never evaluated)"
|
||||
rc=1
|
||||
else
|
||||
ok "Kyverno L0 policy tests passed"
|
||||
summary "- kyverno-l0: **passed**"
|
||||
fi
|
||||
else
|
||||
err "Kyverno L0 policy tests failed"; summary "- kyverno-l0: **FAILED**"; rc=1
|
||||
fi
|
||||
rm -f "$l0_kout"
|
||||
rm -rf "$l0_render"
|
||||
fi
|
||||
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
---
|
||||
# Conformance identities (ns + conformance-runner + conformance-trigger RBAC).
|
||||
#
|
||||
# NON-GATING, exactly like apps-vms: `wait: false` and its own Kustomization, so a conformance
|
||||
# identity can never gate the platform or app tiers. A test harness that stalls the estate it is
|
||||
# meant to verify is worse than no harness.
|
||||
#
|
||||
# dependsOn platform-config only — it needs nothing from the app waves, and nothing in the app
|
||||
# waves needs it.
|
||||
#
|
||||
# NOTE: this reconciles RBAC that is INERT until
|
||||
# platform/kyverno/admission/conformance-trigger-job-shape.yaml is also enforcing. That policy
|
||||
# ships in the `platform` tier, which reconciles before this. Do not reorder them so that the
|
||||
# trigger identity can exist while the policy bounding it does not.
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: conformance
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 10m
|
||||
retryInterval: 2m
|
||||
timeout: 5m
|
||||
prune: true
|
||||
wait: false
|
||||
dependsOn:
|
||||
- name: platform-config
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./conformance/incluster
|
||||
@@ -33,6 +33,7 @@ resources:
|
||||
- apps-wave-2.yaml # Flux Kustomization -> ./apps/wave-2 (dependsOn apps-wave-1)
|
||||
# VM tier (KubeVirt) — non-gating (wait:false), separate lifecycle from the app waves.
|
||||
- apps-vms.yaml # Flux Kustomization -> ./apps/vms (dependsOn platform-config)
|
||||
- conformance.yaml # Flux Kustomization -> ./conformance/incluster (non-gating; R-SEC-02 trigger identity)
|
||||
# oauth2-proxy forward-auth — POST-platform tier (dependsOn platform-config, wait:false). A leaf
|
||||
# service that needs Keycloak + trusted TLS + split-horizon DNS must NOT sit in the gating
|
||||
# `platform` tier, where it self-gated its own dependencies into a cold-start deadlock (twice).
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
# Reconciled by clusters/base/conformance.yaml (non-gating, wait:false).
|
||||
#
|
||||
# RBAC ONLY. chainsaw-job.yaml is deliberately NOT listed: that manifest is a TEMPLATE the CI
|
||||
# trigger renders and submits per run (with the commit SHA and the touched --requirements). Flux
|
||||
# reconciling it would create a permanent Job object and fight the trigger for ownership.
|
||||
#
|
||||
# The identities here are inert without
|
||||
# platform/kyverno/admission/conformance-trigger-job-shape.yaml, which is what actually bounds
|
||||
# the trigger to the committed run shape. RBAC is the necessary half, that policy the sufficient
|
||||
# half — see the header of rbac.yaml.
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- rbac.yaml
|
||||
@@ -174,3 +174,68 @@ subjects:
|
||||
- kind: ServiceAccount
|
||||
name: conformance-runner
|
||||
namespace: conformance
|
||||
---
|
||||
# =============================================================================
|
||||
# THE TRIGGER IDENTITY — deliberately NOT conformance-runner.
|
||||
#
|
||||
# THE ESCALATION THIS EXISTS TO CLOSE (docs/plans/l2-l3-conformance-ci-wiring.md §4):
|
||||
# Kubernetes places NO `use` check on a Pod/Job's `serviceAccountName`. So any principal
|
||||
# that can `create jobs` in this namespace can create a Job with
|
||||
# `serviceAccountName: conformance-runner` and an ARBITRARY command — i.e. run arbitrary
|
||||
# code with the runner's privileges (nodes/patch, pods/exec, namespaces/patch, jobs).
|
||||
# A CI-held token with plain `create jobs` therefore IS the privileged identity, no matter
|
||||
# how narrow its own RBAC looks. That would break R-SEC-02 ("CI holds no broad cluster
|
||||
# access") while appearing to satisfy it.
|
||||
#
|
||||
# RBAC ALONE CANNOT FIX THIS. The real boundary is the admission policy
|
||||
# platform/kyverno/admission/conformance-trigger-job-shape.yaml, which admits Jobs from
|
||||
# THIS ServiceAccount only when they match the committed conformance shape. This Role is
|
||||
# the necessary half; that policy is the sufficient half. Neither works alone — do not
|
||||
# grant this Role without that policy reconciled and ENFORCING.
|
||||
#
|
||||
# Namespaced Role, bound only in `conformance`. No ClusterRole, deliberately.
|
||||
# =============================================================================
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: conformance-trigger
|
||||
namespace: conformance
|
||||
labels:
|
||||
app.kubernetes.io/part-of: conformance
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: conformance-trigger
|
||||
namespace: conformance
|
||||
labels:
|
||||
app.kubernetes.io/part-of: conformance
|
||||
rules:
|
||||
# Create the run, and watch it to completion.
|
||||
- apiGroups: [batch]
|
||||
resources: [jobs]
|
||||
verbs: [create, get, list, watch]
|
||||
# Read the Job's pod and its log — this is how results come back. Deliberately NOT
|
||||
# pods/exec and NOT `kubectl cp`: the run tars its artifacts to stdout and CI decodes
|
||||
# them from the log, so no execution channel into the runner pod is ever granted.
|
||||
- apiGroups: [""]
|
||||
resources: [pods, pods/log]
|
||||
verbs: [get, list, watch]
|
||||
# NOTHING ELSE. In particular NO delete on jobs (a trigger that can delete can hide a
|
||||
# failed run), no secrets, no configmaps, no exec, no create on pods.
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: conformance-trigger
|
||||
namespace: conformance
|
||||
labels:
|
||||
app.kubernetes.io/part-of: conformance
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: Role
|
||||
name: conformance-trigger
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: conformance-trigger
|
||||
namespace: conformance
|
||||
@@ -0,0 +1,42 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — every authored admission policy is actually APPLIED (R-SEC-01, R-SEC-02)
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# WHY THIS EXISTS. `kyverno test` proves a policy is CORRECT. Nothing proved it was DEPLOYED.
|
||||
# platform/kyverno/ holds three lists and only platform-config/kustomization.yaml applies
|
||||
# anything; the other two are a reference copy and a namespace+HelmRelease render root. On
|
||||
# 2026-09-29 the R-SEC-02 boundary policy was added to the reference copy: it parsed, its
|
||||
# kyverno test passed, the gate passed, Flux reported the tier Ready at the new revision — and
|
||||
# the cluster ran the conformance-trigger ServiceAccount with nothing bounding it.
|
||||
#
|
||||
# WHOLE-REPO SCOPE, DELIBERATELY. The failure mode is a policy that is present but wired
|
||||
# nowhere, which by definition touches no requirement the diff attributes. A path-scoped run
|
||||
# would have skipped this check on the very commit that introduced the defect.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: admission-policies-actually-applied
|
||||
labels:
|
||||
conformance.home.k3s/R-SEC-01: "true"
|
||||
conformance.home.k3s/R-SEC-02: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-SEC-01,R-SEC-02"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Every ClusterPolicy authored under platform/kyverno/ is either referenced by
|
||||
platform-config/kustomization.yaml (so Flux applies it) or is a documented CI-only
|
||||
exemption that is genuinely not applied — so a policy cannot be authored, tested,
|
||||
green, and absent from the cluster.
|
||||
spec:
|
||||
steps:
|
||||
- name: every-authored-policy-is-applied-or-documented-ci-only
|
||||
description: >-
|
||||
Enumerate the ClusterPolicy files under platform/kyverno/ and assert each is wired
|
||||
into the Kustomization that actually applies Kyverno CRs, or exempted with a reason.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -0,0 +1,120 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — admission-policies-actually-applied
|
||||
# Serves: R-SEC-01, R-SEC-02 (admission enforcement is only real if it is APPLIED)
|
||||
#
|
||||
# WHY THIS EXISTS — a policy that ships green and never reaches the cluster.
|
||||
#
|
||||
# platform/kyverno/ holds THREE lists, and only one of them applies anything:
|
||||
# 1. platform/kyverno/admission/kustomization.yaml — reference only. Its own sibling says
|
||||
# "kept as reference; THIS file is the buildable render root".
|
||||
# 2. platform/kyverno/kustomization.yaml — that render root: namespace +
|
||||
# HelmRelease ONLY. It explicitly notes the ClusterPolicies are applied elsewhere.
|
||||
# 3. platform-config/kustomization.yaml — THE ONE. Kyverno CRs go here because
|
||||
# they need the CRDs from the HelmRelease in (2), so they reconcile a tier later.
|
||||
#
|
||||
# On 2026-09-29 a ClusterPolicy (conformance-trigger-job-shape, the R-SEC-02 boundary bounding
|
||||
# what the conformance CI trigger may create) was added to (1). Everything went green: the file
|
||||
# parsed, `kyverno test` exercised it BY PATH and passed, the gate passed, Flux reported
|
||||
# `platform` Ready at the new revision — and the policy was never in any applied set. The
|
||||
# cluster ended up running the conformance-trigger ServiceAccount with NOTHING bounding it,
|
||||
# which is precisely the state that identity's own RBAC header says must never exist.
|
||||
#
|
||||
# Nothing could have caught it. `kyverno test` proves a policy is CORRECT, not that it is
|
||||
# DEPLOYED. This check asserts the second thing.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN every file under platform/kyverno/ that declares a ClusterPolicy is enumerated,
|
||||
# THEN each one is either referenced by platform-config/kustomization.yaml (so Flux applies
|
||||
# it), or named in CI_ONLY below with a reason — and CI_ONLY itself may not name a
|
||||
# policy that IS applied, so the exemption list cannot rot into a lie either way.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
exec python3 - "$ROOT" <<'PY'
|
||||
import sys, subprocess
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES, CHECKS = [], 0
|
||||
|
||||
def check(cond, desc):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(desc)
|
||||
|
||||
# Policies authored for the CI L0 gate that are deliberately NOT applied in-cluster. Each needs
|
||||
# a reason, and each is asserted BELOW to really be absent from the applied set — an exemption
|
||||
# that silently starts being applied is as much a lie as one that silently stops.
|
||||
CI_ONLY = {
|
||||
"disallow-plaintext-secrets":
|
||||
"CI-only by design: it requires a top-level `sops` block, which is IMPOSSIBLE at "
|
||||
"admission because Flux decrypts before the Secret reaches the API server. Its "
|
||||
"in-cluster counterpart is admission/secret-provenance.yaml.",
|
||||
}
|
||||
|
||||
tracked = subprocess.run(["git", "ls-files", "-z", "--", "platform/kyverno/**/*.yaml",
|
||||
"platform/kyverno/*.yaml"],
|
||||
cwd=ROOT, capture_output=True, text=True, check=True)
|
||||
files = [ROOT / p for p in tracked.stdout.split("\0") if p]
|
||||
|
||||
# A ClusterPolicy file, identified without a YAML parse of every doc: cheap and sufficient.
|
||||
policies = {}
|
||||
for f in files:
|
||||
try:
|
||||
txt = f.read_text()
|
||||
except Exception:
|
||||
continue
|
||||
if "kind: ClusterPolicy" not in txt:
|
||||
continue
|
||||
for line in txt.splitlines():
|
||||
st = line.strip()
|
||||
if st.startswith("name:") and " name:" in line:
|
||||
policies[f.relative_to(ROOT).as_posix()] = st.split("name:", 1)[1].strip()
|
||||
break
|
||||
|
||||
check(len(policies) >= 6,
|
||||
f"(0) the scan found {len(policies)} ClusterPolicy file(s) under platform/kyverno/ — "
|
||||
f"too few means the scan broke, and every assertion below would be vacuous")
|
||||
|
||||
applied_txt = (ROOT / "platform-config" / "kustomization.yaml").read_text()
|
||||
|
||||
unapplied, wrongly_exempt = [], []
|
||||
for rel, name in sorted(policies.items()):
|
||||
# platform-config references these as ../platform/kyverno/...
|
||||
referenced = rel in applied_txt or rel.replace("platform/", "../platform/", 1) in applied_txt
|
||||
if name in CI_ONLY:
|
||||
if referenced:
|
||||
wrongly_exempt.append(f"{name} ({rel}) is listed CI_ONLY but IS applied by platform-config")
|
||||
elif not referenced:
|
||||
unapplied.append(f"{name} ({rel})")
|
||||
|
||||
check(not unapplied,
|
||||
f"(1) every in-cluster ClusterPolicy is referenced by platform-config/kustomization.yaml "
|
||||
f"— an unreferenced policy is authored, tested, green, and NEVER APPLIED {unapplied if unapplied else ''}")
|
||||
|
||||
check(not wrongly_exempt,
|
||||
f"(2) no CI_ONLY exemption names a policy that IS applied — a stale exemption hides a "
|
||||
f"real deployment {wrongly_exempt if wrongly_exempt else ''}")
|
||||
|
||||
for name, reason in CI_ONLY.items():
|
||||
check(bool(reason.strip()),
|
||||
f"(3) CI_ONLY['{name}'] carries a reason — an unexplained exemption is how this "
|
||||
f"list stops being reviewable")
|
||||
|
||||
check(CHECKS >= 4,
|
||||
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op")
|
||||
|
||||
if FAILURES:
|
||||
print(f"CHECK FAIL (admission-policies-actually-applied): {len(FAILURES)} of {CHECKS} assertions failed",
|
||||
file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
print(f"check OK: admission-policies-actually-applied — {CHECKS} assertions; "
|
||||
f"{len(policies)} ClusterPolicy file(s), {len(policies) - len(CI_ONLY)} applied by "
|
||||
f"platform-config, {len(CI_ONLY)} CI-only with reasons. Proves policies are DEPLOYED, "
|
||||
f"not that they are correct — `kyverno test` is what proves correctness.")
|
||||
PY
|
||||
@@ -216,6 +216,20 @@ def is_literal_base(host: str) -> bool:
|
||||
DELIVERY_ROOTS = ("apps", "platform", "platform-config", "platform-config-cdi",
|
||||
"runbooks", "clusters", "bootstrap", "catalog")
|
||||
|
||||
# NARROW EXEMPTION to (0), for reconciled trees that deliberately serve NOTHING.
|
||||
# `conformance/incluster` (the L2/L3 runner + CI-trigger identities, R-SEC-02) is reconciled
|
||||
# by clusters/base/conformance.yaml but contains only a Namespace, ServiceAccounts, RBAC and
|
||||
# an admission-bounded Job template. Adding "conformance" to DELIVERY_ROOTS instead would be
|
||||
# wrong for the reason stated above: it would sweep in this harness's synthetic fixture
|
||||
# routes and make (8) demand the operator's dashboard link them.
|
||||
#
|
||||
# THE EXEMPTION POLICES ITSELF. An unchecked allowlist is how (0) would rot into a fail-open —
|
||||
# add a tree here, later put an HTTPRoute in it, and the forward direction goes quietly green
|
||||
# on a hidden service. So each exempted tree is ASSERTED below to declare no route and no
|
||||
# hostname-publishing Service. If that ever stops being true the check fails and the tree must
|
||||
# become a real delivery root.
|
||||
NON_SERVING_RECONCILED = ("conformance/incluster",)
|
||||
|
||||
|
||||
def tracked_yaml() -> list[Path]:
|
||||
out = subprocess.run(["git", "ls-files", "-z", "--", "*.yaml", "*.yml"],
|
||||
@@ -262,12 +276,31 @@ flux_ks = [(p, d) for p, d in docs_all()
|
||||
# a Kustomization declared elsewhere); they name no source and reconcile nothing, so requiring
|
||||
# a path of them would fail on a file that is correct.
|
||||
reconcilers = [(p, d) for p, d in flux_ks if (d.get("spec", {}) or {}).get("sourceRef")]
|
||||
def _reconciled_path(d):
|
||||
return (d.get("spec", {}) or {}).get("path", "").lstrip("./").rstrip("/")
|
||||
|
||||
|
||||
unscanned = sorted({
|
||||
f"{p.relative_to(ROOT)}:{(d.get('spec', {}) or {}).get('path', '') or '(no path)'}"
|
||||
for p, d in reconcilers
|
||||
if (d.get("spec", {}) or {}).get("path", "").lstrip("./").split("/")[0]
|
||||
not in DELIVERY_ROOTS
|
||||
if _reconciled_path(d).split("/")[0] not in DELIVERY_ROOTS
|
||||
and _reconciled_path(d) not in NON_SERVING_RECONCILED
|
||||
})
|
||||
|
||||
# The exemption's own guard: an exempted tree must actually serve nothing.
|
||||
exempt_serving = sorted({
|
||||
f"{p.relative_to(ROOT)} ({d.get('kind')})"
|
||||
for p, d in docs_all()
|
||||
if any(str(p.relative_to(ROOT)).startswith(t + "/") for t in NON_SERVING_RECONCILED)
|
||||
and (d.get("kind") == "HTTPRoute"
|
||||
or (d.get("kind") == "Service"
|
||||
and any("hostname" in str(k).lower()
|
||||
for k in ((d.get("metadata", {}) or {}).get("annotations", {}) or {}))))
|
||||
})
|
||||
check(not exempt_serving,
|
||||
f"(0b) every NON_SERVING_RECONCILED tree really serves nothing — an exempted tree that "
|
||||
f"gains a route would hide a service from this check entirely "
|
||||
f"{exempt_serving if exempt_serving else ''}")
|
||||
check(len(reconcilers) >= 10 and not unscanned,
|
||||
f"(0) all {len(reconcilers)} Flux-reconciled path(s) lie inside a scanned root — a "
|
||||
f"reconciled tree this check does not read is a service it cannot require a link for "
|
||||
|
||||
@@ -38,8 +38,21 @@ agents. So the credential model must be decided by the operator before anything
|
||||
| Runner deployment | `apps/wave-1/gitea-actions.yaml` | act_runner + DinD (Docker executor). **`automountServiceAccountToken: false`** — the runner and its job containers hold NO SA token. Broad egress allowed (`- {}`), so it can *reach* the kube-API VIP `192.168.1.49:6443`, but has no credential. |
|
||||
| Secrets encryption | `.sops.yaml` (repo root) | age recipient `age175l8m…mfzg`; encrypts `data`/`stringData` of `*.sops.yaml` under `platform|apps|catalog/**` and `secrets/**`. Private key is the in-cluster `sops-age` Secret Flux uses. |
|
||||
|
||||
**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the dev cluster
|
||||
is LIVE (Phase 6). So `CLUSTER_AVAILABLE` is the only remaining gate, exactly as the blueprint says.
|
||||
**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the cluster is
|
||||
LIVE (and it is now PROD; dev was decommissioned 2026-09-12).
|
||||
|
||||
> **CORRECTION (2026-09-29): `CLUSTER_AVAILABLE` is NOT "the only remaining gate", and setting it
|
||||
> today is actively DANGEROUS.** Measured: with `CLUSTER_AVAILABLE=true` and no reachable cluster,
|
||||
> `conformance/run.sh` exits 0 in scaffold mode having marked all 54 tests `<skipped>`, and
|
||||
> `ci/conformance.sh` then printed `[ ok ] L2/L3 conformance passed for touched subset` — **rc 0, a
|
||||
> fake green**, on a merge gate that is now an enforcing required status check. The CI runner has
|
||||
> no cluster credential at all (`automountServiceAccountToken: false`, by design), so the "local
|
||||
> run" this stage attempts can never succeed; it can only skip and, before the guard added on
|
||||
> 2026-09-29, lie about it.
|
||||
>
|
||||
> `ci/conformance.sh` now BLOCKS (exit 1) when a cluster is declared but every test skipped. **Do
|
||||
> not set `CLUSTER_AVAILABLE=true` until §3's trigger path actually exists** — the variable does
|
||||
> not enable L2/L3, it only changes which way this stage fails.
|
||||
|
||||
---
|
||||
|
||||
|
||||
+32
-4
@@ -86,6 +86,32 @@ enabling enforcement and probing the push. `require-gate.sh` is the enforcement
|
||||
|
||||
Re-run `verify-then-enforce.sh` after any future change to the act-runner Deployment.
|
||||
|
||||
### `expected-branch-protection.json` + `ci/gitea-settings.sh` — the detector
|
||||
|
||||
`expected-branch-protection.json` **declares** the desired rule. `ci/gitea-settings.sh` runs as
|
||||
a merge-gate stage, reads the LIVE rule, and **fails the gate on any divergence** — plus asserts
|
||||
at least one runner is online. It is the only thing in this repository that can see Gitea's own
|
||||
database.
|
||||
|
||||
It reads `/branch_protections/main`, not `/branches/main`: the branch view does **not** expose
|
||||
`block_admin_merge_override`, and it showed all-green for hours while the rule was overridable.
|
||||
|
||||
Behaviour, proven by test rather than assumed:
|
||||
|
||||
| Situation | Exit | Effect |
|
||||
|---|---|---|
|
||||
| live matches the declaration | 0 | green |
|
||||
| any key drifted, or no runner online | 1 | **blocks the merge** |
|
||||
| no `GITEA_SETTINGS_TOKEN`, or Gitea unreachable | 3 | skips **loudly**; never a pass, never a block |
|
||||
|
||||
The skip is deliberately *not* a block: an unwired credential blocking every PR is the
|
||||
lock-yourself-out failure this repo keeps hitting. Wire the token with `set-ci-token.sh`, confirm
|
||||
the stage goes **green on a real PR**, and only then set `REQUIRE_GITEA_SETTINGS=true` in
|
||||
`gate.yml` so a future missing token blocks instead of skipping.
|
||||
|
||||
**Changing the declaration is how you change policy.** If the live state is what you now want,
|
||||
edit this JSON in the same commit — deliberately, in review — rather than letting the two drift.
|
||||
|
||||
### Why these scripts exist at all, and what is still missing
|
||||
|
||||
Gitea's repo settings live in **its own database**, not in anything Flux reconciles or the
|
||||
@@ -94,10 +120,12 @@ the root cause of the entire 2026-09-29 session: branch protection was off and t
|
||||
gone for a month while the log recorded both as working, and **no check existed that could
|
||||
have caught it.**
|
||||
|
||||
These scripts encode the desired state and can re-assert it, but they are not a detector. The
|
||||
durable fix is a conformance check that reads the live Gitea API and fails when it diverges
|
||||
from what is declared here — turning a class of silent, invisible failure into a loud one.
|
||||
**That check does not exist yet and is the highest-value remaining work in this directory.**
|
||||
These scripts encode the desired state and can re-assert it; `ci/gitea-settings.sh` above is the
|
||||
**detector** that turns divergence from a silent, invisible failure into a loud one. It closes the
|
||||
root cause of the 2026-09-29 session.
|
||||
|
||||
It is only as good as its wiring: **until `GITEA_SETTINGS_TOKEN` exists it skips**, and a skipped
|
||||
detector detects nothing. That is the remaining step.
|
||||
|
||||
## Historical sequence (kept for context)
|
||||
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"_comment": "DECLARED desired state of Gitea branch protection on main. ci/gitea-settings.sh fails the merge gate when the LIVE Gitea diverges from this. Gitea settings live in Gitea database - invisible to Flux, unreadable by every other check - so this file plus that check are the only drift detection that exists for them.",
|
||||
"_generated": "2026-09-29 from the live rule, after the session that found protection OFF and zero runners for a month",
|
||||
"branch": "main",
|
||||
"runners_online_min": 1,
|
||||
"protection": {
|
||||
"enable_push": false,
|
||||
"enable_push_whitelist": false,
|
||||
"enable_merge_whitelist": false,
|
||||
"enable_status_check": true,
|
||||
"status_check_contexts": [
|
||||
"gate / gate (pull_request)"
|
||||
],
|
||||
"required_approvals": 1,
|
||||
"enable_approvals_whitelist": false,
|
||||
"block_admin_merge_override": true,
|
||||
"block_on_rejected_reviews": true,
|
||||
"dismiss_stale_approvals": true
|
||||
}
|
||||
}
|
||||
Executable
+56
@@ -0,0 +1,56 @@
|
||||
#!/usr/bin/env bash
|
||||
# Mint a Gitea API token and store it as the repo Actions secret GITEA_SETTINGS_TOKEN,
|
||||
# so ci/gitea-settings.sh can read the live branch-protection rule during the gate.
|
||||
#
|
||||
# The token value is NEVER printed. It goes from the token API into the secrets API and
|
||||
# nowhere else -- not to stdout, not to a file, not into an agent transcript.
|
||||
#
|
||||
# Scope: `read:repository` + `read:admin` are what the check needs (the rule object, and the
|
||||
# runner list). Gitea token scopes are coarse; grant the least that works on your version and
|
||||
# confirm the check goes green rather than assuming the scope name.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
|
||||
# NOT GITEA_* — Gitea reserves that prefix for secret names and returns HTTP 400
|
||||
# "invalid secret name". The first version of this script used GITEA_SETTINGS_TOKEN, sent the
|
||||
# PUT to /dev/null, and printed "stored" regardless; the secret never existed and nobody knew.
|
||||
SECRET=SETTINGS_API_TOKEN
|
||||
NAME="ci-gitea-settings-$(date +%Y%m%d%H%M%S)"
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
basic() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
||||
|
||||
echo "==> minting token '${NAME}' as ${U}"
|
||||
TOKEN="$(basic -X POST -H 'Content-Type: application/json' \
|
||||
-d "{\"name\":\"${NAME}\",\"scopes\":[\"read:repository\",\"read:admin\"]}" \
|
||||
"https://${HOST}/api/v1/users/${U}/tokens" | jq -r '.sha1 // empty')"
|
||||
[ -n "$TOKEN" ] || { echo "token creation failed (check scopes for your Gitea version)" >&2; exit 1; }
|
||||
echo "==> got a ${#TOKEN}-character token (value not shown)"
|
||||
|
||||
echo "==> storing it as repo Actions secret ${SECRET}"
|
||||
code="$(basic -o /tmp/set-ci-token.out -w '%{http_code}' -X PUT -H 'Content-Type: application/json' \
|
||||
-d "$(jq -n --arg d "$TOKEN" '{data:$d}')" \
|
||||
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets/${SECRET}")"
|
||||
unset TOKEN
|
||||
if [ "$code" != 201 ] && [ "$code" != 204 ]; then
|
||||
echo " PUT failed: HTTP ${code} -- $(head -c 200 /tmp/set-ci-token.out)" >&2
|
||||
rm -f /tmp/set-ci-token.out; exit 1
|
||||
fi
|
||||
rm -f /tmp/set-ci-token.out
|
||||
|
||||
# VERIFY BY READING IT BACK. curl -sS exits 0 on an HTTP error, so without this the script
|
||||
# reports success for a store that never happened -- which is exactly what it did.
|
||||
names="$(basic "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets" \
|
||||
| jq -rc '[.. | objects | select(has("name")) | .name] | unique')"
|
||||
grep -q "$SECRET" <<<"$names" \
|
||||
|| { echo " stored HTTP ${code} but ${SECRET} is NOT in ${names} -- refusing to claim success" >&2; exit 1; }
|
||||
echo "==> stored and VERIFIED present: ${names}"
|
||||
|
||||
cat <<'NOTE'
|
||||
|
||||
Next:
|
||||
1. .gitea/workflows/gate.yml already passes this secret to the gate step.
|
||||
2. Open any PR and confirm the gate prints "gitea-settings: green" -- it SKIPS silently
|
||||
without the token, so a green run is the only proof the wiring works.
|
||||
3. THEN set REQUIRE_GITEA_SETTINGS=true in gate.yml, so a future missing token BLOCKS
|
||||
instead of skipping. Do that only after you have seen it green, or you lock the repo.
|
||||
NOTE
|
||||
@@ -34,6 +34,14 @@ resources:
|
||||
# policies, the next import is blocked in the gap.
|
||||
- ../platform/kyverno/policies/label-cdi-worker-pods.yaml
|
||||
- ../platform/kyverno/admission/secret-provenance.yaml
|
||||
# R-SEC-02 — bounds what the conformance CI trigger may create. MUST be listed HERE: this
|
||||
# Kustomization is what actually applies ClusterPolicies. Adding a policy only to
|
||||
# platform/kyverno/admission/kustomization.yaml is a SILENT NO-OP — that file is reference
|
||||
# only ("kept as reference; THIS file is the buildable render root" in
|
||||
# platform/kyverno/kustomization.yaml, which itself ships only the namespace + HelmRelease).
|
||||
# That is exactly how this policy shipped green and never reached the cluster on 2026-09-29,
|
||||
# leaving the conformance-trigger identity live with nothing bounding it.
|
||||
- ../platform/kyverno/admission/conformance-trigger-job-shape.yaml
|
||||
# cilium: L2 LB IP pool + announcement policy (cilium.io CRDs already registered by Cilium)
|
||||
- cilium/l2-ip-pool.yaml
|
||||
- cilium/l2-announcement-policy.yaml
|
||||
|
||||
+179
-10
@@ -21,7 +21,7 @@ Cilium L2 LoadBalancer (game protocols authenticate players themselves).
|
||||
| PVC subPaths | `servers`→`/crafty/servers`, `backups`→`/crafty/backups`, `logs`→`/crafty/logs`, `import`→`/crafty/import`, `config`→`/crafty/app/config` |
|
||||
| Identity | `runAsUser:1000 runAsGroup:0 fsGroup:0 runAsNonRoot:true` (see below) |
|
||||
| Web host | `crafty.home.keithjackson.dev` → shared Gateway → `crafty` Service :80 → sidecar :4180 → Crafty :8443 |
|
||||
| Game VIP | **192.168.1.55** (Cilium L2 LoadBalancer; `.53`=Gateway, `.54`=unifi, `.55`=crafty) |
|
||||
| Game VIP | `${CRAFTY_L4_VIP}` (Cilium L2 LoadBalancer) — **prod `192.168.2.22`** (live `EXTERNAL-IP`, measured 2026-09-29); base/dev `192.168.1.55` |
|
||||
| Game ports | starter set: **TCP 25565-25567** (Java), **UDP 19132-19134** (Bedrock) |
|
||||
|
||||
### fsGroup / user decision (the seerr lesson, applied)
|
||||
@@ -76,15 +76,16 @@ can't span namespaces). Both are age-encrypted to the repo `sops-age` recipient
|
||||
## Game-port L4 exposure
|
||||
|
||||
The Gateway is L7-HTTPS-only; raw TCP/UDP game ports can't ride it. `service-l4.yaml` exposes them as
|
||||
two `type=LoadBalancer` Services (TCP + UDP) sharing the pinned VIP **192.168.1.55**
|
||||
two `type=LoadBalancer` Services (TCP + UDP) sharing the pinned VIP `${CRAFTY_L4_VIP}` (**prod
|
||||
`192.168.2.22`**)
|
||||
(`lbipam.cilium.io/sharing-key: crafty`), ARP-announced by the existing `default-l2` policy. This path
|
||||
is **NOT auth-gated** — Minecraft Java (Mojang/Microsoft) and Bedrock (Xbox Live) authenticate players
|
||||
themselves. `external-dns` publishes `crafty-mc.home.keithjackson.dev → 192.168.1.55`.
|
||||
themselves. `external-dns` publishes `crafty-mc.home.keithjackson.dev → ${CRAFTY_L4_VIP}`.
|
||||
|
||||
### Adding more game ports (when the operator creates servers)
|
||||
|
||||
The operator has **not created any servers yet**; the starter set covers 3 Java + 3 Bedrock slots. To
|
||||
add capacity:
|
||||
As of 2026-09-29 **two** servers exist (see *Runtime state* below): Java `homelab` on 25565 and
|
||||
Bedrock `K3s-Bedrock` on 19132. The starter set covers 3 Java + 3 Bedrock slots. To add capacity:
|
||||
|
||||
1. In the Crafty UI, create the server and bind it to a NEW port (Java TCP or Bedrock UDP).
|
||||
2. Add that port to the matching Service in `service-l4.yaml` (one
|
||||
@@ -93,6 +94,149 @@ add capacity:
|
||||
4. (Optional) add a `containerPort` in `deployment.yaml` for documentation.
|
||||
5. Commit → Flux reconciles.
|
||||
|
||||
### Bedrock on UDP 19132 — the transport trap, and the ERROR banner that is NOT a refusal (MEASURED on prod 2026-09-29)
|
||||
|
||||
Two traps in sequence. The first hides the UDP socket; the second frightens you into undoing the fix.
|
||||
As of 2026-09-29 21:34 this server is **reachable on `19132/udp` and answering**, measured from
|
||||
off-cluster. Read both traps before changing anything.
|
||||
|
||||
#### Trap 1 — `transport=nethernet` (the shipped default) opens no RakNet UDP listener
|
||||
|
||||
Measured 20:54–21:32 against `K3s-Bedrock` (`c5837301-12b1-404e-b50e-70eafb37226b`, DB
|
||||
`type=minecraft-bedrock`, `server_port=19132`, process `./bedrock_server` alive):
|
||||
|
||||
- `/proc/net/udp` and `/proc/net/udp6` held **no socket on 19132** (`0x4ABC`) at all. Port 19132 was
|
||||
bound as **TCP** (`/proc/net/tcp6`, `…:4ABC`, state `0A`/LISTEN), not UDP.
|
||||
- That is nethernet working as designed, and the **shipped `bedrock_server_how_to.html`** — the
|
||||
vendor's own doc, sitting on the PVC beside the binary — says exactly why: nethernet *"listens on a
|
||||
single dual-stack **TCP** socket on `server-port` for an HTTP-based signaling handshake, then
|
||||
negotiates a UDP connection with each client that carries the gameplay traffic."*
|
||||
- **So nethernet could never have worked here anyway, on two independent counts:** `service-l4.yaml`
|
||||
exposes 19132 **only in the UDP Service** (`crafty-l4-tcp` carries 25565-25567 and nothing else), and
|
||||
`crafty-allow-ingress` allows `19132 UDP` and **not** `19132 TCP`. The signaling port was unreachable
|
||||
at both the Service and the policy layer. The server's own `packet-statistics.txt` for that run
|
||||
agrees: `Total Received: 0, 0B` over 2293 s.
|
||||
- Crafty does not manage this property (`transport` appears nowhere under `/crafty/app/classes/`) and
|
||||
its *only* Bedrock probe is `raknet_ping.py`, so the panel logged `WARNING
|
||||
app.classes.remote_stats.raknet_ping - Got timeout while issuing bedrock ping to 127.0.0.1:19132`
|
||||
every ~5 s — **155 such lines** — against a server it had started successfully.
|
||||
|
||||
#### Trap 2 — under `transport=raknet` this build prints a scary ERROR banner and then works anyway
|
||||
|
||||
The operator set `transport=raknet` at 21:33:31 (`/crafty/logs/audit.log`: `Edited file
|
||||
/crafty/servers/c5837301-…/server.properties`) and started the server at 21:33:58. BDS **1.26.52**
|
||||
printed, at ERROR level:
|
||||
|
||||
```
|
||||
[ERROR] ================ TRANSPORT TYPE ERROR ===================
|
||||
[ERROR] Your current connection type is not set to NetherNet. In this release, NetherNet is the only
|
||||
supported transport type.
|
||||
[ERROR] Players will not be able to connect to your game without NetherNet.
|
||||
[ERROR] To switch, set 'transport=nethernet' in server.properties.
|
||||
```
|
||||
|
||||
**That banner is advisory, not fatal. Do not revert `transport` on seeing it.** Every claim it makes
|
||||
about connectivity was false on this cluster within five seconds of it being printed:
|
||||
|
||||
- The process **kept running** (`./bedrock_server`, pid 47006) and **bound the RakNet sockets**:
|
||||
`/proc/net/udp` `00000000:4ABC` (19132, IPv4) and `/proc/net/udp6` `…:4ABD` (19133, IPv6), both owned
|
||||
by `bedrock_server` — `/proc/47006/fd/25` and `fd/26` → those socket inodes. The earlier **TCP** bind
|
||||
on `4ABC` is **gone**.
|
||||
- Crafty's warning **changed kind** at that instant: last `Got timeout` at `21:34:04,306`, first
|
||||
`Bedrock ping returned a string with too few fields` at `21:34:09,854`. That distinction is
|
||||
load-bearing — read `raknet_ping.py`: `Got timeout` is emitted *only* when `sock.recv` raises
|
||||
`TimeoutError` (nothing came back at all), whereas `too few fields` is reachable *only after* a
|
||||
`0x1C` Unconnected Pong has arrived **and its magic has validated**. The server is answering pings.
|
||||
- **End-to-end, from off-cluster:** a RakNet Unconnected Ping sent from the LAN to the prod VIP
|
||||
`192.168.2.22:19132` returns a 139-byte `0x1c` Unconnected Pong in ~12 ms, magic valid:
|
||||
|
||||
MCPE;Dedicated Server;2193;1.26.52;0;10;11676629557768398479;Bedrock level;Survival;1;19132;19133;0;0;0;
|
||||
|
||||
Edition `MCPE`, protocol `2193`, version **`1.26.52`**, `0/10` players. **Players can reach this
|
||||
server now**, through the VIP, the Service, and the CiliumNetworkPolicy, with no manifest change.
|
||||
- The vendor doc on disk contradicts the banner and matches the observed behaviour. `transport` is
|
||||
documented as *"Allowed values: raknet, nethernet"*, and: *"**raknet** — The traditional Bedrock UDP
|
||||
transport. The server listens directly on `server-port` (IPv4) and `server-portv6` (IPv6), and
|
||||
clients connect to that address and port."* RakNet is also still fully compiled into the binary —
|
||||
`strings bedrock_server` yields `RakNetConnector`, `createUniqueRakPeer`, `RakPeerInterface` and the
|
||||
vendored build path `…/src-deps/raknet/raknet/RakNetSocket2.cpp`.
|
||||
|
||||
So *"NetherNet is the only supported transport type"* is a **support statement, not a capability
|
||||
statement**: Microsoft declines to support RakNet and ships it working. Treat the banner as a
|
||||
deprecation notice with an **unknown expiry** — a future BDS may genuinely remove RakNet, and that is
|
||||
the day this exposure needs GeyserMC (see *If RakNet is ever really removed* below).
|
||||
|
||||
> **The shipped-comment trap, stated plainly.** `server.properties` lists `raknet` as allowed and the
|
||||
> binary logs an ERROR saying it is not supported. Both are the vendor, and they disagree. The tiebreak
|
||||
> is neither prose: it is `/proc/net/udp` and a pong on the wire. Whenever this file says a Bedrock
|
||||
> server is or is not reachable, it is because someone pinged it — not because a comment said so.
|
||||
|
||||
#### The steady-state `too few fields` warning is benign and is now the expected signature
|
||||
|
||||
Crafty will log `Bedrock ping returned a string with too few fields - missing values [None]` every
|
||||
~5 s forever. **That means the ping SUCCEEDED.** BDS 1.26.52's pong carries 16 `;`-separated fields;
|
||||
Crafty 4.10.8's `pingpong_string_fields` table knows 14 and expects the last to be an empty
|
||||
terminator. It instead finds `'0'`, so `last_enumeration` stops at index 12 and the parser reports the
|
||||
one `None` terminator entry as missing. Every field that matters (edition, MOTD, protocol, version
|
||||
name, player count, max players, uuid, game mode, both ports) parsed correctly. **Do not chase this
|
||||
warning.** The failure signature is the *other* message, `Got timeout`.
|
||||
|
||||
#### Decision: `transport=raknet` stays; the 19132-19134/UDP exposure is correct and live
|
||||
|
||||
No manifest change is needed or wanted. `service-l4.yaml`, `networkpolicy.yaml` and the
|
||||
`containerPort`s were already shaped for exactly RakNet, and that shape is now proven end-to-end
|
||||
rather than assumed. The three options that would have required manifest work are all **not
|
||||
recommended** today:
|
||||
|
||||
| Option | Manifest consequence | Verdict |
|
||||
|---|---|---|
|
||||
| **Keep `transport=raknet`** (recommended) | **none** — the existing Service, policy and ports are already correct and now proven | **Take this.** Zero change, works today, vanilla parity, Bedrock clients join a real Bedrock world. |
|
||||
| **GeyserMC + Floodgate** on the `homelab` Paper server | none either (Geyser also binds `19132/udp`), but it would **collide** with the live BDS on 19132 — one of them must move | Hold in reserve. Only worth it if RakNet is really removed, or if you *want* one shared world. Cost: Bedrock players join the **Java** world, not a Bedrock world; Java-only mechanics and some content differ; it is another moving part to patch. |
|
||||
| **Pin/downgrade BDS** to a pre-nethernet release | none | Unnecessary — the *current* build already speaks RakNet. Also fragile: Microsoft removes old BDS downloads, and Crafty's Bedrock installer tracks latest. UNVERIFIED whether Crafty can pin a version at all. |
|
||||
| **A third-party server** (PocketMine-MP, Nukkit) | none | No. Loses vanilla parity and the world format; a large change to solve a problem that is not occurring. |
|
||||
| **Accept NetherNet** | 19132-19134/UDP becomes **inert**; nethernet would instead need **TCP** 19132 added to `crafty-l4-tcp` *and* to `crafty-allow-ingress`, plus `server-udp-ports` pinned and exposed for the negotiated media path | No. Strictly more manifest work than the option that already works, and the WebRTC media path behind a shared VIP and SNAT is unproven here. |
|
||||
|
||||
#### If RakNet is ever really removed
|
||||
|
||||
Then BDS opens no UDP listener and this exposure has nothing to bind to — an architecture problem, not
|
||||
a config one. The migration is **GeyserMC + Floodgate as a plugin on the existing `homelab` Paper
|
||||
server**, because Geyser speaks RakNet on `19132/udp` itself: the VIP, the Service, the policy and the
|
||||
`containerPort`s all stay **byte-for-byte as they are**, and Bedrock clients keep connecting to
|
||||
`crafty-mc.home.keithjackson.dev:19132`. What changes is the world: Bedrock players land in the Java
|
||||
world via Floodgate (no Xbox-authenticated Bedrock account needed on the Java side), the Bedrock server
|
||||
is retired, and `K3s-Bedrock` stops the day Geyser starts — **they cannot share 19132**. Installing it
|
||||
is a PVC write plus a Paper restart, i.e. an **operator action**: drop `Geyser-Spigot.jar` and
|
||||
`floodgate-spigot.jar` into `/crafty/servers/13c7b85d-…/plugins/` and restart `homelab`, then set
|
||||
Geyser's `bedrock.port: 19132` and `clone-remote-port: false` in its generated `config.yml`.
|
||||
Geyser's own Bedrock-version support is **external and unmeasured here** — check that it supports the
|
||||
Bedrock client version in use before committing.
|
||||
|
||||
### Bedrock realities that are NOT platform defects but present identically
|
||||
|
||||
- **LAN / Friends-tab discovery can never work here.** Bedrock discovery is a UDP broadcast on 19132
|
||||
inside one L2 segment; this server sits behind a LoadBalancer VIP (and SNAT), not on the LAN, so it
|
||||
will **never** appear in a client's Friends/LAN list. Players must use **Add Server** and type the
|
||||
address and port by hand: `crafty-mc.home.keithjackson.dev` (or the VIP) with port **19132**.
|
||||
`external-dns` publishes that name from the **-tcp** Service, so it resolves to the shared VIP and
|
||||
is correct for Bedrock too — only the port differs from Java's 25565.
|
||||
- **Consoles cannot reach this at all.** Switch/Xbox/PlayStation Bedrock clients have no
|
||||
arbitrary-server-address field; only Windows/mobile/Fire do. Reaching it from a console needs a DNS
|
||||
redirect of a featured-server hostname, which is out of scope for `service-l4.yaml`.
|
||||
- **`enable-lan-visibility=true` (the shipped default) collides with Bedrock slots 1 and 2 — now
|
||||
OBSERVED, not merely documented.** Under `transport=raknet` it makes a server bind the **default**
|
||||
ports 19132 *and* 19133 even when `server-port`/`server-portv6` say otherwise; the shipped
|
||||
`bedrock_server_how_to.html` says so, and on 2026-09-29 a RakNet ping to
|
||||
`192.168.2.22:19133` returned **the same pong, from the same `server_guid`, as `:19132`** — one
|
||||
server occupying two of the three Bedrock slots. `:19134` correctly timed out. So a second Bedrock
|
||||
server put on 19133 will fight the first. Set `enable-lan-visibility=false` on every Bedrock server
|
||||
after the first; LAN discovery cannot work here anyway (above).
|
||||
- **`server-portv6` is the IPv6 listener and is not exposed; `19133` is nonetheless in use.**
|
||||
`service-l4.yaml` is IPv4 single-stack, so the *IPv6* socket `…:4ABD` that BDS opens for
|
||||
`server-portv6=19133` is unreachable from the LAN. But because `enable-lan-visibility=true` (above)
|
||||
the server *also* answers on **IPv4 19133**, which the UDP Service does expose — measured. So
|
||||
`19133` in `server.properties` genuinely does mean "slot 1 is taken", contrary to what this file
|
||||
previously said.
|
||||
|
||||
## Policies
|
||||
|
||||
`namespace.yaml` sets only `gateway-access=shared` (platform namespace, like headlamp/homepage), so
|
||||
@@ -114,14 +258,35 @@ Gateway VIP). The sidecar→Crafty hop is pod-local (`127.0.0.1`) and not subjec
|
||||
local admin. A **non-platform-admins** user → **403** (authenticated but not authorized).
|
||||
2. **Pod health / fsGroup:** `kubectl -n crafty get pod` Ready 2/2; Crafty writes its DB under
|
||||
`/crafty/app/config` and `/crafty/servers` without EACCES.
|
||||
3. **Server-create + game reach:** in the Crafty UI create a Java server bound to `25565`; start it;
|
||||
connect a Minecraft client to `crafty-mc.home.keithjackson.dev:25565` (or `192.168.1.55:25565`)
|
||||
from the LAN and confirm the world loads. (Bedrock: a server on `19132/udp` reached at the VIP.)
|
||||
3. **Server-create + game reach (Java):** in the Crafty UI create a Java server bound to `25565`;
|
||||
start it; connect a Minecraft client to `crafty-mc.home.keithjackson.dev:25565` (or
|
||||
`192.168.2.22:25565`) from the LAN and confirm the world loads.
|
||||
4. **Server-create + game reach (Bedrock):** creating and starting the server is **not sufficient**.
|
||||
Check, in this order (see *the transport trap* above): `transport=raknet` in `server.properties`;
|
||||
a live **UDP** listener (`grep 4ABC /proc/net/udp /proc/net/udp6` → a wildcard bind); and a real
|
||||
pong from off-cluster. **Ignore the `TRANSPORT TYPE ERROR` banner in the console and ignore
|
||||
Crafty's `too few fields` warning — both appear on a fully working server.** The authoritative
|
||||
test is a RakNet Unconnected Ping to `192.168.2.22:19132` returning a `0x1c` packet whose MOTD
|
||||
string starts `MCPE;`. Then **Add Server** by hand in a Windows/mobile client at
|
||||
`crafty-mc.home.keithjackson.dev` port `19132`. It will never show up in the LAN/Friends tab, and a
|
||||
console cannot reach it.
|
||||
|
||||
## Runtime state: the servers themselves are NOT in git
|
||||
|
||||
As of 2026-09-14 one Java server exists — **"homelab"**, Paper 26.2, TCP 25565, 2 GiB heap, autostart
|
||||
on, UUID `13c7b85d-00bb-4a58-8c86-2433ca47891c`. It was created through Crafty's v2 API, and **Mojang's
|
||||
As of 2026-09-29 **two** servers exist (both measured live, `select * from servers`):
|
||||
|
||||
| Name | Type | Port | UUID | `auto_start` |
|
||||
|---|---|---|---|---|
|
||||
| `homelab` | `minecraft-java` (Paper 26.2, 2 GiB heap) | TCP 25565 | `13c7b85d-00bb-4a58-8c86-2433ca47891c` | **1** |
|
||||
| `K3s-Bedrock` | `minecraft-bedrock` | UDP 19132 | `c5837301-12b1-404e-b50e-70eafb37226b` | **0** ⚠ |
|
||||
|
||||
`K3s-Bedrock` was created 2026-09-29 20:51 and **still** has `auto_start=0` **and**
|
||||
`crash_detection=0` — re-measured 2026-09-29 21:41 straight out of `crafty.sqlite` (read-only), so the
|
||||
create-API trap documented at the bottom of this file is still live and still unfixed. **It will not
|
||||
come back after a pod restart**, and merging any change to `deployment.yaml` restarts the pod
|
||||
(`strategy: Recreate`). PATCH both to true and read them back.
|
||||
|
||||
`homelab` was created through Crafty's v2 API, and **Mojang's
|
||||
EULA (<https://aka.ms/MinecraftEULA>) was accepted on the operator's behalf** to let it start; see the
|
||||
DECISIONS-LOG entry of that date.
|
||||
|
||||
@@ -132,6 +297,10 @@ application state on the `crafty-data` Longhorn PVC, in two places, and **Flux c
|
||||
heap sizes, `auto_start` / `crash_detection`.
|
||||
- `/crafty/servers/<uuid>/` (`servers` subPath) — `paper.jar`, `eula.txt`, `server.properties`, and
|
||||
**the generated world**. The jar is re-downloadable; the world is not.
|
||||
**`transport=raknet` lives here too.** It is the one setting that makes Bedrock reachable at all (see
|
||||
*the transport trap*), it is not in git, Flux cannot restore it, and a freshly created or reinstalled
|
||||
Bedrock server gets the vendor default `transport=nethernet` back — i.e. it silently regresses to
|
||||
"nothing bound on 19132/udp". Re-apply and re-ping after any Bedrock server recreate or BDS update.
|
||||
|
||||
So a rebuild from git alone yields a healthy but **empty** Crafty with nothing listening on 25565.
|
||||
Recreating a server by hand means: create it in the UI or API, accept the EULA again, and PATCH
|
||||
|
||||
@@ -0,0 +1,144 @@
|
||||
# Serves: R-SEC-02 (CI holds no broad cluster access)
|
||||
#
|
||||
# THIS POLICY IS THE SECURITY BOUNDARY. The RBAC in conformance/incluster/rbac.yaml is not.
|
||||
#
|
||||
# Kubernetes places NO `use` check on a Pod/Job's `serviceAccountName`. Any principal that can
|
||||
# `create jobs` in namespace `conformance` can therefore create a Job that runs as
|
||||
# `conformance-runner` — a privileged SA (nodes/patch, pods/exec, namespaces/patch) — with an
|
||||
# arbitrary command. So the CI trigger credential, however narrow its own Role looks, IS the
|
||||
# privileged identity unless something constrains the SHAPE of what it may create.
|
||||
#
|
||||
# That is what this does: for Jobs created by `conformance-trigger` ONLY, admit exactly the
|
||||
# committed conformance run and deny everything else. The operator chose this model
|
||||
# (docs/plans/l2-l3-conformance-ci-wiring.md §4, primary model) over the zero-credential
|
||||
# GitOps-native alternative.
|
||||
#
|
||||
# ENFORCE AND FAIL-CLOSED, NOT SUBSTITUTED. Note this does NOT use
|
||||
# ${KYVERNO_VALIDATION_ACTION} like the other admission policies: a security boundary that an
|
||||
# environment overlay can downgrade to Audit is not a boundary. `failurePolicy: Fail` means a
|
||||
# Kyverno outage blocks conformance Jobs rather than silently admitting unconstrained ones.
|
||||
# Blocking CI is the correct failure direction here; admitting arbitrary privileged code is not.
|
||||
#
|
||||
# SCOPE: it constrains ONLY `conformance-trigger`. Jobs created by humans, by Flux, or by any
|
||||
# other SA are untouched — so a broken policy cannot wedge the rest of the cluster.
|
||||
#
|
||||
# IMAGE PINNING — HONEST LIMIT, READ BEFORE RELYING ON THIS. The plan specifies pinning the
|
||||
# exact `CONFORMANCE_RUNNER_IMAGE` digest. That image does not exist yet, so this enforces the
|
||||
# enforceable half: the image must live under the conformance repository path AND be
|
||||
# digest-pinned (`@sha256:`), never a mutable tag. Residual surface: anyone who can PUSH to that
|
||||
# registry path could publish a different digest. TIGHTEN THIS when the image is built by
|
||||
# replacing the `image` pattern below with the literal digest — a one-line change, reviewed in a
|
||||
# PR, which is exactly where a change to the security boundary belongs.
|
||||
apiVersion: kyverno.io/v1
|
||||
kind: ClusterPolicy
|
||||
metadata:
|
||||
name: conformance-trigger-job-shape
|
||||
annotations:
|
||||
policies.kyverno.io/title: Constrain what the conformance CI trigger may create
|
||||
policies.kyverno.io/category: Supply chain
|
||||
policies.kyverno.io/severity: critical
|
||||
homelab.keithjackson.dev/requirements: R-SEC-02
|
||||
homelab.keithjackson.dev/scope: admission-only
|
||||
spec:
|
||||
validationFailureAction: Enforce
|
||||
failurePolicy: Fail
|
||||
background: false
|
||||
rules:
|
||||
- name: trigger-may-create-only-the-committed-conformance-run
|
||||
# resources AND subjects in the SAME filter entry — that is how Kyverno ANDs them.
|
||||
# As two separate `all` entries it is a schema error ("match resources not specified")
|
||||
# and the whole policy loads as INVALID, which the CLI reports as a SKIP.
|
||||
match:
|
||||
all:
|
||||
- resources:
|
||||
kinds:
|
||||
- Job
|
||||
namespaces:
|
||||
- conformance
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: conformance-trigger
|
||||
namespace: conformance
|
||||
validate:
|
||||
message: >-
|
||||
conformance-trigger may create ONLY the committed conformance run: a single container,
|
||||
digest-pinned conformance-runner image, serviceAccountName conformance-runner,
|
||||
command ["./run.sh"], no host namespaces, non-root with privilege escalation disabled,
|
||||
and only the `workspace` and `results` volumes. See
|
||||
platform/kyverno/admission/conformance-trigger-job-shape.yaml.
|
||||
pattern:
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: conformance-runner
|
||||
=(hostNetwork): "false"
|
||||
=(hostPID): "false"
|
||||
=(hostIPC): "false"
|
||||
containers:
|
||||
- image: "gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*"
|
||||
command:
|
||||
- "./run.sh"
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: false
|
||||
=(privileged): "false"
|
||||
|
||||
- name: trigger-volumes-must-be-workspace-or-results
|
||||
# resources AND subjects in the SAME filter entry — that is how Kyverno ANDs them.
|
||||
# As two separate `all` entries it is a schema error ("match resources not specified")
|
||||
# and the whole policy loads as INVALID, which the CLI reports as a SKIP.
|
||||
match:
|
||||
all:
|
||||
- resources:
|
||||
kinds:
|
||||
- Job
|
||||
namespaces:
|
||||
- conformance
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: conformance-trigger
|
||||
namespace: conformance
|
||||
validate:
|
||||
message: >-
|
||||
conformance-trigger Jobs may mount only the `workspace` and `results` volumes. An extra
|
||||
volume is how a constrained pod reaches a host path, a secret, or another SA's token.
|
||||
# foreach, NOT a `pattern` list: a Kyverno array pattern asserts "at least one element
|
||||
# matches", which would happily admit a Job carrying `workspace` PLUS a hostPath. This
|
||||
# asserts EVERY volume is one of the two allowed names.
|
||||
foreach:
|
||||
- list: "request.object.spec.template.spec.volumes"
|
||||
deny:
|
||||
conditions:
|
||||
all:
|
||||
- key: "{{ element.name }}"
|
||||
operator: AnyNotIn
|
||||
value: ["workspace", "results"]
|
||||
|
||||
- name: trigger-args-must-be-the-committed-flags
|
||||
# resources AND subjects in the SAME filter entry — that is how Kyverno ANDs them.
|
||||
# As two separate `all` entries it is a schema error ("match resources not specified")
|
||||
# and the whole policy loads as INVALID, which the CLI reports as a SKIP.
|
||||
match:
|
||||
all:
|
||||
- resources:
|
||||
kinds:
|
||||
- Job
|
||||
namespaces:
|
||||
- conformance
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: conformance-trigger
|
||||
namespace: conformance
|
||||
validate:
|
||||
message: >-
|
||||
conformance-trigger Job args may contain only the conformance run's own flags
|
||||
(--layer, --requirements, --include-scope, --strict-coverage) and their plain values.
|
||||
Anything else — a shell, a path, a redirect — is denied.
|
||||
foreach:
|
||||
- list: "request.object.spec.template.spec.containers[0].args"
|
||||
deny:
|
||||
conditions:
|
||||
all:
|
||||
- key: "{{ regex_match('^(--layer|--requirements|--include-scope|--strict-coverage|[A-Za-z0-9][A-Za-z0-9,._-]*)$', element) }}"
|
||||
operator: Equals
|
||||
value: false
|
||||
@@ -20,3 +20,7 @@ resources:
|
||||
# --- secrets: the admission variant REPLACES the CI-only sops-block policy ---
|
||||
# (../policies/disallow-plaintext-secrets.yaml is CI-only; see secret-provenance.yaml)
|
||||
- secret-provenance.yaml
|
||||
# --- R-SEC-02: constrain what the conformance CI trigger may create ---
|
||||
# THE security boundary for CI cluster access. Enforce + fail-closed, deliberately NOT
|
||||
# subject to ${KYVERNO_VALIDATION_ACTION}. See the file header.
|
||||
- conformance-trigger-job-shape.yaml
|
||||
@@ -0,0 +1,57 @@
|
||||
# R-SEC-02 — the conformance CI trigger cannot escalate to arbitrary privileged execution.
|
||||
#
|
||||
# Each DENY case below is a real escalation path, not a style violation. The one that matters
|
||||
# most is `evil-command`: correct SA, correct image, arbitrary command. That is the attack the
|
||||
# whole model exists to stop, and RBAC alone cannot — Kubernetes places no `use` check on
|
||||
# serviceAccountName, so `create jobs` in this namespace IS the privileged identity without
|
||||
# this policy.
|
||||
#
|
||||
# user-info.yaml supplies the requesting identity. WITHOUT it the policy's `subjects` match
|
||||
# never fires and every case trivially passes — a test that proves nothing.
|
||||
apiVersion: cli.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: conformance-trigger-job-shape
|
||||
policies:
|
||||
- ../../admission/conformance-trigger-job-shape.yaml
|
||||
resources:
|
||||
- resources.yaml
|
||||
userinfo: user-info.yaml
|
||||
results:
|
||||
# ---- the committed run is ADMITTED ----
|
||||
- policy: conformance-trigger-job-shape
|
||||
rule: trigger-may-create-only-the-committed-conformance-run
|
||||
kind: Job
|
||||
resources: [good-run]
|
||||
result: pass
|
||||
- policy: conformance-trigger-job-shape
|
||||
rule: trigger-volumes-must-be-workspace-or-results
|
||||
kind: Job
|
||||
resources: [good-run]
|
||||
result: pass
|
||||
- policy: conformance-trigger-job-shape
|
||||
rule: trigger-args-must-be-the-committed-flags
|
||||
kind: Job
|
||||
resources: [good-run]
|
||||
result: pass
|
||||
|
||||
# ---- THE escalation: runner SA + arbitrary command ----
|
||||
- policy: conformance-trigger-job-shape
|
||||
rule: trigger-may-create-only-the-committed-conformance-run
|
||||
kind: Job
|
||||
resources: [evil-command, evil-image, tag-not-digest, host-namespaces]
|
||||
result: fail
|
||||
|
||||
# ---- an extra volume is the way out of a constrained pod ----
|
||||
- policy: conformance-trigger-job-shape
|
||||
rule: trigger-volumes-must-be-workspace-or-results
|
||||
kind: Job
|
||||
resources: [extra-volume]
|
||||
result: fail
|
||||
|
||||
# ---- args beyond the committed flags ----
|
||||
- policy: conformance-trigger-job-shape
|
||||
rule: trigger-args-must-be-the-committed-flags
|
||||
kind: Job
|
||||
resources: [injected-args]
|
||||
result: fail
|
||||
@@ -0,0 +1,154 @@
|
||||
# The committed shape — must be ADMITTED.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: good-run
|
||||
namespace: conformance
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: conformance-runner
|
||||
containers:
|
||||
- name: runner
|
||||
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
|
||||
command: ["./run.sh"]
|
||||
args: ["--layer", "l2,l3", "--requirements", "R-NET-01"]
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: false
|
||||
volumes:
|
||||
- name: workspace
|
||||
emptyDir: {}
|
||||
- name: results
|
||||
emptyDir: {}
|
||||
---
|
||||
# THE ESCALATION: runner SA, but an arbitrary command. Must be DENIED.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: evil-command
|
||||
namespace: conformance
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: conformance-runner
|
||||
containers:
|
||||
- name: runner
|
||||
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
|
||||
command: ["/bin/sh", "-c", "kubectl get secrets -A"]
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: false
|
||||
volumes:
|
||||
- name: workspace
|
||||
emptyDir: {}
|
||||
---
|
||||
# An arbitrary image running as the privileged SA. Must be DENIED.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: evil-image
|
||||
namespace: conformance
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: conformance-runner
|
||||
containers:
|
||||
- name: runner
|
||||
image: docker.io/library/alpine:latest
|
||||
command: ["./run.sh"]
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: false
|
||||
volumes:
|
||||
- name: workspace
|
||||
emptyDir: {}
|
||||
---
|
||||
# A mutable TAG instead of a digest — republishable under the same reference. Must be DENIED.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: tag-not-digest
|
||||
namespace: conformance
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: conformance-runner
|
||||
containers:
|
||||
- name: runner
|
||||
image: gitea.home.keithjackson.dev/home-network/conformance-runner:latest
|
||||
command: ["./run.sh"]
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: false
|
||||
volumes:
|
||||
- name: workspace
|
||||
emptyDir: {}
|
||||
---
|
||||
# Host namespaces + privilege escalation. Must be DENIED.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: host-namespaces
|
||||
namespace: conformance
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: conformance-runner
|
||||
hostPID: true
|
||||
containers:
|
||||
- name: runner
|
||||
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
|
||||
command: ["./run.sh"]
|
||||
securityContext:
|
||||
runAsNonRoot: false
|
||||
allowPrivilegeEscalation: true
|
||||
volumes:
|
||||
- name: workspace
|
||||
emptyDir: {}
|
||||
---
|
||||
# An extra volume — the way out of a constrained pod. Must be DENIED.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: extra-volume
|
||||
namespace: conformance
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: conformance-runner
|
||||
containers:
|
||||
- name: runner
|
||||
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
|
||||
command: ["./run.sh"]
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: false
|
||||
volumes:
|
||||
- name: workspace
|
||||
emptyDir: {}
|
||||
- name: hostroot
|
||||
hostPath:
|
||||
path: /
|
||||
---
|
||||
# Injected args beyond the committed flags. Must be DENIED.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: injected-args
|
||||
namespace: conformance
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: conformance-runner
|
||||
containers:
|
||||
- name: runner
|
||||
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
|
||||
command: ["./run.sh"]
|
||||
args: ["--layer", "l2", ";", "curl", "http://evil/"]
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: false
|
||||
volumes:
|
||||
- name: workspace
|
||||
emptyDir: {}
|
||||
@@ -0,0 +1,12 @@
|
||||
# The requesting identity the policy keys on. Without this the `subjects` match never fires and
|
||||
# every case below would trivially "pass" — proving nothing.
|
||||
apiVersion: cli.kyverno.io/v1alpha1
|
||||
kind: UserInfo
|
||||
metadata:
|
||||
name: conformance-trigger
|
||||
userInfo:
|
||||
username: system:serviceaccount:conformance:conformance-trigger
|
||||
groups:
|
||||
- system:serviceaccounts
|
||||
- system:serviceaccounts:conformance
|
||||
- system:authenticated
|
||||
Reference in new issue
Block a user